Strictly standardized mean difference

In statistics, the strictly standardized mean difference (SSMD) is a measure of effect size. It is the mean divided by the standard deviation of a difference between two random values each from one of two groups. It was initially proposed for quality control ^[1] and hit selection ^[2] in high-throughput screening (HTS) and has become a statistical parameter measuring effect sizes for the comparison of any two groups with random values. ^[3]

Background

In high-throughput screening (HTS), quality control (QC) is critical. An important QC characteristic in a HTS assay is how much the positive controls, test compounds, and negative controls differ from one another. This QC characteristic can be evaluated using the comparison of two well types in HTS assays. Signal-to-noise ratio (S/N), signal-to-background ratio (S/B), and the Z-factor have been adopted to evaluate the quality of HTS assays through the comparison of two investigated types of wells. However, the S/B does not take into account any information on variability; and the S/N can capture the variability only in one group and hence cannot assess the quality of assay when the two groups have different variabilities. ^[1] Zhang JH et al. proposed the Z-factor. ^[4] The advantage of the Z-factor over the S/N and S/B is that it takes into account the variabilities in both compared groups. As a result, the Z-factor has been broadly used as a QC metric in HTS assays. ^{[ citation needed ]} The absolute sign in the Z-factor makes it inconvenient to derive its statistical inference mathematically.

To derive a better interpretable parameter for measuring the differentiation between two groups, Zhang XHD ^[1] proposed SSMD to evaluate the differentiation between a positive control and a negative control in HTS assays. SSMD has a probabilistic basis due to its strong link with d⁺-probability (i.e., the probability that the difference between two groups is positive). ^[2] To some extent, the d⁺-probability is equivalent to the well-established probabilistic index P(X > Y) which has been studied and applied in many areas. ^{[5] [6] [7] [8] [9]} Supported on its probabilistic basis, SSMD has been used for both quality control and hit selection in high-throughput screening. ^{[1] [2] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]}

Concept

Statistical parameter

As a statistical parameter, SSMD (denoted as ${\displaystyle \beta }$) is defined as the ratio of mean to standard deviation of the difference of two random values respectively from two groups. Assume that one group with random values has mean ${\displaystyle \mu _{1}}$ and variance ${\displaystyle \sigma _{1}^{2}}$ and another group has mean ${\displaystyle \mu _{2}}$ and variance ${\displaystyle \sigma _{2}^{2}}$. The covariance between the two groups is ${\displaystyle \sigma _{12}.}$ Then, the SSMD for the comparison of these two groups is defined as ^[1]

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{\sqrt {\sigma _{1}^{2}+\sigma _{2}^{2}-2\sigma _{12}}}}.}$

If the two groups are independent,

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{\sqrt {\sigma _{1}^{2}+\sigma _{2}^{2}}}}.}$

If the two independent groups have equal variances ${\displaystyle \sigma ^{2}}$,

${\displaystyle \beta ={\frac {\mu _{1}-\mu _{2}}{{\sqrt {2}}\sigma }}.}$

In the situation where the two groups are correlated, a commonly used strategy to avoid the calculation of ${\displaystyle \sigma _{12}}$ is first to obtain paired observations from the two groups and then to estimate SSMD based on the paired observations. Based on a paired difference ${\displaystyle D}$ with population mean ${\displaystyle \mu _{D}}$ and ${\displaystyle \sigma _{D}^{2}}$, SSMD is

${\displaystyle \beta ={\frac {\mu _{D}}{\sigma _{D}}}.}$

Statistical estimation

In the situation where the two groups are independent, Zhang XHD ^[1] derived the maximum-likelihood estimate (MLE) and method-of-moment (MM) estimate of SSMD. Assume that groups 1 and 2 have sample mean ${\displaystyle {\bar {X}}_{1},{\bar {X}}_{2}}$, and sample variances ${\displaystyle s_{1}^{2},s_{2}^{2}}$. The MM estimate of SSMD is then ^[1]

${\displaystyle {\hat {\beta }}={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{\sqrt {s_{1}^{2}+s_{2}^{2}}}}.}$

When the two groups have normal distributions with equal variance, the uniformly minimal variance unbiased estimate (UMVUE) of SSMD is, ^[10]

${\displaystyle {\hat {\beta }}={\frac {{\bar {X}}_{1}-{\bar {X}}_{2}}{\sqrt {{\frac {2}{K}}((n_{1}-1)s_{1}^{2}+(n_{2}-1)s_{2}^{2})}}},}$

where ${\displaystyle n_{1},n_{2}}$ are the sample sizes in the two groups and ${\displaystyle K\approx n_{1}+n_{2}-3.48}$. ^[3]

In the situation where the two groups are correlated, based on a paired difference with a sample size ${\displaystyle n}$, sample mean ${\displaystyle {\bar {D}}}$ and sample variance ${\displaystyle s_{D}^{2}}$, the MM estimate of SSMD is

${\displaystyle {\hat {\beta }}={\frac {\bar {D}}{s_{D}}}.}$

The UMVUE estimate of SSMD is ^[22]

${\displaystyle {\hat {\beta }}={\frac {\Gamma ({\frac {n-1}{2}})}{\Gamma ({\frac {n-2}{2}})}}{\sqrt {\frac {2}{n-1}}}{\frac {\bar {D}}{s_{D}}}.}$

SSMD looks similar to t-statistic and Cohen's d, but they are different with one another as illustrated in. ^[3]

Application in high-throughput screening assays

SSMD is the ratio of mean to the standard deviation of the difference between two groups. When the data is preprocessed using log-transformation as we normally do in HTS experiments, SSMD is the mean of log fold change divided by the standard deviation of log fold change with respect to a negative reference. In other words, SSMD is the average fold change (on the log scale) penalized by the variability of fold change (on the log scale) ^[23] . For quality control, one index for the quality of an HTS assay is the magnitude of difference between a positive control and a negative reference in an assay plate. For hit selection, the size of effects of a compound (i.e., a small molecule or an siRNA) is represented by the magnitude of difference between the compound and a negative reference. SSMD directly measures the magnitude of difference between two groups. Therefore, SSMD can be used for both quality control and hit selection in HTS experiments.

Quality control

The number of wells for the positive and negative controls in a plate in the 384-well or 1536-well platform is normally designed to be reasonably large . ^[24] Assume that the positive and negative controls in a plate have sample mean ${\displaystyle {\bar {X}}_{P},{\bar {X}}_{N}}$, sample variances ${\displaystyle s_{P}^{2},s_{N}^{2}}$, and sample sizes ${\displaystyle n_{P},n_{N}}$. Usually, the assumption that the controls have equal variance in a plate holds. In such a case, The SSMD for assessing quality in that plate is estimated as ^[10]

${\displaystyle {\hat {\beta }}={\frac {{\bar {X}}_{P}-{\bar {X}}_{N}}{\sqrt {{\frac {2}{K}}((n_{P}-1)s_{P}^{2}+(n_{N}-1)s_{N}^{2})}}},}$

where ${\displaystyle K\approx n_{P}+n_{N}-3.48}$. When the assumption of equal variance does not hold, the SSMD for assessing quality in that plate is estimated as ^[1]

${\displaystyle {\hat {\beta }}={\frac {{\bar {X}}_{P}-{\bar {X}}_{N}}{\sqrt {s_{P}^{2}+s_{N}^{2}}}}.}$

If there are clearly outliers in the controls, the SSMD can be estimated as ^[23]

${\displaystyle {\hat {\beta }}={\frac {{\tilde {X}}_{P}-{\tilde {X}}_{N}}{1.4826{\sqrt {{\tilde {s}}_{P}^{2}+{\tilde {s}}_{N}^{2}}}}},}$

where ${\displaystyle {\tilde {X}}_{P},{\tilde {X}}_{N},{\tilde {s}}_{P},{\tilde {s}}_{N}}$ are the medians and median absolute deviations in the positive and negative controls, respectively.

The Z-factor based QC criterion is popularly used in HTS assays. However, it has been demonstrated that this QC criterion is most suitable for an assay with very or extremely strong positive controls. ^[10] In an RNAi HTS assay, a strong or moderate positive control is usually more instructive than a very or extremely strong positive control because the effectiveness of this control is more similar to the hits of interest. In addition, the positive controls in the two HTS experiments theoretically have different sizes of effects. Consequently, the QC thresholds for the moderate control should be different from those for the strong control in these two experiments. Furthermore, it is common that two or more positive controls are adopted in a single experiment. ^[11] Applying the same Z-factor-based QC criteria to both controls leads to inconsistent results as illustrated in the literatures. ^{[10] [11]}

The SSMD-based QC criteria listed in the following table ^[20] take into account the effect size of a positive control in an HTS assay where the positive control (such as an inhibition control) theoretically has values less than the negative reference.

Quality Type	A: Moderate Control	B: Strong Control	C: Very Strong Control	D: Extremely Strong Control
Excellent	${\displaystyle \beta \leq -2}$	${\displaystyle \beta \leq -3}$	${\displaystyle \beta \leq -5}$	${\displaystyle \beta \leq -7}$
Good	${\displaystyle -2<\beta \leq -1}$	${\displaystyle -3<\beta \leq -2}$	${\displaystyle -5<\beta \leq -3}$	${\displaystyle -7<\beta \leq -5}$
Inferior	${\displaystyle -1<\beta \leq -0.5}$	${\displaystyle -2<\beta \leq -1}$	${\displaystyle -3<\beta \leq -2}$	${\displaystyle -5<\beta \leq -3}$
Poor	${\displaystyle \beta >-0.5}$	${\displaystyle \beta >-1}$	${\displaystyle \beta >-2}$	${\displaystyle \beta >-3}$

In application, if the effect size of a positive control is known biologically, adopt the corresponding criterion based on this table. Otherwise, the following strategy should help to determine which QC criterion should be applied: (i) in many small molecule HTS assay with one positive control, usually criterion D (and occasionally criterion C) should be adopted because this control usually has very or extremely strong effects; (ii) for RNAi HTS assays in which cell viability is the measured response, criterion D should be adopted for the controls without cells (namely, the wells with no cells added) or background controls; (iii) in a viral assay in which the amount of viruses in host cells is the interest, criterion C is usually used, and criterion D is occasionally used for the positive control consisting of siRNA from the virus. ^[20]

Similar SSMD-based QC criteria can be constructed for an HTS assay where the positive control (such as an activation control) theoretically has values greater than the negative reference. More details about how to apply SSMD-based QC criteria in HTS experiments can be found in a book. ^[20]

Hit selection

In an HTS assay, one primary goal is to select compounds with a desired size of inhibition or activation effect. The size of the compound effect is represented by the magnitude of difference between a test compound and a negative reference group with no specific inhibition/activation effects. A compound with a desired size of effects in an HTS screen is called a hit. The process of selecting hits is called hit selection. There are two main strategies of selecting hits with large effects. ^[20] One is to use certain metric(s) to rank and/or classify the compounds by their effects and then to select the largest number of potent compounds that is practical for validation assays. ^{[17] [19] [22]} The other strategy is to test whether a compound has effects strong enough to reach a pre-set level. In this strategy, false-negative rates (FNRs) and/or false-positive rates (FPRs) must be controlled. ^{[14] [15] [16] [25] [26]}

SSMD can not only rank the size of effects but also classify effects as shown in the following table based on the population value (${\displaystyle \beta }$) of SSMD. ^{[20] [27]}

Effect subtype	Thresholds for negative SSMD	Thresholds for positive SSMD
Extremely strong	${\displaystyle \beta \leq -5}$	${\displaystyle \beta \geq 5}$
Very strong	${\displaystyle -5<\beta \leq -3}$	${\displaystyle 5>\beta \geq 3}$
Strong	${\displaystyle -3<\beta \leq -2}$	${\displaystyle 3>\beta \geq 2}$
Fairly strong	${\displaystyle -2<\beta \leq -1.645}$	${\displaystyle 2>\beta \geq 1.645}$
Moderate	${\displaystyle -1.645<\beta \leq -1.28}$	${\displaystyle 1.645>\beta \geq 1.28}$
Fairly moderate	${\displaystyle -1.28<\beta \leq -1}$	${\displaystyle 1.28>\beta \geq 1}$
Fairly weak	${\displaystyle -1<\beta \leq -0.75}$	${\displaystyle 1>\beta \geq 0.75}$
Weak	${\displaystyle -0.75<\beta <-0.5}$	${\displaystyle 0.75>\beta >0.5}$
Very weak	${\displaystyle -0.5\leq \beta <-0.25}$	${\displaystyle 0.5\geq \beta >0.25}$
Extremely weak	${\displaystyle -0.25\leq \beta <0}$	${\displaystyle 0.25\geq \beta >0}$
No effect	${\displaystyle \beta =0}$

The estimation of SSMD for screens without replicates differs from that for screens with replicates. ^{[20] [23]}

In a primary screen without replicates, assuming the measured value (usually on the log scale) in a well for a tested compound is ${\displaystyle X_{i}}$ and the negative reference in that plate has sample size ${\displaystyle n_{N}}$, sample mean ${\displaystyle {\bar {X}}_{N}}$, median ${\displaystyle {\tilde {X}}_{N}}$, standard deviation ${\displaystyle s_{N}}$ and median absolute deviation ${\displaystyle {\tilde {s}}_{N}}$, the SSMD for this compound is estimated as ^{[20] [23]}

${\displaystyle {\text{SSMD}}={\frac {X_{i}-{\bar {X}}_{N}}{s_{N}{\sqrt {2(n_{N}-1)/K}}}},}$

where ${\displaystyle K\approx n_{N}-2.48}$. When there are outliers in an assay which is usually common in HTS experiments, a robust version of SSMD ^[23] can be obtained using

${\displaystyle {\text{SSMD*}}={\frac {X_{i}-{\tilde {X}}_{N}}{1.4826{\tilde {s}}_{N}{\sqrt {2(n_{N}-1)/K}}}}}$

In a confirmatory or primary screen with replicates, for the i-th test compound with ${\displaystyle n}$ replicates, we calculate the paired difference between the measured value (usually on the log scale) of the compound and the median value of a negative control in a plate, then obtain the mean ${\displaystyle {\bar {d}}_{i}}$ and variance ${\displaystyle s_{i}^{2}}$ of the paired difference across replicates. The SSMD for this compound is estimated as ^[20]

${\displaystyle {\text{SSMD}}={\frac {\Gamma ({\frac {n-1}{2}})}{\Gamma ({\frac {n-2}{2}})}}{\sqrt {\frac {2}{n-1}}}{\frac {{\bar {d}}_{i}}{s_{i}}}}$

In many cases, scientists may use both SSMD and average fold change for hit selection in HTS experiments. The dual-flashlight plot ^[28] can display both average fold change and SSMD for all test compounds in an assay and help to integrate both of them to select hits in HTS experiments ^[29] . The use of SSMD for hit selection in HTS experiments is illustrated step-by-step in ^[23]

See also

• Effect size
• high-throughput screening
• Z-factor
• Hit selection
• SMCV
• c⁺-probability
• Contrast variable
• Dual-flashlight plot

Further reading

• Zhang XHD (2011) "Optimal High-Throughput Screening: Practical Experimental Design and Data Analysis for Genome-scale RNAi Research, Cambridge University Press"

References

1. Zhang XHD (2007). "A pair of new statistical parameters for quality control in RNA interference high-throughput screening assays". Genomics. 89 (4): 552–61. doi: 10.1016/j.ygeno.2006.12.014 . PMID   17276655.
2. Zhang XHD (2007). "A new method with flexible and balanced control of false negatives and false positives for hit selection in RNA interference high-throughput screening assays". Journal of Biomolecular Screening. 12 (5): 645–55. doi: 10.1177/1087057107300645 . PMID   17517904.
3. Zhang XHD (2010). "Strictly standardized mean difference, standardized mean difference and classical t-test for the comparison of two groups". Statistics in Biopharmaceutical Research. 2 (2): 292–99. doi:10.1198/sbr.2009.0074. S2CID   119825625.
4. Zhang JH, Chung TDY, Oldenburg KR (1999). "A simple statistical parameter for use in evaluation and validation of high throughput screening assays". Journal of Biomolecular Screening. 4 (2): 67–73. doi: 10.1177/108705719900400206 . PMID   10838414. S2CID   36577200.
5. Owen DB, Graswell KJ, Hanson DL (1964). "Nonparametric upper confidence bounds for P(Y < X) and confidence limits for P(Y < X) when X and Y are normal". Journal of the American Statistical Association. 59 (307): 906–24. doi:10.2307/2283110. hdl: 2027/mdp.39015094992651 . JSTOR   2283110.
6. Church JD, Harris B (1970). "The estimation of reliability from stress-strength relationships". Technometrics. 12: 49–54. doi:10.1080/00401706.1970.10488633.
7. Downton F (1973). "The estimation of Pr(Y < X) in normal case". Technometrics. 15 (3): 551–8. doi:10.2307/1266860. JSTOR   1266860.
8. Reiser B, Guttman I (1986). "Statistical inference for of Pr(Y-less-thaqn-X) - normal case". Technometrics. 28 (3): 253–7. doi:10.2307/1269081. JSTOR   1269081.
9. Acion L, Peterson JJ, Temple S, Arndt S (2006). "Probabilistic index: an intuitive non-parametric approach to measuring the size of treatment effects". Statistics in Medicine. 25 (4): 591–602. doi:10.1002/sim.2256. PMID   16143965.
10. Zhang XHD (2008). "Novel analytic criteria and effective plate designs for quality control in genome-wide RNAi screens". Journal of Biomolecular Screening. 13 (5): 363–77. doi: 10.1177/1087057108317062 . PMID   18567841. S2CID   12688742.
11. Zhang XHD, Espeseth AS, Johnson E, Chin J, Gates A, Mitnaul L, Marine SD, Tian J, Stec EM, Kunapuli P, Holder DJ, Heyse JF, Stulovici B, Ferrer M (2008). "Integrating experimental and analytic approaches to improve data quality in genome-wide RNAi screens". Journal of Biomolecular Screening. 13 (5): 378–89. doi: 10.1177/1087057108317145 . PMID   18480473. S2CID   22679273.
12. Zhang XHD, Ferrer M, Espeseth AS, Marine SD, Stec EM, Crackower MA, Holder DJ, Heyse JF, Strulovici B (2007). "The use of strictly standardized mean difference for hit selection in primary RNA interference high-throughput screening experiments". Journal of Biomolecular Screening. 12 (4): 645–55. doi: 10.1177/1087057107300646 . PMID   17435171. S2CID   7542230.
13. Quon K, Kassner PD (2009). "RNA interference screening for the discovery of oncology targets". Expert Opinion on Therapeutic Targets. 13 (9): 1027–35. doi:10.1517/14728220903179338. PMID   19650760. S2CID   10714162.
14. Zhang XHD (2010). "An effective method controlling false discoveries and false non-discoveries in genome-scale RNAi screens". Journal of Biomolecular Screening. 15 (9): 1116–22. doi: 10.1177/1087057110381783 . PMID   20855561.
15. Zhang XHD, Lacson R, Yang R, Marine SD, McCampbell A, Toolan DM, Hare TR, Kajdas J, Berger JP, Holder DJ, Heyse JF, Ferrer M (2010). "The use of SSMD-based false discovery and false non-discovery rates in genome-scale RNAi screens". Journal of Biomolecular Screening. 15 (9): 1123–31. doi: 10.1177/1087057110381919 . PMID   20852024.
16. Zhang XHD, Marine SD, Ferrer M (2009). "Error rates and power in genome-scale RNAi screens". Journal of Biomolecular Screening. 14 (3): 230–38. doi: 10.1177/1087057109331475 . PMID   19211781.
17. Birmingham A, Selfors LM, Forster T, Wrobel D, Kennedy CJ, Shanks E, Santoyo-Lopez J, Dunican DJ, Long A, Kelleher D, Smith Q, Beijersbergen RL, Ghazal P, Shamu CE (2009). "Statistical methods for analysis of high-throughput RNA interference screens". Nature Methods. 6 (8): 569–75. doi:10.1038/nmeth.1351. PMC   2789971 . PMID   19644458.
18. Klinghoffer RA, Frazier J, Annis J, Berndt JD, Roberts BS, Arthur WT, Lacson R, Zhang XHD, Ferrer M, Moon RT, Cleary MA (2010). Bereswill S (ed.). "A lentivirus-mediated genetic screen identifies dihydrofolate reductase (DHFR) as a modulator of beta-catenin/GSK3 signaling". PLOS ONE. 4 (9): e6892. doi: 10.1371/journal.pone.0006892 . PMC   2731218 . PMID   19727391.
19. Malo N, Hanley JA, Carlile G, Liu J, Pelletier J, Thomas D, Nadon R (2010). "Experimental design and statistical methods for improved hit detection in high-throughput screening". Journal of Biomolecular Screening. 15 (8): 990–1000. doi: 10.1177/1087057110377497 . PMID   20817887. S2CID   41358896.
20. Zhang XHD (2011). Optimal High-Throughput Screening: Practical Experimental Design and Data Analysis for Genome-scale RNAi Research. Cambridge University Press. ISBN   978-0-521-73444-8.
21. Zhou HL, Xu M, Huang Q, Gates AT, Zhang XD, Castle JC, Stec E, Ferrer M, Strulovici B, Hazuda DJ, Espeseth AS (2008). "Genome-scale RNAi screen for host factors required for HIV replication". Cell Host & Microbe. 4 (5): 495–504. doi: 10.1016/j.chom.2008.10.004 . PMID   18976975.
22. Zhang XHD (2010). "Genome-wide screens for effective siRNAs through assessing the size of siRNA effects". BMC Research Notes. 1: 33. doi: 10.1186/1756-0500-1-33 . PMC   2526086 . PMID   18710486.
23. Zhang XHD (2011). "Illustration of SSMD, z Score, SSMD*, z* Score, and t Statistic for Hit Selection in RNAi High-Throughput Screens". Journal of Biomolecular Screening. 16 (7): 775–85. doi: 10.1177/1087057111405851 . PMID   21515799.
24. Zhang XHD, Heyse JF (2009). "Determination of sample size in genome-scale RNAi screens". Bioinformatics. 25 (7): 841–44. doi: 10.1093/bioinformatics/btp082 . PMID   19223447.
25. Malo N, Hanley JA, Cerquozzi S, Pelletier J, Nadon R (2006). "Statistical practice in high-throughput screening data analysis". Nature Biotechnology. 24 (2): 167–75. doi:10.1038/nbt1186. PMID   16465162. S2CID   6158255.
26. Zhang XHD, Kuan PF, Ferrer M, Shu X, Liu YC, Gates AT, Kunapuli P, Stec EM, Xu M, Marine SD, Holder DJ, Stulovici B, Heyse JF, Espeseth AS (2009). "Hit selection with false discovery rate control in genome-scale RNAi screens". Nucleic Acids Research. 36 (14): 4667–79. doi:10.1093/nar/gkn435. PMC   2504311 . PMID   18628291.
27. Zhang XHD (2009). "A method for effectively comparing gene effects in multiple conditions in RNAi and expression-profiling research". Pharmacogenomics. 10 (3): 345–58. doi:10.2217/14622416.10.3.345. PMID   20397965.
28. Zhang XHD (2010). "Assessing the size of gene or RNAi effects in multifactor high-throughput experiments". Pharmacogenomics. 11 (2): 199–213. doi:10.2217/PGS.09.136. PMID   20136359.
29. Zhao WQ, Santini F, Breese R, Ross D, Zhang XD, Stone DJ, Ferrer M, Townsend M, Wolfe AL, Seager MA, Kinney GG, Shughrue PJ, Ray WJ (2010). "Inhibition of calcineurin-mediated endocytosis and alpha-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid (AMPA) receptors prevents amyloid beta oligomer-induced synaptic disruption". Journal of Biological Chemistry. 285 (10): 7619–32. doi: 10.1074/jbc.M109.057182 . PMC   2844209 . PMID   20032460.