Two-proportion Z-test

The two-proportion Z-test or two-sample proportion Z-test is a statistical method used to determine whether the difference between the proportions of two groups, coming from a binomial distribution is statistically significant. ^[1] This approach relies on the assumption that the sample proportions follow a normal distribution under the Central Limit Theorem, allowing the construction of a z-test for hypothesis testing and confidence interval estimation. ^[2] It is used in various fields to compare success rates, response rates, or other proportions across different groups.

Hypothesis test

The z-test for comparing two proportions is a frequentist statistical hypothesis test used to evaluate whether two independent samples have different population proportions for a binary outcome. Under mild regularity conditions (sufficiently large sample sizes and independent sampling), the sample proportions (which is the average of observations coming from a Bernoulli distribution) are approximately normally distributed under the central limit theorem, which permits using a z-statistic constructed from the difference of sample proportions and an estimated standard error. ^[2]

The test involves two competing hypotheses:

• null hypothesis (H₀): The proportions in the two populations are equal, i.e., ${\displaystyle p_{1}=p_{2}}$.
• alternative hypothesis (H₁): The proportions in the two populations are not equal, i.e., ${\displaystyle p_{1}\neq p_{2}}$ (two-tailed) or ${\displaystyle p_{1}>p_{2}}$ / ${\displaystyle p_{1}<p_{2}}$ (one-tailed).

The z-statistic for comparing two proportions is computed using: ^{[3] [4] [5] [2] : 10.6.2}$${\displaystyle z={\frac {{\hat {p}}_{1}-{\hat {p}}_{2}}{\sqrt {{\hat {p}}(1-{\hat {p}})\left({\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}\right)}}},}$$where ${\displaystyle {\hat {p}}_{1}}$ and ${\displaystyle {\hat {p}}_{2}}$ are the sample proportion in the first and second sample, ${\displaystyle n_{1}}$ and ${\displaystyle n_{2}}$ are the size of the first and second sample, respectively, ${\displaystyle {\hat {p}}}$ is pooled proportion, calculated as ${\displaystyle {\hat {p}}={\frac {x_{1}+x_{2}}{n_{1}+n_{2}}}}$, where ${\displaystyle x_{1}}$ and ${\displaystyle x_{2}}$ are the counts of successes in the two samples The pooled proportion is used to estimate the shared probability of success under the null hypothesis, and the standard error accounts for variability across the two samples.

The z-test determines statistical significance by comparing the calculated z-statistic to a critical value. E.g., for a significance level of ${\displaystyle \alpha =0.05}$ we reject the null hypothesis if ${\displaystyle |z|>1.96}$ (for a two-tailed test). Or, alternatively, by computing the p-value and rejecting the null hypothesis if ${\displaystyle p<\alpha }$.

Confidence interval

The confidence interval for the difference between two proportions, based on the definitions above, is: ^{[5] [2] : 10.6.3}$${\displaystyle ({\hat {p}}_{1}-{\hat {p}}_{2})\pm z_{\alpha /2}{\sqrt {{\frac {{\hat {p}}_{1}(1-{\hat {p}}_{1})}{n_{1}}}+{\frac {{\hat {p}}_{2}(1-{\hat {p}}_{2})}{n_{2}}}}},}$$where ${\displaystyle z_{\alpha /2}}$ is the critical value of the standard normal distribution (e.g., 1.96 for a 95% confidence level).

This interval provides a range of plausible values for the true difference between population proportions.

Notice how the variance estimation is different between the hypothesis testing and the confidence intervals. The first uses a pooled variance (based on the null hypothesis), while the second has to estimate the variance using each sample separately (so as to allow for the confidence interval to accommodate a range of differences in proportions). This difference may lead to slightly different results if using the confidence interval as an alternative to the hypothesis testing method.

Sample size determination and Minimum detectable effect

Sample size determination is the act of choosing the number of observations to include in each group for running the statistical test. For the Two-proportion Z-test, this is closely-related with deciding on the minimum detectable effect.

For finding the required sample size (given some effect size ${\displaystyle |p_{1}-p_{2}|}$, power ${\displaystyle \pi =(1-\beta )}$, and type I error ${\displaystyle \alpha }$), we define that ${\displaystyle n_{1}=\kappa n_{2}}$, (when κ = 1, equal sample size is assumed for each group), then: ^{[6] [7]}

${\displaystyle n_{2}={\frac {(Z_{\alpha /2}+Z_{\beta })^{2}}{(p_{1}-p_{2})^{2}}}\times \left({\frac {p_{1}(1-p_{1})}{\kappa }}+p_{2}(1-p_{2})\right)^{2}}$

The minimum detectable effect or MDE is the smallest difference between two proportions (${\displaystyle p_{1}}$ and ${\displaystyle p_{2}}$) that a statistical test can detect for a chosen type I error level (${\displaystyle \alpha }$), statistical power (${\displaystyle 1-\beta }$), and sample sizes (${\displaystyle n_{1}}$ and ${\displaystyle n_{2}}$). It is commonly used in study design to determine whether the sample sizes allows for a test with sufficient sensitivity to detect meaningful differences.

The MDE for when using the (two-sided) z-test formula for comparing two proportions, incorporating critical values for ${\displaystyle \alpha }$ and ${\displaystyle 1-\beta }$, and the standard errors of the proportions: ^{[8] [9]} $${\displaystyle {\text{MDE}}=|p_{1}-p_{2}|=z_{1-\alpha /2}{\sqrt {p_{0}(1-p_{0})\left({\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}\right)}}+z_{1-\beta }{\sqrt {{\frac {p_{1}(1-p_{1})}{n_{1}}}+{\frac {p_{2}(1-p_{2})}{n_{2}}}}},}$$where ${\displaystyle z_{1-\alpha /2}}$ is critical value for the significance level, ${\displaystyle z_{1-\beta }}$ is quantile for the desired power, and ${\displaystyle p_{0}=p_{1}=p_{2}}$ is when assuming the null is correct.

The MDE depends on the sample sizes, baseline proportions (${\displaystyle p_{1},p_{2}}$), and test parameters. When the baseline proportions are not known, they need to be assumed or roughly estimated from a small study. Larger samples or smaller power requirements leads to a smaller MDE, making the test more sensitive to smaller differences. Researchers may use the MDE to assess the feasibility of detecting meaningful differences before conducting a study.

[Proof]

The Minimal Detectable Effect (MDE) is the smallest difference, denoted as ${\displaystyle \Delta =|p_{1}-p_{2}|}$, that satisfies two essential criteria in hypothesis testing:

1. The null hypothesis (${\displaystyle H_{0}:p_{1}=p_{2}}$) is rejected at the specified significance level (${\displaystyle \alpha }$).
2. Statistical power (${\displaystyle 1-\beta }$) is achieved under the alternative hypothesis (${\displaystyle H_{a}:p_{1}\neq p_{2}}$).

Given that the distribution is normal under the null and the alternative hypothesis, for the two criteria to happen, it is required that the distance of ${\displaystyle |p_{1}-p_{2}|}$ will be such that the critical value for rejecting the null (${\displaystyle X_{\text{critical}}}$) is exactly in the location in which the probability of exceeding this value, under the null, is (${\displaystyle \alpha }$), and also that the probability of exceeding this value, under the alternative, is ${\displaystyle 1-\beta }$.

The first criterion establishes the critical value required to reject the null hypothesis. The second criterion specifies how far the alternative distribution must be from ${\displaystyle X_{\text{critical}}}$ to ensure that the probability of exceeding it under the alternative hypothesis is at least ${\displaystyle 1-\beta }$. ^{[10] [11]}

Condition 1: Rejecting ${\displaystyle H_{0}}$

Under the null hypothesis, the test statistic is based on the pooled standard error (${\displaystyle {\text{SE}}_{\text{null}}}$): ${\displaystyle Z_{\text{test}}={\frac {|p_{1}-p_{2}|}{{\text{SE}}_{\text{null}}}},\quad {\text{where }}{\text{SE}}_{\text{null}}={\sqrt {p_{0}(1-p_{0})\left({\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}\right)}}.}$

${\displaystyle p_{0}}$ might be estimated (as described above).

To reject ${\displaystyle H_{0}}$, the observed difference must exceed the critical threshold (${\displaystyle Z_{\text{critical}}=z_{\alpha /2}}$) after properly inflating it to the SE: ${\displaystyle |p_{1}-p_{2}|\geq X_{critical}=z_{\alpha /2}\cdot {\text{SE}}_{\text{null}}}$

If the MDE is defined solely as ${\displaystyle MDE=z_{\alpha /2}\cdot {\text{SE}}_{\text{null}}}$, the statistical power would be only 50% because the alternative distribution is symmetric about the threshold. To achieve a higher power level, an additional component is required in the MDE calculation.

Condition 2: Achieving power ${\displaystyle 1-\beta }$

Under the alternative hypothesis, the standard error is (${\displaystyle {\text{SE}}_{\text{alt}}={\sqrt {{\frac {p_{1}(1-p_{1})}{n_{1}}}+{\frac {p_{2}(1-p_{2})}{n_{2}}}}}}$). It means that if the alternative distribution was centered around some value (e.g., ${\displaystyle X_{\text{critical}}}$), then the minimal ${\displaystyle |p_{1}-p_{2}|}$ must be at least larger than ${\displaystyle z_{\alpha /2}\cdot {\text{SE}}_{\text{null}}}$ to ensure that the probability of detecting the difference under the alternative hypothesis is at least ${\displaystyle 1-\beta }$.

Combining conditions

To meet both conditions, the total detectable difference incorporates components from both the null and alternative distributions. The MDE is defined as: ${\displaystyle {\text{MDE}}=z_{1-\alpha /2}\cdot {\text{SE}}_{\text{null}}+z_{1-\beta }\cdot {\text{SE}}_{\text{alt}}.}$

By summing the critical thresholds from the null and adding to it the relevant quantile from the alternative distributions, the MDE ensures the test satisfies the dual requirements of rejecting ${\displaystyle H_{0}}$ at significance level ${\displaystyle \alpha }$ and achieving statistical power of at least ${\displaystyle 1-\beta }$.

Assumptions and conditions

To ensure valid results, the following assumptions must be met:

1. Independent random samples: The samples must be drawn independently from the populations of interest.
2. Large sample sizes: Typically, ${\displaystyle n_{1}+n_{2}}$ should exceed 20. ^{[12] : 48}
3. Success or failure condition: ^{[2] : 10.6.1}
   1. ${\displaystyle n_{1}{\hat {p}}_{1}>5}$ and ${\displaystyle n_{1}(1-{\hat {p}}_{1})>5}$
   2. ${\displaystyle n_{2}{\hat {p}}_{2}>5}$ and ${\displaystyle n_{2}(1-{\hat {p}}_{2})>5}$

The z-test is most reliable when sample sizes are large, and all assumptions are satisfied.

Relation to other statistical methods

Using the z-test confidence intervals for hypothesis testing would give the same results as the chi-squared test for a two-by-two contingency table. ^{[13] : 216–7  [14] : 875} Fisher’s exact test is more suitable for when the sample sizes are small.

Treatment of 2-by-2 contingency table has been investigated as early as the 19th century ^[15] , with further work during the 20th century. ^[16]

Notice that:

• When one or more cell counts are small (e.g. below 5 ^{[12] : 48}), prefer exact tests (e.g., Fisher's exact test) or exact confidence intervals.
• For paired or matched binary data use McNemar's test rather than the two-sample z-test.
• The choice between pooled and unpooled variance matters: pooled variance is appropriate for hypothesis testing of equality (${\displaystyle H_{0}:p_{1}=p_{2}}$), whereas the unpooled variance is used for confidence intervals.
• Multiple testing, selection effects, and nonrandom sampling can invalidate p-values and CIs; these design issues should be addressed in the study methods.

Example

Suppose group 1 has 120 successes out of 1000 trials (${\displaystyle {\hat {p}}_{1}=0.12}$) and group 2 has 150 successes out of 1000 trials (${\displaystyle {\hat {p}}_{2}=0.15}$). The pooled proportion is ${\displaystyle {\hat {p}}=(120+150)/(1000+1000)=0.135}$. The pooled standard error is$${\displaystyle {\text{SE}}_{\text{pooled}}={\sqrt {0.135\times 0.865\times \left({\dfrac {1}{1000}}+{\dfrac {1}{1000}}\right)}}\approx 0.01529.}$$

The z-statistic is$${\displaystyle z={\dfrac {0.12-0.15}{0.01529}}\approx -1.96,}$$giving a two-sided p-value of about 0.0497 (just under 0.05). An approximate 95% confidence interval for the difference using the unpooled standard error is$${\displaystyle (0.12-0.15)\pm 1.96{\sqrt {{\dfrac {0.12\times 0.88}{1000}}+{\dfrac {0.15\times 0.85}{1000}}}}\approx (-0.0599,\ -0.0001).}$$Because the 95% CI (just barely) excludes 0 and the p-value is ≈0.0497, the difference is statistically significant at the 5% level by the usual large-sample criteria (but is borderline; conclusions should account for study context and multiple testing if applicable).

Software implementation

Implementations are available in many statistical environments. See below for implementation details in some popular languages. Other implementations also exists for SPSS ^[17] , SAS ^[18] , and Minitab ^[5] .

R

Use prop.test() with continuity correction disabled:

prop.test(x=c(120,150),n=c(1000,1000),correct=FALSE)

Output includes z-test equivalent results: chi-squared statistic, p-value, and confidence interval:

 2-sample test for equality of proportions without continuity correction  data:  c(120, 150) out of c(1000, 1000) X-squared = 3.8536, df = 1, p-value = 0.04964 alternative hypothesis: two.sided 95 percent confidence interval:  -5.992397e-02 -7.602882e-05 sample estimates: prop 1 prop 2    0.12   0.15 

Python

Use proportions_ztest from statsmodels:

fromstatsmodels.stats.proportionimportproportions_ztestz,p=proportions_ztest([120,150],[1000,1000],0)# For CI: from statsmodels.stats.proportion import proportions_diff_confint_indep

SQL

(This is using Presto flavor of SQL)

-- Calculate group sizes and accuracy per groupWITHgroup_statsAS(SELECT-- Number of samples in each groupCOUNT_IF(ab_group='test')ASn_test,COUNT_IF(ab_group='control')ASn_control,-- Proportion correct (accuracy) in each groupCOUNT_IF(ab_group='test'ANDis_success=1)*1.0/NULLIF(COUNT_IF(ab_group='test'),0)ASp_test,COUNT_IF(ab_group='control'ANDis_success=1)*1.0/NULLIF(COUNT_IF(ab_group='control'),0)ASp_controlFROMsome_table)SELECT-- Sample sizesn_test,n_control,-- Accuracy (proportion correct) per groupROUND(p_test,3)ASp_test,ROUND(p_control,3)ASp_control,-- Difference in accuracy between groupsROUND(p_test-p_control,3)ASdiff_p,-- Standard error of the difference in proportionsROUND(SQRT((p_test*(1-p_test)/n_test)+(p_control*(1-p_control)/n_control)),3)ASse_diff,-- Total sample sizen_test+n_controlASn_total,-- Pooled proportion (overall accuracy)ROUND((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0),3)ASp_pooled,-- 95% Confidence Interval for the difference in proportionsROUND((p_test-p_control)-1.96*SQRT((p_test*(1-p_test)/n_test)+(p_control*(1-p_control)/n_control)),3)ASdiff_p_ci_lower,ROUND((p_test-p_control)+1.96*SQRT((p_test*(1-p_test)/n_test)+(p_control*(1-p_control)/n_control)),3)ASdiff_p_ci_upper,-- Ratio of test to control accuracyROUND(p_test/NULLIF(p_control,0),3)ASaccuracy_ratio,-- Two-sided p-value for difference in proportions (z-test)ROUND(2*(1-NORMAL_CDF(0,1,ABS(p_test-p_control)/SQRT((1-((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0)))*((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0))*(1.0/n_test+1.0/n_control)))),3)ASdiff_p_p_value,-- Statistical significance at 95% confidenceABS(p_test-p_control)/SQRT((1-((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0)))*((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0))*(1.0/n_test+1.0/n_control))>1.96ASis_diff_stat_sig,-- Minimal Detectable Effect (MDE) at 95% confidence, 80% powerROUND((1.96*SQRT(((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0))*(1-((n_test*p_test+n_control*p_control)/NULLIF(n_test+n_control,0)))*(1.0/n_test+1.0/n_control))+0.84*SQRT((p_test*(1-p_test)/n_test)+(p_control*(1-p_control)/n_control))),3)ASminimal_detectable_effectFROMgroup_stats

Output would be something like this:

A/B Test Summary Statistics

n_test	n_control	p_test	p_control	diff_p	se_diff	p_pooled	diff_p_ci_lower	diff_p_ci_upper	p_value	is_stat_sig	accuracy_ratio	mde
1000	1000	0.120	0.150	-0.030	0.015	0.135	-0.0599	0.0001	0.050	TRUE	0.800	0.043

See also

• Two-sample hypothesis testing
• A/B testing
• Pearson's chi-squared test (2×2 tables)
• McNemar's test

References

1. Hypothesis Test: Difference Between Proportions
2. Su, Wanhua (2024). "Introduction to Applied Statistics (10.6 Inferences for Two Population Proportions)". MacEwan University Open Textbooks.
3. "Introductory Statistics 2e — §10.3 Comparing Two Independent Population Proportions". OpenStax. 2023.
4. How can we determine whether two processes produce the same proportion of defectives?
5. Kiernan, D. (2014). "Natural Resources Biometrics — Chapter 4: Inferences about the Differences of Two Populations --- Section 4". Milne Publishing (SUNY Geneseo).
6. Kim HY. Statistical notes for clinical researchers: Sample size calculation 2. Comparison of two independent proportions. Restor Dent Endod. 2016 May;41(2):154-6. doi: 10.5395/rde.2016.41.2.154. Epub 2016 Apr 5. PMID: 27200285; PMCID: PMC4868880.
7. Wang, H. and Chow, S.-C. 2007. Sample Size Calculation for Comparing Proportions. Wiley Encyclopedia of Clinical Trials.
8. COOLSerdash (https://stats.stackexchange.com/users/21054/coolserdash), Two proportion sample size calculation, URL (version: 2023-04-14): https://stats.stackexchange.com/q/612894
9. Chow S-C, Shao J, Wang H, Lokhnygina Y (2018): Sample size calculations in clinical research. 3rd ed. CRC Press.
10. Calculating Sample Sizes for A/B Tests
11. Power, minimal detectable effect, and bucket size estimation in A/B tests (has some nice figures to illustrate the tradeoffs)
12. VanVoorhis, CR Wilson, and Betsy L. Morgan. "Understanding power and rules of thumb for determining sample sizes." Tutorials in quantitative methods for psychology 3.2 (2007): 43-50.
13. Confidence Intervals for the Difference Between Two Proportions
14. Newcombe, R. G. 1998. 'Interval Estimation for the Difference Between Independent Proportions: Comparison of Eleven Methods.' Statistics in Medicine, 17, pp. 873-890.
15. Stigler, Stephen M. (2002). "The missing early history of contingency tables". Annales de la Faculté des sciences de Toulouse : Mathématiques. 11 (4): 563–573.
16. Hitchcock, David B. (2009). "Yates and contingency tables: 75 years later" (PDF). Electronic Journal for History of Probability & Statistics.
17. Z-Test for 2 Independent Proportions – Quick Tutorial
18. Usage Note 22561: Testing the equality of two or more proportions from independent samples

External links

• Two Sample Independent Proportions Test Calculator
• Sample size / power online calculators:
  • Comparing Two Proportions – Sample Size
  • Inference for Proportions: Comparing Two Independent Samples