Two-Proportion Z-Test Calculator

The two-proportion z-test asks: given two samples from two groups, is the difference in their proportions statistically significant, or could it be explained by random sampling variation?

Setup: group 1 has x1 successes out of n1 trials (proportion p1 = x1/n1). Group 2 has x2 successes out of n2 trials (proportion p2 = x2/n2).

The pooled proportion (assuming the null hypothesis H0: p1 = p2 is true): p_pool = (x1 + x2) / (n1 + n2)

The z-statistic: z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))

A large |z| means the observed difference is unlikely under H0. Common critical values: |z| > 1.645 for p < 0.10 (one-tailed), |z| > 1.96 for p < 0.05, |z| > 2.576 for p < 0.01.

This calculator uses a normal approximation. It is valid when both n1p_pool and n1(1-p_pool) are at least 5 (and similarly for group 2). For small samples, use Fisher’s exact test instead.

The 95% confidence interval for (p1 - p2) uses the unpooled standard error: SE = sqrt(p1*(1-p1)/n1 + p2*(1-p2)/n2) CI: (p1 - p2) +/- 1.96 * SE

If this interval excludes zero, the difference is significant at the 5% level, which is consistent with the pooled test without being identical to it.

Two different standard errors, and why

Notice the test and the interval use different formulas for the standard error. That is deliberate, not an inconsistency. The test assumes the null hypothesis is true, and under that assumption both groups share one proportion, so pooling the data gives a better estimate of the common variance. The confidence interval makes no such assumption: it is trying to describe the difference itself, so it takes each group’s variance on its own terms. In borderline cases the two can disagree, with a p-value just under 0.05 alongside an interval that just barely contains zero. Neither is a mistake, and the interval is usually the one to trust for reporting.

Read the interval, not the verdict

The most common misuse of this test is treating “not significant” as “no difference”. A p-value of 0.30 with 8 observations per group means almost nothing; the same p-value with 8,000 per group is decent evidence the effect is small. The interval carries that information and the significance label throws it away. If your interval runs from minus 3 to plus 24 percentage points, the honest summary is that you have not measured this precisely enough to say anything useful, not that the groups match.

Sample size is the lever. Detecting a 5-point difference between proportions near 50% at the usual thresholds takes roughly 1,500 per group. Detecting a 10-point difference takes about 400. People consistently underestimate this by an order of magnitude, which is why so many A/B tests get called early on noise.