A/B Test Significance Calculator
Determine if A/B test results are statistically significant at 95% or 99% from control and variant traffic.
Returns p-value, uplift, and z-score.
A/B testing (also called split testing) is a controlled experiment that compares two versions of a webpage, email, or ad to determine which one performs better. The test is only valid if the difference between variants is statistically significant — meaning it is unlikely to be due to random chance.
Conversion Rate formula:
Conversion Rate = (Conversions / Visitors) × 100
Z-Score for two proportions:
Z = (p₁ − p₂) / √[p̂(1 − p̂)(1/n₁ + 1/n₂)]
Where p̂ = (x₁ + x₂) / (n₁ + n₂) is the pooled conversion rate.
What each variable means:
- p₁ — conversion rate of Variant A (control)
- p₂ — conversion rate of Variant B (challenger)
- n₁, n₂ — number of visitors in each group
- x₁, x₂ — number of conversions in each group
- Z — test statistic; compared to critical values (1.645 for 90% confidence, 1.96 for 95%, 2.576 for 99%)
- p-value — probability the observed difference is due to chance; you want p < 0.05 for 95% significance
Worked example: Variant A: 4,200 visitors, 210 conversions → p₁ = 5.00% Variant B: 4,200 visitors, 273 conversions → p₂ = 6.50%
p̂ = (210 + 273) / (4,200 + 4,200) = 483 / 8,400 = 0.0575 SE = √[0.0575 × 0.9425 × (1/4200 + 1/4200)] = √[0.000025768] = 0.005076 Z = (0.065 − 0.050) / 0.005076 = 2.955
Z > 2.576 → 99% confidence that B outperforms A. Variant B shows a relative lift of 30%.
Sample size guidance: For 95% confidence with 80% power: need ~1,600 visitors per group to detect a 20% relative lift from a 5% baseline. Run the test until you reach this sample size — stopping early inflates false positive rates.
Common mistakes: Peeking at results daily and stopping when significance appears (p-hacking), not accounting for multiple testing (Bonferroni correction needed), and ignoring seasonal effects.