A/B Test Significance Calculator
Determine if A/B test results are statistically significant at 95% or 99% from control and variant traffic.
Returns p-value, uplift, and z-score.
A/B testing (also called split testing) is a controlled experiment that compares two versions of a webpage, email, or ad to determine which one performs better. The test is only valid if the difference between variants is statistically significant — meaning it is unlikely to be due to random chance.
Conversion Rate formula:
Conversion Rate = (Conversions / Visitors) × 100
Z-Score for two proportions:
Z = (p₁ − p₂) / √[p̂(1 − p̂)(1/n₁ + 1/n₂)]
Where p̂ = (x₁ + x₂) / (n₁ + n₂) is the pooled conversion rate.
What each variable means:
- p₁: conversion rate of Variant A (control)
- p₂: conversion rate of Variant B (challenger)
- n₁, n₂: number of visitors in each group
- x₁, x₂: number of conversions in each group
- Z: test statistic; compared to critical values (1.645 for 90% confidence, 1.96 for 95%, 2.576 for 99%)
- p-value: probability the observed difference is due to chance; you want p < 0.05 for 95% significance
Worked example: Variant A: 4,200 visitors, 210 conversions → p₁ = 5.00% Variant B: 4,200 visitors, 273 conversions → p₂ = 6.50%
p̂ = (210 + 273) / (4,200 + 4,200) = 483 / 8,400 = 0.0575 SE = √[0.0575 × 0.9425 × (1/4200 + 1/4200)] = √[0.0000258066] = 0.005080 Z = (0.065 − 0.050) / 0.005080 = 2.953
Z > 2.576, and the two-tailed p-value works out at 0.0031. That clears the 99% bar, so B genuinely outperforms A here, with a relative lift of 30%.
Sample size: the number most people get badly wrong
Detecting a small lift takes far more traffic than intuition suggests. For 95% confidence at 80% power, needed per group:
| Baseline rate | Lift you want to detect | Visitors per group |
|---|---|---|
| 5% | 10% relative (5.0 → 5.5%) | ~31,200 |
| 5% | 20% relative (5.0 → 6.0%) | ~8,200 |
| 5% | 40% relative (5.0 → 7.0%) | ~2,200 |
| 2% | 20% relative | ~21,100 |
| 10% | 20% relative | ~3,800 |
Two things fall out of that table. Halving the effect you are chasing roughly quadruples the traffic you need, because sample size scales with the inverse square of the difference. And low baseline rates are punishing: at 2% you need nearly three times the traffic of a 10% baseline to catch the same relative improvement.
The practical consequence is unwelcome but worth facing. A site with 2,000 visitors a month cannot detect a 20% lift on a 5% conversion rate in any reasonable timeframe, and running the test anyway produces a number that looks like an answer and is not one. Test bigger changes, or accept that the decision has to be made on judgement.
Common mistakes
Peeking daily and stopping the moment significance appears is the big one. It is called p-hacking, and it works: check often enough and almost any test will cross p < 0.05 at some point through noise alone. Decide the sample size first, then look.
After that: running several variants at once without correcting for multiple comparisons, and ignoring the day-of-week and seasonal cycles that make a Tuesday-to-Thursday test unrepresentative. A full week is the practical minimum, and two is better.
How we build and check this calculator
This calculator runs entirely in your browser, so the numbers you enter stay on your device. The math behind it is written by hand and tested against worked examples and standard references before the page goes live.
SuperGlobalCalculator is independently built and maintained. See how we build and verify our calculators.