A/B Test Significance Calculator

A/B testing (also called split testing) is a controlled experiment that compares two versions of a webpage, email, or ad to determine which one performs better. The test is only valid if the difference between variants is statistically significant — meaning it is unlikely to be due to random chance.

Conversion Rate formula: Conversion Rate = (Conversions / Visitors) × 100

Z-Score for two proportions: Z = (p₁ − p₂) / √[p̂(1 − p̂)(1/n₁ + 1/n₂)]

Where p̂ = (x₁ + x₂) / (n₁ + n₂) is the pooled conversion rate.

What each variable means:

• p₁: conversion rate of Variant A (control)
• p₂: conversion rate of Variant B (challenger)
• n₁, n₂: number of visitors in each group
• x₁, x₂: number of conversions in each group
• Z: test statistic; compared to critical values (1.645 for 90% confidence, 1.96 for 95%, 2.576 for 99%)
• p-value: probability the observed difference is due to chance; you want p < 0.05 for 95% significance

Worked example: Variant A: 4,200 visitors, 210 conversions → p₁ = 5.00% Variant B: 4,200 visitors, 273 conversions → p₂ = 6.50%

p̂ = (210 + 273) / (4,200 + 4,200) = 483 / 8,400 = 0.0575 SE = √[0.0575 × 0.9425 × (1/4200 + 1/4200)] = √[0.000025768] = 0.005076 Z = (0.065 − 0.050) / 0.005076 = 2.955

Z > 2.576 → 99% confidence that B outperforms A. Variant B shows a relative lift of 30%.

Sample size guidance: For 95% confidence with 80% power: need ~1,600 visitors per group to detect a 20% relative lift from a 5% baseline. Run the test until you reach this sample size — stopping early inflates false positive rates.

Common mistakes: Peeking at results daily and stopping when significance appears (p-hacking), not accounting for multiple testing (Bonferroni correction needed), and ignoring seasonal effects.