A/B Test Significance Calculator
Know whether a test result is real or noise.
How the a/b test significance calculator works
A higher conversion rate in your variant does not mean the variant is better. With small samples, ordinary randomness produces differences that vanish when the test runs longer. Significance testing tells you whether the difference is likely real.
Enter the visitors and conversions for your control and variant. The calculator runs a two-proportion z-test, reports the p-value, and converts it to a confidence level. At 95% confidence or above, the result is unlikely to be chance.
Significance is necessary but not sufficient. A result can be significant and still too small to matter, or significant on a sample too small to trust. Pair this with a sample-size plan before you launch.
The formula
How this is calculated
A two-proportion z-test compares the variant and control conversion rates against the pooled rate. The z-score maps to a two-tailed p-value; confidence is one minus that p-value. Below a 5% p-value (95% confidence) the result is conventionally significant.
Worked example
5.0% vs 6.0% on 5,000 each clears 95% confidence, a genuine 20% relative uplift.
Frequently asked questions
What confidence level should I require?
95% (a p-value under 0.05) is the common standard. It means there is under a 5% chance of seeing this result if the variant were no different from control. Higher-stakes changes sometimes use 99%.
My result is not significant yet. What should I do?
Keep the test running until you reach your planned sample size, or accept that the effect is too small to detect. Stopping early the moment a result looks good inflates false positives.
Does significance mean the variant is definitely better?
No. It means the observed difference is unlikely to be pure chance. Effect size, sample size and test duration all matter for whether you should ship the change.
Related tools
Related reading