Definition
Statistical significance is the measure of how likely an experiment result is to be real rather than the product of random chance. In A/B testing, it answers the question that decides whether you should trust a winner: is the difference between the variants large enough, on enough data, that it is unlikely to have happened by luck?
A higher conversion rate in a variant does not by itself mean the variant is better. With small samples, ordinary randomness produces differences that vanish when the test runs longer. Significance is the discipline that separates a genuine effect from noise.
How significance is measured
Significance is usually expressed through two linked numbers:
- The p-value is the probability of seeing a result at least this extreme if there were actually no difference between the variants. A lower p-value means chance is a less plausible explanation.
- The confidence level is one minus the p-value, expressed as a percentage. A p-value of 0.05 corresponds to 95% confidence.
By convention, a result is called statistically significant at 95% confidence, meaning a p-value below 0.05: under a 5% chance of seeing this result if the variant were truly no different. You can test a result with the A/B test significance calculator, which runs a two-proportion comparison and reports the p-value and confidence.
Why significance matters for revenue
Acting on a result that is not significant is how teams ship changes that quietly do nothing, or worse, hurt. A variant that looks like a winner on day two often regresses once more data arrives, because the early lead was noise. Requiring significance before you conclude protects you from that expensive mistake.
Significance is necessary but not sufficient. A result can be significant and still too small to matter, or significant on a sample too small to represent your real traffic. That is why significance pairs with a minimum detectable effect and a planned sample size: decide the smallest effect worth detecting, size the test for it, then run to significance rather than stopping the moment the number looks good.
From a Revenue Intelligence perspective, significance is the confidence check that turns an estimated improvement into a measured one. It is the same instinct the Revenue Intelligence framework applies throughout: do not trust that a fix worked until the evidence says it did, which matters whether you are testing a checkout change or any other leak.
Related metrics
Statistical significance works with the minimum detectable effect, which sets how large an effect you are testing for, and incrementality, which extends the same causal discipline to marketing channels. Together they keep experimentation honest.
In short
Statistical significance measures whether a test result is real or chance, through the p-value and confidence level. The 95% convention is standard, significance is necessary but not sufficient, and requiring it before you conclude is what stops teams from shipping noise as if it were a win.