Is your A/B test result real?
If variant B converts at 13% and A at 10%, B looks better. But with small samples, a gap like that can appear by pure chance. Statistical significance tells you how unlikely your result would be if the two versions actually performed the same.
Pooled rate p = (Conversions A + Conversions B) ÷ (Visitors A + Visitors B)
z = (Rate B − Rate A) ÷ √[ p × (1 − p) × (1/Visitors A + 1/Visitors B) ]
The p-value is the chance of seeing a difference at least this big if there were no real difference. At 95% confidence, a result counts as significant when p is below 0.05.
Worked example
A: 100 conversions from 1,000 visitors (10%). B: 130 from 1,000 (13%), a 30% relative uplift. The pooled rate is 11.5%, z ≈ 2.10, and the p-value is about 0.035. That’s below 0.05, so B wins at 95% confidence. At 99% it would not yet be significant.
How many visitors do you need?
Decide before you start. Small changes need large samples. To detect a 20% relative lift on a 5% baseline (5% → 6%) at 95% confidence and 80% power, you need about 8,158 visitors per variant. Halve the lift you want to detect and you need roughly four times as many visitors.
Common A/B testing mistakes
- Stopping as soon as it looks significant. Checking daily and stopping at the first green result inflates false positives. Set the sample size up front.
- Running for less than a full week. Weekday and weekend visitors behave differently. Run whole weeks.
- Testing tiny changes on low traffic. A button colour test on 200 visitors a day may need months. Test bigger ideas first.
- Trusting broken tracking. If conversions are double-counted in GA4, your test is too. Check your setup first.
Frequently asked questions
What does statistically significant mean in an A/B test?
It means the difference between A and B is unlikely to be due to random chance alone. At 95% confidence, there is less than a 5% probability of seeing a difference this large if both versions performed the same.
What p-value do I need?
For the standard 95% confidence level, the p-value must be below 0.05. For 99% confidence it must be below 0.01.
How long should an A/B test run?
Until it reaches the sample size you planned, and for at least one full week so every day of the week is represented. Use the sample size section of this calculator to estimate it.
What is minimum detectable effect?
The smallest relative improvement you want the test to reliably detect. Smaller effects need many more visitors to detect.
Can I trust a result that's not significant?
Not as a win. It means the test hasn't shown a real difference yet. Either keep collecting data to your planned sample size, or treat the variants as equivalent.