A/B Test Significance Calculator
Find out whether your A/B test result is statistically significant, or just noise.
How this is calculated
This runs a two-proportion z-test: it pools both variants' conversion data to compute a shared standard error, then measures how many standard errors apart the two observed conversion rates are (the z-score). That z-score converts to a two-tailed p-value — the probability of seeing a difference this large (or larger) purely by chance. A p-value below 0.05 is treated as statistically significant at 95% confidence, the standard threshold for A/B testing.
With this calculator's default inputs — 1,000 visitors and 100 conversions in variant A (10%) versus 1,000 visitors and 130 conversions in variant B (13%) — that 3-percentage-point gap comes back statistically significant at 95% confidence, a 30% relative uplift for variant B. Before running a test like this, use the Sample Size calculator to work out how many visitors you'll need to reliably detect the effect size you're hoping for, and once you have a result, the Confidence Interval calculator can express the uncertainty around each variant's conversion rate as a range rather than a single point estimate.
Frequently asked questions
- What counts as "statistically significant" here?
- A p-value below 0.05 — meaning there's less than a 5% chance the observed difference between variants happened purely by random chance. This is the standard 95% confidence threshold used across most A/B testing tools.
- What is a two-proportion z-test?
- It's the standard statistical test for comparing two conversion rates (proportions) measured on two independent groups — it pools both variants' data to estimate a shared standard error, then measures how many standard errors apart the two observed rates are.
- My result says "not yet significant" — what should I do?
- Keep running the test and collecting more data. A non-significant result doesn't mean there's no difference — it often means you don't have enough visitors yet to detect the difference reliably. Stopping a test early based on a promising-looking but not-yet-significant result is a common source of false positives.
- What is relative uplift?
- The percentage change in variant B's conversion rate compared to variant A — e.g. if A converts at 10% and B at 13%, that's a 30% relative uplift, even though the absolute difference is only 3 percentage points.
- Can I use this in my own app?
- Yes — every calculator on Stupidly Clever has a matching REST API and MCP tool that runs the same underlying logic.
- What is the difference between one-tailed and two-tailed significance tests?
- A two-tailed test (used here) checks for a difference in either direction — your variant could be better or worse. A one-tailed test only checks one direction (e.g., variant is better). Two-tailed is more conservative and is the standard for honest A/B testing, since a variant performing significantly worse is also a meaningful result.
- What is the risk of peeking at results before the test is complete?
- Stopping an A/B test early when you first see significance dramatically inflates your false positive rate. If you peek at p-values repeatedly and stop the moment you see p < 0.05, you may be fooled by random variation — always determine your sample size upfront and commit to it before looking at results.