You have two conversion rates and one question: is version B actually better, or is the gap just noise. This quick checker answers it with a two-proportion z-test, returning both rates, the uplift, the z-score, the p-value, and a plain-language verdict at 95 percent confidence. Enter visitors and conversions for each version, and the verdict tells you whether to trust the difference or keep the test running.
The checker is built for speed on routine A/B checks, not for every statistical situation. It assumes simple conversion data with reasonable sample sizes, and it cannot judge whether the experiment itself was clean. The sections below explain how to read the numbers, which mistakes fake significance, and how to turn a significant result into a sensible shipping decision.
What this quick checker does
This calculator answers one narrow question: given two conversion rates and their sample sizes, is the difference between them large enough to trust, or could it be chance. It uses the two-proportion z-test, which pools both groups into one overall rate, works out the standard error of the difference, and turns that difference into a z-score and a p-value. A p-value below 0.05, matching a z-score beyond plus or minus 1.96, is the usual 95 percent confidence threshold, and the calculator states its verdict in plain language alongside the numbers.
The checker is deliberately quick: it takes four numbers and returns a verdict, without the test selection, confidence intervals, or power analysis that a full significance suite provides. That speed suits the common situation of a marketer or product manager checking an A/B test mid-flight and wondering whether to call it. When the answer matters for a major decision, confirm it with a fuller analysis, because a quick check cannot catch every problem in the experiment behind the numbers.
Sample size is the hidden driver of every result here. A three-point gap on a thousand visitors each side is significant, while the same gap on a hundred visitors each side is not, because small samples bounce around far more. The calculator makes that visible: run the same rates with smaller samples and watch the p-value rise. That lesson alone prevents more bad decisions than any other part of statistics.
Reading the z-score and p-value
The z-score measures the difference in units of standard error, which is a scale-free way of asking how surprising the gap is. A z-score near zero means the rates are practically identical, while a z-score of two or more means the gap is about twice the noise level, which is unusual under the assumption of no real difference. The sign tells direction: positive means version B converts better, negative means version A is ahead.
The p-value translates the z-score into a probability: if there were truly no difference between the versions, how often would chance alone produce a gap at least this large. A p-value of 0.03 means about three times in a hundred, which is uncommon enough that most teams accept the result. A p-value of 0.40 means forty times in a hundred, which is entirely ordinary and proves nothing. The p-value is not the probability that the result is true, and treating it as one is the most common statistical misreading in business.
Confidence language helps here. Saying a result is significant at 95 percent confidence means the method would rarely mislead you if the experiment were clean, not that this particular result is 95 percent certain. Cleanliness matters: a test with uneven traffic splits, broken tracking, or visitors counted twice can produce a significant p-value for a difference that never existed. Check the experiment before trusting the statistic.
Mistakes that fake significance
Peeking is the most damaging habit in A/B testing: checking the result every day and stopping the moment it turns significant. A test checked twenty times has far more chances to cross the threshold by luck than a test checked once, so peeking inflates false positives dramatically. Decide the sample size or run time in advance, and let the test run to that point before judging it, even when early numbers look exciting.
Stopping early for a winner and extending losers is the same error in disguise. If you stop tests that look good and keep running tests that look bad until they improve, your winners list fills with lucky accidents. Apply the same stopping rule to every test, and record the rule before the test starts, so nobody can quietly move the goalposts mid-test.
Multiple comparisons multiply the problem. Testing ten button colours at once and celebrating the one that hits significance is almost guaranteed to find a false winner, because ten chances at a five percent error rate produce roughly a forty percent chance of at least one false hit. When you test many variants, use a stricter threshold or a proper correction, and be honest about how many things you tried.
From significance to a decision
Significance alone does not justify shipping a change. A result can be statistically significant and commercially trivial, such as a 0.1 percent uplift on a huge sample that took months to detect. Weigh the uplift against the cost of implementing and maintaining the change, and against the risk if the result turns out to be a false positive. Small, cheap wins are worth shipping on thinner evidence than expensive, risky ones.
Also consider what the test did not measure. A change that lifts sign-ups but hurts retention, increases support tickets, or annoys paying customers can look like a win in a conversion test and a loss in the business. Check secondary metrics before shipping, especially revenue, retention, and complaints, and be ready to roll back if they move the wrong way.
Finally, remember that one significant test is a signal, not proof. Repeating a winning change in a fresh test, or rolling it out gradually and watching real metrics, turns a statistical claim into business knowledge. The quick checker tells you when a result deserves that next step, and that is exactly the job it is built for.
A pre-launch checklist for your test
Most A/B test failures are decided before launch, when the sample size, stopping rule, and tracking are set. Decide the minimum sample size or run time in advance using your baseline rate and the uplift you want to detect, and write the stopping rule down where the whole team can see it. Confirm that tracking fires correctly for both versions, that traffic splits evenly, and that no other change launches during the test window, because a concurrent promotion or a broken tracking tag can manufacture a difference that no statistic can untangle afterwards.
Agree the decision rule before the first visitor arrives. State what uplift would justify shipping, which secondary metrics would block a launch even on a significant result, and who makes the final call. That agreement removes the temptation to move goalposts when early numbers look exciting or disappointing, and it turns the test from an argument into a measurement. A test launched with a written plan is worth more than a larger test launched on hope.
- Calculate the required sample size from baseline rate and target uplift before launching.
- Write the stopping rule down and share it with everyone involved in the test.
- Verify tracking fires correctly for both versions on every device and browser.
- Confirm the traffic split is even and no other change runs during the test.
- List secondary metrics that could block a launch even on a significant result.
- Agree who decides the outcome and what uplift justifies shipping the change.
The ninety-five percent threshold itself is a convention rather than a law of nature, and knowing its history keeps it in perspective. It traces back to the statistician R. A. Fisher, who suggested in the nineteen twenties that a result occurring less than once in twenty times by chance was worth a second look, and the suggestion hardened into a default across science and business. The threshold balances two errors: calling a false result real, and missing a real result by demanding too much proof. Cheap, reversible experiments can reasonably use a looser threshold, while expensive or risky changes deserve a stricter one, and the calculator verdict is a starting point for that judgement rather than a substitute for it.
Frequently Asked Questions
What does statistical significance mean in an A/B test?
It means the difference between two versions is large enough, relative to sample size, that chance alone is an unlikely explanation. This calculator uses a p-value below 0.05, or a z-score beyond plus or minus 1.96, as the 95 percent confidence threshold. Significance suggests a real difference, but it does not guarantee one, especially if the experiment had flaws.
How do I use this stat sig calculator?
Enter visitors and conversions for version A and version B, then read the rates, uplift, z-score, p-value, and verdict. A significant verdict at 95 percent confidence means version B looks like a real improvement. If the verdict is not significant, keep the test running to gather more data or keep the current version.
What is a good p-value for an A/B test?
Below 0.05 is the standard threshold for 95 percent confidence, and below 0.01 is stronger evidence at 99 percent confidence. Lower is not automatically better for decisions: a tiny p-value on a trivial uplift may not justify shipping a change. Judge the uplift size and the business cost alongside the p-value.
What is a z-score in plain language?
A z-score measures a difference in units of standard error, which is the natural noise scale for the sample sizes involved. A z-score near zero means no real gap, while a z-score beyond 1.96 in either direction crosses the usual significance threshold. Positive scores favour version B and negative scores favour version A in this calculator.
Why is my result not significant even though B looks better?
Small samples produce wide uncertainty, so a visible gap can still be within the range of chance. A 3 percent gap on 100 visitors each side is unremarkable, while the same gap on 1,000 visitors each side is significant. The fix is usually more data, not a different calculator: let the test run longer or send it more traffic.
Can I check my test every day until it becomes significant?
Checking repeatedly and stopping at the first significant moment, called peeking, inflates false positives sharply. A test checked twenty times has many more chances to cross the threshold by luck than one checked once. Decide the sample size or run time in advance and judge the test only when it reaches that point.
What sample size do I need for an A/B test?
It depends on your baseline rate and the uplift you want to detect: smaller uplifts and lower baseline rates need larger samples. As a rough guide, detecting a 20 percent relative uplift on a 10 percent baseline needs roughly 1,500 visitors per side. Use a sample size calculator before launching, rather than guessing mid-test.
Does significance prove version B caused the improvement?
Only if the experiment was properly randomised with clean tracking and no other changes during the test. Significance measures the numbers, not the experiment design, so a biased split, broken tracking, or a concurrent promotion can produce a significant result for a difference that is not real. Audit the test setup before acting on the verdict.
What is the difference between this checker and a full significance test?
This checker runs one fast two-proportion z-test and returns a plain verdict, which suits quick A/B checks. A full suite adds test selection, confidence intervals, power analysis, and tests for means or categories, which matter when data is not simple conversion rates. Use the quick checker for routine checks and the full suite for major decisions.
What should I do with a significant result?
Check secondary metrics such as revenue, retention, and support volume, weigh the uplift against implementation cost, and consider a gradual rollout or a repeat test before full shipping. Significance is a green light to proceed carefully, not a command to ship immediately, especially for expensive or risky changes.
What should I do with a non-significant result?
Do not ship the change on hope, and do not immediately declare the idea dead either. Either gather more data by running longer, or accept the current version and move to the next idea. Repeatedly extending only losing tests until they turn significant creates false winners, so apply the same stopping rule to every test.
Can significance be faked by testing many variants?
Yes, testing many variants at once and celebrating the one that hits significance almost guarantees false winners, because each test carries its own five percent error chance. With ten variants, the chance of at least one false hit is roughly forty percent. Use a stricter threshold or a correction when testing many variants at once.
Is a 95 percent confidence level always right?
It is the common default, but stricter levels suit high-stakes decisions and looser levels suit cheap, reversible experiments. A 99 percent threshold needs more data but misleads less often, while a 90 percent threshold decides faster with more false alarms. Match the threshold to the cost of being wrong.
Why do rates and uplift matter alongside the p-value?
A result can be significant yet trivial, such as a 0.1 percent uplift detected on a huge sample. The uplift tells you whether the change is worth the effort, while the p-value tells you whether it is likely real. Decisions need both: real but tiny effects may not justify shipping, and large but uncertain effects may justify more testing.
Does this calculator work for email or ad tests?
Yes, any test with two groups, a count of people reached, and a count who converted fits the same maths, whether the conversion is a click, a sign-up, or a purchase. Enter visitors as the reach for each version and conversions as the action count, and the z-test, p-value, and verdict apply exactly as they do for website tests.