How the Statistically Significant Calculator Works
The Statistically Significant Calculator tests whether the difference between two conversion rates is real or could plausibly be random noise. It starts from the conversion rate of each variant, which is conversions divided by visitors, then pools both variants into one overall rate. The pooled rate feeds the standard error, which measures how much the difference between two rates would wobble by chance at the given sample sizes. Dividing the observed difference by the standard error gives the z-score, which is converted into a p-value using the normal distribution. A z-score of at least 1.96 in either direction gives a p-value below 0.05, which is the conventional threshold for significance at 95 percent confidence.
Sample size is the hidden driver of the whole test. A three percentage point gap on 1,000 visitors per variant can be significant, while the same gap on 100 visitors per variant is usually not, because small samples wobble more. The calculator makes this visible by showing the standard error logic in plain terms: larger samples shrink the standard error, which inflates the z-score and shrinks the p-value for the same observed gap. This is why experienced experimenters wait for enough traffic before declaring a winner, and why stopping a test the moment it first looks significant produces unreliable winners that fail to repeat.
How to Use the Calculator
Enter visitors and conversions for variant A, then the same pair for variant B, and press the test button. Read the two conversion rates first to confirm the inputs match the experiment, then read the relative uplift, which expresses the improvement as a percentage of the baseline rate. The z-score shows how many standard errors the difference represents, and the p-value shows the probability of seeing a gap at least this large if the variants were truly identical. The verdict line then states plainly whether the result clears the 95 percent bar, which is the point at which most teams feel comfortable acting on the result.
- Enter visitors and conversions for variant A.
- Enter visitors and conversions for variant B.
- Press the test button and check the two conversion rates first.
- Read the relative uplift to see the size of the improvement.
- Use the verdict line to decide whether the result clears 95 percent confidence.
Worked Example
Consider an experiment with 1,000 visitors and 100 conversions for variant A, and 1,000 visitors and 130 conversions for variant B. Variant A converts at 10.00 percent and variant B at 13.00 percent, so the absolute gap is 3.00 percentage points and the relative uplift is 30.00 percent. The pooled rate is 230 conversions divided by 2,000 visitors, which is 11.50 percent. The standard error is the square root of 0.115 times 0.885 times the sum of one over 1,000 and one over 1,000, which is 0.0143. Dividing the gap of 0.03 by 0.0143 gives a z-score of 2.1027, and the two-sided p-value is 0.0355, which is below 0.05, so the verdict is significant at 95 percent confidence.
A second example shows why sample size matters. With 500 visitors and 50 conversions for variant A and 500 visitors and 55 conversions for variant B, the rates are 10.00 percent and 11.00 percent, a gap of only 1.00 percentage point. The pooled rate is 10.50 percent, the standard error is 0.0194, the z-score is 0.5158 and the p-value is 0.6060, which is far above 0.05, so the verdict is not significant. The observed winner could easily be chance, and the honest conclusion is to keep running the test or accept that the variants perform similarly. Reporting the non-significant result matters as much as celebrating a significant one, because it prevents a weak change from being shipped on false confidence.
Understanding the Results
The conversion rates are the raw facts of the experiment, and they should always be read before the verdict, because a significant result on a tiny absolute gap may not matter commercially. The relative uplift translates the gap into business language, showing how much better the challenger performed relative to the baseline. The z-score measures the gap in units of chance variation, so larger absolute values mean stronger evidence, while the p-value translates that evidence into a probability: a p-value of 0.0355 means a gap at least this large would appear about 3.55 percent of the time if the variants were identical. The verdict applies the conventional 0.05 threshold, but it is a decision aid, not a proof, and results near the threshold deserve extra traffic before action.
- Conversion rates are the raw facts and should be read before the verdict.
- Relative uplift expresses the improvement as a percentage of baseline.
- The z-score measures the gap in units of chance variation.
- A p-value below 0.05 is the conventional bar for 95 percent confidence.
- Results near the threshold deserve more traffic before any decision.
Tips for Reliable Experiments
The most valuable habit in experimentation is deciding the sample size before the test starts, because peeking at results and stopping early inflates false positives dramatically. Random assignment matters just as much: if the two variants reach different audiences, the test measures audience differences rather than variant differences. Run the test for full weekly cycles to cover day-of-week effects, and avoid changing the variants mid-test, since a mid-test change restarts the question being asked. Finally, record the primary metric in advance, because testing many metrics and reporting only the significant one is another reliable way to manufacture false winners.
- Fix the required sample size before the test starts and do not stop early.
- Assign visitors randomly so variants reach comparable audiences.
- Run full weekly cycles to cover day-of-week effects.
- Freeze the variants during the test and record the primary metric in advance.
Common Mistakes to Avoid
The most common mistake is confusing statistical significance with practical importance, since a tiny improvement on huge traffic can be significant yet worthless. Another is running many tests and celebrating the one that passes, which guarantees false positives across the set. A third mistake is ignoring the direction of the result: a significant result can mean the challenger is worse, and shipping it anyway because the test was significant is a costly error. Teams should also remember that significance assumes clean data, so broken tracking, bot traffic or duplicate visitors can all produce confident-looking nonsense that no p-value can rescue.
Power and Sample Size
Statistical power is the probability that a test detects a real effect when one exists, and 80 percent power is the conventional target in most experimentation programmes. Power depends on the baseline rate, the size of the effect being hunted, the sample size and the significance threshold, and small effects demand large samples by a law that surprises newcomers. Detecting a move from a 10 percent baseline to 12 percent, a 20 percent relative improvement, needs roughly 3,600 visitors per variant at 80 percent power and 95 percent confidence, which is far more traffic than most small sites collect in a week. This arithmetic explains why small experiments so often return non-significant results: they were never large enough to detect the effect they were looking for, and the honest conclusion is that the test was underpowered rather than that the change had no effect.
The minimum detectable effect is the smallest improvement a test can reliably find at its planned sample size, and it should be set before the test starts. A team that needs a 30 percent relative uplift to justify an engineering investment should not run a test powered only for a 5 percent uplift, because the test will either run for months or return a noisy verdict. Conversely, a team testing a trivial copy change does not need power for a 1 percent effect, since even a proven 1 percent gain may not repay the effort. Aligning the minimum detectable effect with the business decision keeps experimentation efficient: power the test for the effect size that would actually change the decision, and accept that smaller effects will remain unknown.
Sequential testing addresses the peeking problem by adjusting the significance threshold as results accumulate, which lets teams monitor tests continuously without inflating false positives as badly as naive peeking does. The adjustment works by spending the total error budget gradually across looks, so early looks face a stricter bar than the final analysis. Tools that implement sequential methods exist, but they require the method to be chosen in advance and the spending schedule to be respected, which is more discipline than many teams apply. The simpler and still reliable approach remains the fixed horizon test: decide the sample size, run to it without stopping, and analyse once, which is the design this calculator assumes.
Beyond the P-Value
Confidence intervals carry more information than a p-value alone, because they show the range of effects consistent with the data rather than a single yes or no verdict. A result showing a 3 percentage point improvement with a 95 percent confidence interval from 0.2 to 5.8 points is significant, but the interval reveals that the true effect could be barely positive or quite large, which matters for forecasting revenue. A result with an interval from 2.5 to 3.5 points is also significant, but its narrow range supports confident planning. Reporting intervals alongside p-values therefore improves decisions, especially when the interval crosses a business threshold such as the minimum gain needed to repay implementation cost.
Practical significance thresholds should be set before data collection, so that a statistically significant result still has to clear a business bar before shipping. A team might decide that only improvements above 2 percentage points justify the engineering maintenance of a new variant, in which case a significant 0.5 point gain would be recorded and rejected rather than shipped. This discipline prevents the accumulation of marginal changes that each pass a statistical test but together add complexity without meaningful benefit. It also protects against the opposite error, where a large but non-significant result is discarded even though more traffic might have confirmed it, since the threshold discussion forces the team to state what would change their mind.
Bayesian interpretations offer a different framing of the same data, expressing results as the probability that one variant is better rather than as a p-value against a null hypothesis. A Bayesian analysis might report an 87 percent chance that variant B beats variant A, which many stakeholders find more intuitive than a p-value of 0.13. The two frameworks can disagree in edge cases, especially with strong prior beliefs or small samples, so teams should pick one framework per experiment and avoid switching frameworks mid-test to chase a favourable verdict. Whichever framing is chosen, the underlying data requirements are the same: clean random assignment, adequate sample size and a pre-declared primary metric.
- Target 80 percent power and set the minimum detectable effect before testing.
- Report confidence intervals alongside p-values for better decisions.
- Set a practical significance threshold that a result must clear before shipping.
- Pick one statistical framework per experiment and do not switch mid-test.
Testing several variants or several metrics at once inflates the chance of a false positive, because each additional test is another opportunity for chance to cross the threshold. The simplest correction, named after Bonferroni, divides the significance threshold by the number of tests, so five simultaneous tests would each need a p-value below 0.01 rather than 0.05 to count as significant. The correction is conservative and can hide real effects when many tests run, which is why teams often designate one primary metric in advance and treat other metrics as exploratory. Exploratory results can still guide future tests, but they should not drive shipping decisions with the same confidence as a pre-declared primary result.
Frequently Asked Questions
What does the Statistically Significant Calculator do?
It compares two conversion rates using a two-proportion z-test and reports a z-score, p-value and a significant or not verdict at 95 percent confidence.
What is a p-value?
A p-value is the probability of seeing a gap at least as large as the observed one if the two variants were truly identical, so smaller values mean stronger evidence.
What does significant at 95 percent mean?
It means the p-value is below 0.05, which is the conventional threshold at which the difference is treated as unlikely to be chance alone.
What is a z-score?
A z-score measures the observed difference in units of standard error, so a z-score of 2.1027 means the gap is about 2.10 standard errors from zero.
Why does sample size matter so much?
Larger samples shrink the standard error, so the same percentage gap produces a larger z-score and a smaller p-value with more visitors.
What is relative uplift?
Relative uplift expresses the improvement as a percentage of the baseline rate, so a move from 10 percent to 13 percent is a 30 percent relative uplift.
Does significance mean the improvement is large?
No, significance measures evidence against chance, not size, so a tiny improvement on huge traffic can be significant yet commercially worthless.
What is a pooled conversion rate?
The pooled rate combines both variants into one overall rate, which is used to estimate the standard error of the difference.
Can a significant result mean the challenger is worse?
Yes, significance only means the difference is unlikely to be chance, so a significant negative result means the challenger is reliably worse.
Why should I not stop a test early?
Stopping the moment a result first looks significant inflates false positives, because random wobbles cross the threshold often during a test.
What is the 1.96 threshold?
A z-score of 1.96 marks the two-sided 5 percent cutoff of the normal distribution, which is the standard bar for 95 percent confidence.
Does this calculator work for click-through rates?
Yes, any rate formed from successes divided by trials works, including click-through rates, signup rates and purchase rates.
What if my sample sizes are very different?
The test handles unequal sample sizes through the pooled standard error, although very unbalanced splits reduce statistical power.
Should I act on a result just above the threshold?
Results near the threshold deserve more traffic before action, since small additional data can move the verdict in either direction.
Does significance guarantee the result will repeat?
No, significance is a probability statement about this experiment, and replication still depends on clean data and a stable audience.