Different data needs different tests, and this calculator covers the three most common ones in one place. Choose the two-proportion z-test for yes-or-no outcomes, the two-mean t-test for averages with standard deviations, or the chi-square test for two-by-two tables of counts, and it returns the test statistic, the p-value, and a verdict at 95 percent confidence. One set of inputs serves all three tests, with each field changing meaning according to the test you pick.
The suite suits analysts, marketers, and students who need more than a single quick check: it shows the working behind each verdict so the result can be reported and questioned. The sections below explain how to choose a test, how each one works, which assumptions to check, and how to turn a p-value into a decision without overclaiming what it proves.
Choosing the right test
Statistical tests are tools, and each one fits a different kind of data, so choosing well matters more than calculating precisely. Use the two-proportion z-test when each observation is a yes or no outcome, such as converted or not, clicked or not, passed or failed, and you want to compare the rates in two groups. Use the two-mean t-test when each observation is a number on a continuous scale, such as revenue per user, time on page, or test scores, and you want to compare the averages. Use the chi-square test when your data forms a two-by-two table of counts across two categories, such as device type against conversion, and you want to know whether the categories are related at all.
The calculator asks for the test type first because the inputs change meaning with it. In the proportions test, the two value fields are success counts that must not exceed their group sizes. In the means test, the same fields are group means, and the standard deviation fields become essential, because a t-test cannot judge a difference in means without knowing how spread out each group is. In the chi-square test, the inputs rebuild the four cells of the table from group sizes and success counts, so the same four numbers serve all three tests with different interpretations.
Picking the wrong test usually still produces a number, which is what makes the mistake dangerous. A t-test on yes-or-no data treats a 0-or-1 outcome as if it were a continuous measurement and gives a misleading answer, while a proportions test on revenue data throws away most of the information in the numbers. When in doubt, describe your data in one sentence: if the sentence ends with a rate, use proportions; if it ends with an average, use means; if it ends with a table of counts, use chi-square.
How each test works
The two-proportion z-test pools both groups into one overall rate, because under the assumption of no real difference both groups estimate the same underlying rate. It then measures the gap between the two rates in units of standard error, producing a z-score, and converts that score into a p-value with the normal distribution. A z-score beyond 1.96 in either direction gives a p-value below 0.05, the usual significance threshold, and the calculator reports both numbers so you can see the size of the effect as well as its significance.
The Welch t-test for two means works the same way in spirit but handles unequal variances and unequal sample sizes, which the older pooled t-test does not. It divides the difference in means by a standard error built from both standard deviations and both sample sizes, and it computes degrees of freedom from a formula that reflects how much information each group actually contributes. This calculator approximates the t-test p-value with the normal distribution, which is accurate for moderate and large samples and slightly optimistic for very small ones, so treat borderline small-sample results with extra caution.
The chi-square test compares observed counts in a table with the counts you would expect if the two categories were unrelated. It squares each difference between observed and expected, divides by the expected count, and adds the results into a single chi-square statistic. For a two-by-two table, that statistic has one degree of freedom, and its p-value comes from the chi-square distribution, which this calculator computes exactly for one degree of freedom. Large gaps between observed and expected produce large statistics and small p-values.
Assumptions you must check
Every test in this suite assumes independent observations: each visitor, customer, or measurement counted once, with no person appearing in both groups and no repeated measurements treated as fresh data. Violating independence is the most common way real analyses go wrong, because duplicated data looks like a larger sample than it is and produces p-values that are far too confident. If the same users appear in both groups, or if one user generates many observations, the tests here do not apply without adjustment.
The proportions and chi-square tests also need reasonably large expected counts, roughly at least five in each cell of the table, for the approximations behind them to hold. Tiny samples with single-digit counts need exact methods instead, and this calculator will not warn you about that on its own, so check the cell counts before trusting a p-value from small data. The t-test assumes roughly normal distributions within each group, which matters most for small samples and skewed data such as revenue, where a few huge values can dominate the mean.
Randomisation matters as much as sample size. A test comparing this month with last month, or one traffic source with another, compares groups that differ in many ways besides the change you care about, and no test can separate those influences. True experiments assign subjects randomly to groups at the same time, which is what lets a significant result point at the change rather than at the calendar or the channel.
From a p-value to a decision
A significant result in any of these tests means the same thing: the observed pattern would be unusual if nothing were really different. It does not measure how large the difference is, how important it is, or whether it will hold up next month. Always read the effect size alongside the p-value: the gap in rates, the difference in means, or the shape of the table. A significant but tiny effect may not justify action, while a large but non-significant effect may justify gathering more data.
Non-significance is not proof of no difference either. It means the data at hand cannot distinguish the observed gap from noise, which happens constantly with small samples. The honest responses are to collect more data, to accept the current approach, or to move on to a more promising idea, and the right choice depends on the cost of waiting against the value of the answer. Never extend only the tests that look promising, because that selection bias manufactures false winners.
For major decisions, confirm a significant result with a fresh test or a gradual rollout, and check secondary metrics that the primary test ignores. Revenue, retention, support volume, and user complaints all move for reasons a single test cannot see, and a change that wins its primary metric while losing on the others is not a win. The suite tells you when a result deserves that next step, and that discipline turns statistics into reliable decisions.
Reporting your test properly
A significance test is only useful if someone else can check it, and that means reporting more than a p-value. A complete report names the test used, the test statistic, the p-value, the sample sizes, and the effect size, in one short paragraph that any reader can verify. For a proportions test, that means the z statistic, the p-value, both rates, and both sample sizes. For a means test, add both means, both standard deviations, and the degrees of freedom. For a chi-square test, show the full table of counts so the expected counts can be recomputed.
Report the assumptions you checked alongside the numbers. State that observations were independent, that expected counts were large enough, and that groups were randomised at the same time, or admit plainly where an assumption was shaky. Readers trust a report that shows its limits far more than one that hides them, and a shaky assumption flagged honestly can still support a cautious decision, while one discovered later destroys confidence in the whole analysis.
- Name the test used and why it fits the data type in one sentence.
- Report the test statistic, the p-value, and the sample sizes together.
- Report the effect size: the gap in rates, the difference in means, or the table shape.
- State which assumptions were checked and whether each one held.
- Note the stopping rule and whether the test ran to its planned end.
- List secondary metrics checked before any recommendation to act.
Frequently Asked Questions
Which significance test should I choose?
Choose proportions for yes-or-no outcomes compared as rates, means for continuous numbers compared as averages, and chi-square for two-by-two tables of counts. This calculator offers all three so you can match the test to your data type. Picking the wrong test still produces a number, but the number will mislead you, so describe your data first and choose second.
What is the difference between a z-test and a t-test?
A z-test compares proportions using the normal distribution and a pooled rate, while a t-test compares means using standard deviations and degrees of freedom. The z-test suits conversion rates and other yes-or-no data, and the t-test suits averages like revenue per user or test scores. This calculator runs the z-test for proportions and the Welch t-test for means.
When should I use a chi-square test?
Use chi-square when your data forms a table of counts across two categories, such as device type against conversion, or region against subscription. It tests whether the categories are related at all, rather than comparing a single rate or average. For a two-by-two table it has one degree of freedom, which this calculator handles exactly.
What does a p-value below 0.05 mean?
It means the observed pattern would occur less than five times in a hundred by chance alone if nothing were really different. That is the usual 95 percent confidence threshold used across all three tests in this calculator. It suggests a real difference, but it does not prove one, and it says nothing about how large or important the difference is.
Why does the t-test need standard deviations?
A difference in means is meaningless without knowing the spread: a ten-point gap is decisive when scores vary by two points and trivial when they vary by fifty. The standard deviations build the standard error that scales the gap, which is why the means test asks for them while the proportions test derives its spread from the rates themselves.
What are degrees of freedom in a t-test?
Degrees of freedom measure how much independent information the data contains after estimating the means and variances. The Welch formula used here blends both groups and usually gives a fractional value between the smaller group size and the total. Larger degrees of freedom mean a more precise estimate and a p-value closer to the normal approximation.
Can I use this suite for small samples?
With caution. The proportions and chi-square tests need expected counts of roughly five or more per cell, and the t-test p-value here uses a normal approximation that is slightly optimistic for very small samples. Tiny samples with single-digit counts need exact methods instead. Treat borderline small-sample results as suggestive, not decisive.
What is the difference between this suite and a quick stat sig checker?
A quick checker runs one two-proportion z-test and returns a fast verdict for routine A/B checks. This suite adds test selection across proportions, means, and chi-square, with the statistics behind each, which matters when your data is not simple conversion rates. Use the quick checker for daily checks and the suite for varied or major analyses.
Do I need equal sample sizes for these tests?
No, all three tests handle unequal group sizes, and the Welch t-test is specifically built for unequal variances and sizes. Unequal sizes reduce precision compared with balanced groups of the same total, so balanced designs are preferable when you control the split, but imbalance alone never invalidates a test.
What if my data is not independent?
These tests assume each observation is independent, with no person counted twice and nobody in both groups. Repeated measurements from the same users, or users appearing in both versions, break that assumption and produce overconfident p-values. Restructure the data to one observation per subject, or use a method built for paired or clustered data.
How do I report a significance test result?
Report the test used, the statistic, the p-value, the sample sizes, and the effect size, such as the gap in rates or the difference in means. A complete report reads like this: two-proportion z-test, z equals 2.10, p equals 0.036, rates 10 percent versus 13 percent on 1,000 visitors each. Numbers without context cannot be checked or compared.
Is a significant result always worth acting on?
No, significance says a difference is unlikely to be chance, not that it is large or valuable. A significant 0.1 percent uplift may not justify implementation cost, while a large non-significant effect may justify more data. Weigh effect size, cost, risk, and secondary metrics alongside the p-value before deciding.
What does non-significance prove?
Nothing by itself. It means the data cannot distinguish the observed gap from noise, which is common with small samples and small effects. The honest options are more data, acceptance of the current approach, or a new idea. Never extend only promising tests until they turn significant, because that selection creates false winners.
Can I test more than two groups with this calculator?
Not directly. These tests compare exactly two groups, and comparing many groups pairwise inflates false positives. For three or more groups, use methods built for multiple comparisons, such as analysis of variance for means or a larger chi-square table for counts, with appropriate corrections for the number of comparisons.
Are the p-values here exact?
The proportions p-value uses the normal approximation, which is accurate for moderate and large samples. The chi-square p-value is exact for one degree of freedom. The t-test p-value uses a normal approximation rather than the exact t distribution, which is close for larger samples and slightly optimistic for very small ones. For borderline small-sample decisions, confirm with dedicated statistical software.