A/B test significance calculator: is the difference real?

How to use it

1. Enter the number of visitors, recipients or respondents who saw variant A, and how many of them converted.
2. Enter the same two figures for variant B.
3. Read the conversion rates, the relative uplift, the z-score and the p-value.
4. If the result is not significant, check the required sample per variant to see how much longer the test would need to run.

Formula and logic

The calculator runs a two-proportion z-test:

z = (p_B − p_A) / √(p(1 − p) · (1/n_A + 1/n_B))

Here p_A and p_B are the conversion rates of the two variants, n_A and n_B are the numbers of visitors, and p is the pooled conversion rate: all conversions divided by all visitors. The two-sided p-value is the probability of a difference at least this large, in either direction, if the variants truly performed the same. The relative uplift is p_B / p_A − 1.

The required sample per variant is:

n ≈ (z₁₋α/₂ + z₁₋β)² · (p₁(1 − p₁) + p₂(1 − p₂)) / (p₂ − p₁)²

with z₁₋α/₂ = 1.96 for 95% confidence and z₁₋β ≈ 0.84 for 80% power.

Example: A converts 200 of 5,000 visitors (4.0%) and B converts 230 of 5,000 (4.6%). The uplift is +15%, but z = 1.48 and p = 0.14, so the difference is not significant at the 5% level. Detecting a change from 4.0% to 4.6% reliably would need about 17,900 visitors per variant.

How to read the result

A p-value below 0.05 means a difference this large would be unlikely if the variants performed equally; it does not tell you how large the true effect is or whether it is worth acting on. Look at the uplift and its business value alongside the p-value.

The most common error is stopping a test as soon as it first looks significant. Repeated checking inflates the chance of a false winner, so set the sample size in advance and run the test to the end, ideally over whole weeks to cover weekday and weekend behaviour. Choose one primary metric before the start; if you compare many metrics or several variants, some will look significant by chance.

The same test works for survey data, for example to check whether two groups of respondents differ in their answer to a yes/no question, as long as the two groups are independent. For testing creative before launch, see our advertising testing and concept testing services; the glossary entry on statistical significance explains the underlying logic.