Skip to main content
Analytics

A/B Test Math: Why 20 Conversions Is Not a Winner

Twenty extra conversions looks like a win. It usually is not. The gap between a real lift and noise comes down to three numbers you should have before launch: baseline rate, smallest lift worth shipping, and sample size. Everything else is commentary.

Compute the sample size before you look at results

A test that stops at 400 visitors per variant cannot detect the improvements most teams actually ship. Take a 10% baseline conversion rate and a 20% relative lift target (10% to 12%). A two-sided test at 95% confidence with 80% power needs roughly 3,840 visitors per variant. Raise the target to a 30% relative lift (10% to 13%) and the requirement drops to about 1,770 per variant.

The formula behind those numbers is worth knowing, because it explains why small teams keep losing:

n = (z(1-a/2) + z(1-b))^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2

With a = 0.05 and b = 0.20, the squared z-sum is about 7.85. Everything then scales with the inverse square of the effect you are chasing. Halve the expected lift and the sample size quadruples. That is the entire reason a 2% baseline needs tens of thousands of visitors per arm while a 40% baseline needs a few hundred.

Two rules follow. Fix the sample size before launch, and never compute it after seeing the data — post-hoc power is a way of choosing the number that makes your result look significant.

Peeking is the most expensive habit in experimentation

Checking the dashboard every morning and stopping the first time p < 0.05 does not give you a 5% false positive rate. It gives you something much worse. Five equally spaced looks at a nominal 5% alpha produce a real error rate near 14%. Continuous monitoring, the kind a live dashboard invites, can push it past 25%.

At that level one in four of your winners is noise, and you will spend the next quarter shipping regressions that looked like wins in week one. Two fixes actually work: decide the end date in advance and do not read conversion rates until then, or use a method designed for repeated looks (sequential testing or alpha spending). Looking at revenue daily is fine. Deciding daily is not.

What a p-value does not tell you

  • It is not the probability that your variant is better.
  • It is not the probability that the result happened by chance.
  • It says nothing about the size of the effect.
  • It says nothing about the probability that your decision is wrong.

A p-value of 0.04 means this: if there were no real difference between the two designs, a gap at least this large would appear about 4% of the time. That is a statement about the procedure, not about your button colour.

Put the interval on the lift, not on each variant

Control: 200 conversions from 2,000 visitors, so 10.0%. Variant: 240 from 2,000, so 12.0%. The difference is 2.0 percentage points. The standard error of that difference is about 0.99 points, which puts the 95% interval for the lift between roughly 0.06 and 3.94 points. The p-value lands near 0.043 — technically significant, barely.

Now read the interval as a business case. The same data says the variant might add 0.06 points, which is a rounding error, or 3.94 points, which is a 39% relative lift. Both are consistent with what you measured. Shipping is still defensible, but nobody should promise the 20% that went into the slide deck. Feed the observed rate and sample size into a confidence interval calculator before writing the summary, and report the range next to the point estimate.

Streaks are the expected behaviour of randomness

Five conversions in a row for the variant feels like proof. It is not. In 200 fair coin flips the expected number of runs of five or more heads is about six, so a sequence without one would be the surprising outcome. Before you interpret a streak in your funnel, generate one: run a few hundred flips through a coin flip simulator and watch how often long runs appear. For exact counts on specific patterns, use a probability calculator instead of intuition.

Segment before you celebrate

Aggregate lift hides opposite effects. Mobile converting worse and desktop better can average out to zero, or a small desktop gain can dominate a large mobile loss if the traffic mix shifted mid-test. Mix shifts are common: a campaign launched in week two, a bot wave, a tracking tag that broke on one platform only.

Three checks catch most of it. Is the split actually close to 50/50? Is the device and browser mix stable between arms? Is the new-versus-returning ratio the same? If a segment moved, you are comparing two populations, not two designs, and no p-value fixes that.

A pre-launch checklist

  1. One primary metric, defined before launch. Everything else is diagnostic and cannot rescue a failed test.
  2. Sample size computed from the baseline rate and the smallest lift worth shipping.
  3. End date fixed, written down, and visible to whoever will read the dashboard.
  4. Assignment verified: a sample of user IDs should split near 50/50. Draw one with a random number generator and check your bucketing function against it.
  5. Guardrail metrics watched for damage — latency, error rate, refunds, unsubscribes — never for wins.
  6. Decision rule stated in advance: ship if the lower bound of the interval clears the cost of the change.

When the sample is simply too small

Below a few hundred conversions per variant, significance testing is theatre. A 5% relative lift on a 2% baseline needs tens of thousands of visitors per arm; you will not get there on 8,000 monthly sessions. Ship the change on judgement, instrument the metric, and watch it for a month as a regression check. That is a weaker claim than an experiment, and it is honest. What is not honest is running the numbers anyway and reporting whichever comparison crossed 0.05.

The arithmetic is the easy part. The discipline is refusing to call a winner on a sample you chose after seeing the data, and reporting a range when a range is all you have.