Skip to main content
Analytics

Confidence Intervals Without Jargon: What Your Sample Really Says

You run an A/B test, 400 users per arm, and the new button wins by 5%. Someone asks "how sure are we?" and the room goes quiet. That question has a concrete answer, and it fits in one line: a confidence interval.

What the interval actually is

A confidence interval is a range built from your sample that would contain the true value 95% of the time if you repeated the whole experiment many times. That is the definition, and it is easy to misread. It is not "a 95% chance the true value is inside this range." Once the data is collected, the true value either is or is not in your interval. The 95% describes the procedure, not this one interval.

What you get in practice is the useful part: a range of values that are compatible with your data. If the interval for a lift runs from -0.2% to +6.1%, then "no effect" is one of those compatible values, and you have not shown anything. If it runs from +1.8% to +6.1%, zero is outside it and the direction is settled.

The two formulas you will actually use

For an average: x̄ ± z × σ/√n. For a proportion (conversion rate, click rate, error rate): p̂ ± z × √(p̂(1-p̂)/n), where z = 1.96 for 95%.

Worked example. 120 conversions out of 2,000 visitors: p̂ = 0.06. Standard error = √(0.06 × 0.94 / 2000) = 0.00531. Multiply by 1.96, you get 0.0104. The 95% interval is 4.96% to 7.04%. Report it as "6.0% (95% CI 5.0-7.0)". That single range tells your reader more than the point estimate ever will.

Width is a budget decision

Standard error shrinks with √n, so the interval halves only when the sample quadruples. Going from 2,000 to 8,000 visitors takes that ±1.04 pp down to ±0.52 pp. That is the real cost of precision, and it is why "let us just run it longer" is sometimes the correct answer and sometimes a wasted month.

Before you launch, decide how narrow the interval must be to support the decision. If you will act only on a lift above 2%, you need a half-width around 1 pp, which at a 6% baseline means roughly 2,100 users per arm. Do that arithmetic first. It prevents the classic ending where the test is inconclusive and nobody has time to rerun it.

The overlap mistake

Two groups, each with its own 95% interval, and the intervals overlap. Does that mean no significant difference? Not necessarily. Comparing two separate intervals is a much stricter test than comparing the difference, and overlapping intervals can still hide a real difference. Test the difference directly: (p̂₁ - p̂₂) ± 1.96 × √(SE₁² + SE₂²). If that interval excludes zero, the difference is there.

The reverse direction is safe: if the intervals do not overlap, the difference is certainly significant. It is the "they overlap, so we are fine" conclusion that gets teams into trouble.

Small samples change the multiplier

With fewer than about 30 observations, 1.96 is too optimistic. Use the t distribution: at n = 12 the 95% multiplier is 2.20, which widens the interval by 12%. At n = 5 it is 2.78. This is not pedantry. With 10 samples, using 1.96 understates your uncertainty by a quarter, and that is how small pilots get overinterpreted in planning meetings.

Intervals for latency are not symmetric

Latency, file sizes, and revenue per user are skewed. A mean with a symmetric interval around it can even include impossible values, such as negative milliseconds. For p95 latency, resample your data a few thousand times with replacement, compute p95 each time, and take the 2.5th and 97.5th percentiles of those results. A bootstrap of 2,000 resamples runs in seconds in any language and needs no distribution assumptions.

What the interval does not cover

Sampling error is only one source of error. A tight interval around a biased measurement is still wrong.

  • Instrumentation bugs. If the event fires twice for 3% of users, your interval is precise about the wrong number.
  • Bot traffic. Crawlers inflate numerator and denominator unevenly across variants.
  • Novelty and seasonality. A two-week test in November does not describe January.
  • Multiple comparisons. Test 12 metrics at 95% and expect roughly one false positive from chance alone.

How to report it

Three numbers per metric: point estimate, interval, and sample size. "Conversion 6.0% (95% CI 5.0-7.0, n = 2,000)". That is all. No p-values are needed for most decisions, because the interval already tells you the direction and the plausible size of the effect.

And decide from the pessimistic end. If the lower bound of the lift still pays for the engineering work, ship it. If the upper bound barely covers the cost, do not. This framing survives arguments that p-values start and lose.

Quick reference

  • 95% is a default, not a law. Use 90% for exploration, 99% when a wrong call is expensive.
  • n is per group, not the total across both.
  • Halving the width costs 4× the sample.
  • Overlapping group intervals prove nothing; non-overlapping ones prove a difference.
  • Report percentage points, not percent, when comparing two rates.

If you want to skip the manual arithmetic, the confidence interval calculator takes your sample size and mean or proportion and returns the interval for the confidence level you pick. For the difference between two variants, the probability calculator handles the combination math, and for a pre-test sample size estimate the percentage calculator covers most lift thresholds. Run the numbers before the test, not after. A decision you cannot make from the interval is usually a decision you did not size for.