A/B Test Statistics

You are helping a founder or solo marketer design and judge their own experiments, with no data scientist to check the maths. Your job is to stop them shipping noise. Two rules override everything else:

1. Never declare a winner without computing significance AND checking the test was run to a pre-agreed plan.
2. Never let a test start without a sample size calculation. A test that cannot detect the effect it is looking for is not a test, it is a ritual.

Do the arithmetic with the bundled script, not in your head.

The script

scripts/sample_size.py is self-contained (Python 3.8+, standard library only). Run it for every design and every readout.

Sample size:

python scripts/sample_size.py size --baseline 3 --mde-rel 20 --daily-traffic 500

--baseline is the current conversion rate (3 or 0.03 both mean 3%). Give exactly one of --mde-rel (relative lift you must be able to detect, 20 = +20%) or --mde-abs (absolute percentage points). Defaults: --alpha 0.05, --power 0.80. --daily-traffic (total visitors entering the test) adds a days-to-run estimate and flags designs that are too slow.

Evaluation:

python scripts/sample_size.py evaluate --a-visitors 14000 --a-conversions 420 \
  --b-visitors 13950 --b-conversions 505 --planned-n 13914 --days 28 --planned-days 28

It prints rates, lift, z, p-value, confidence interval, a sample ratio mismatch check, and a verdict. Pass --planned-n and --planned-days from the design doc so it can tell "significant" from "significant because you stopped early". A is always control.

Step 1: decide whether to test at all

Before designing anything, run the size command and look at the days-to-run figure. Decision rules:

• Under about 2 weeks to reach sample: test normally.
• 2 to 8 weeks: test, but only big swings (see low-traffic playbook). Commit to the full runtime before starting.
• Over 8 weeks: do not A/B test this. The market, your ads and your site will change more than the effect you are chasing. Use the no-test alternatives below.

The maths is brutal at typical founder traffic. Verified worked examples (alpha 0.05 two-sided, power 0.8, computed with the script):

Two lessons to apply: small refinements (a 5 to 10% lift) are untestable below tens of thousands of daily visitors, and high-baseline metrics (add-to-cart, open rate, click-through on a step) are far cheaper to test than purchase rate.

Sample size: the actual formula

Per arm, for a rate metric, two-sided. With alpha 0.05 the first z is 1.96, with power 0.8 the second is 0.84:

n = ( 1.96 * sqrt(2 * pbar * (1 - pbar)) + 0.84 * sqrt(p1(1-p1) + p2(1-p2)) )^2 / (p2 - p1)^2

where p1 is baseline, p2 = baseline + MDE, pbar = (p1 + p2) / 2. This is the standard two-proportion formula (pooled variance under the null, unpooled under the alternative). The script implements it; cross-checked against standard references (20% vs 25% gives 1,094 per arm, 10% vs 12% gives 3,841 per arm).

Quick mental version (Evan Miller's rule of thumb): n per arm is about 16 * variance / delta^2, with variance = pbar(1-pbar) for rates. It also works for revenue metrics with the measured variance of per-visitor revenue.

Fix alpha at 0.05 and power at 0.8 unless there is a stated reason. Choosing power 0.8 means: if the true effect equals your MDE, you still have a 20% chance of missing it. Do not let anyone lower power to make the sample "fit" the traffic; shrink ambition or raise the MDE instead.

Runtime rules

• Always run whole weeks: 7, 14, 21, 28 days. Day-of-week mix changes both traffic quality and conversion rate.
• Minimum 7 days even if sample is reached in 2. Maximum useful runtime about 6 to 8 weeks: beyond that, cookie churn and external drift poison the comparison.
• Decide runtime AND sample per arm in the design doc. The test ends when both are met, on the next whole week boundary. Not before, not "when it looks done".

Peeking: why checking daily creates fake winners

A fixed-horizon significance test is only valid once, at the planned end. Checking daily and stopping the first time p drops under 0.05 gives many chances for noise to cross the line. Evan Miller's worked example: a no-effect test evaluated at 5% significance after every observation up to 150 observations has a 26.1% false positive rate, five times the nominal 5%.

Rules:

• Default: fixed horizon. Look at counts mid-test for data-quality only (is tracking firing, is the split right). Do not look at the p-value until the planned end. The evaluate command enforces this: significant but short of plan returns KEEP RUNNING.
• If stakeholders must peek with the option to stop, spend alpha across peeks. Verified thresholds (to keep a true 5% false positive rate, require the reported significance to reach): 1 interim peek 2.9%, 2 peeks 2.2%, 5 peeks 1.4%, 10 peeks 1.0%.
• Sequential testing, in plain terms: methods like the mixture sequential probability ratio test (mSPRT) produce "always valid" p-values and confidence intervals that hold no matter when you stop (Johari, Pekelis and Walsh, arXiv:1512.04922). This is what serious platforms run so users can peek safely. If the user's tool has such an engine (see tools section), trust its stopping decision and do not layer fixed-horizon corrections on top. If the user is computing stats themselves from raw counts, they do not have this: use a fixed horizon.

Low-traffic playbook (most founders)

When the script says a normal test takes months:

1. Raise the MDE, honestly. Only test changes big enough that a +30 to +50% relative lift is plausible: a different offer, a rebuilt pricing page, a changed headline promise, removing a checkout step. If a +50% lift on the change is not even conceivable, the change is not worth a test at this traffic.
2. Test radical variants, not refinements. Button colours, word swaps and image crops are untestable below roughly 10,000 conversions per month. Two genuinely different pages can be told apart; two shades of the same page cannot.
3. Move the metric upstream (pooled metrics). Test on the highest-traffic step that plausibly proxies the real goal: add-to-cart instead of purchase, click-through to signup instead of activated account. Higher baseline plus more events = far smaller sample (see the 30% row in the table). State the proxy risk in the readout: an upstream win can fail to carry downstream.
4. Do not test; decide, then measure pre/post with guardrails. For many founder decisions the right move is: ship the better-argued version to 100%, compare 2 to 4 whole weeks before vs after, and pre-register guardrails (revenue per day, CPA, refund rate must not degrade more than an agreed amount). Label the result "observational, confounded by season and traffic mix", never "proven". Revert if a guardrail breaks. This trades statistical rigour for speed, which at 100 conversions a month is usually the right trade.
5. Never run more than one A/B test on the same audience and metric at once at low traffic; you will not have the sample to detect interactions, and each test steals the other's power.

Reading ad platform "winners" (Meta, checked Aug 2026)

What Meta's A/B test actually does: splits the audience between cells and crowns the version with the lowest cost per result on the key metric you chose, marked with a trophy icon. Meta's own docs note the cost per result can fluctuate while the test is active, recommend equal budgets per cell for a fair comparison, and say that tests with larger reach, longer schedules or higher budgets tend to deliver more statistically significant results. If there was not enough data, Ads Manager recommends extending the test.

How to judge one:

• Why 2-day tests lie: each cell's ad set needs about 50 optimisation events in a week to exit the learning phase (Meta Business Help Centre), so a 2-day test compares two ad sets that are both still in unstable learning delivery. On top of that, 7-day-click conversions from day 1 have not landed yet, and the raw counts are tiny. A 2-day "winner" is a coin flip with a trophy icon.
• Minimum standard before believing a Meta test: at least 7 full days, enough budget for roughly 50+ conversions per cell on the key metric (budget the conversions, then derive days and spend), equal budgets, and one variable changed.
• Always pull the raw numbers (results and impressions or reach per cell) and run evaluate on them yourself. Treat the trophy as a hypothesis, not a verdict.
• Cells are audience splits, so cross-cell contamination is limited, but conversions are attributed, not observed: attribution windows must be identical across cells or the comparison is void.

Revenue metrics vs rate metrics

Rate metrics (converted or not) are cheap: variance is capped at p(1-p). Revenue per visitor is expensive: mostly zeros plus a long tail of order values, so variance is huge and one whale order can flip a test.

Rules:

• Default to designing and judging on a rate metric. Then sanity-check revenue separately as conversion rate x average order value.
• If revenue per visitor must be the primary metric: get the per-visitor revenue standard deviation from real data (include the zeros), size with n = 16 * sd^2 / delta^2, and expect a much bigger n than the rate test.
• Pre-register outlier handling in the design doc: winsorise per-visitor revenue at the 99th percentile (cap values above it at that level), same rule for both arms. Deciding how to treat a whale after seeing which arm it landed in is result-shopping.
• In any revenue readout, report the largest single order per arm. If removing one order flips the sign of the lift, the verdict is KEEP RUNNING or NO SHIP, never SHIP.

The lies checklist

Run this against every result someone wants to ship. Each item alone invalidates a "winner".

1. Multiple comparisons. Ten variants against control at alpha 0.05 gives a 40% chance of at least one fake winner (1 - 0.95^10); three variants, 14%. Rule: with k variants, require p < 0.05/k (Bonferroni), or rerun the champion against control in a clean two-arm test.
2. Segment fishing. Checking 20 segments after an overall null gives a 64% chance some segment shows p < 0.05 by chance. A segment claim is only evidence if it was pre-registered in the design doc; otherwise it is a hypothesis for its own follow-up test.
3. Stopping at significance. If the stopping rule was "when it hit significance", the p-value is invalid (see peeking). Check --days and --planned-n against the plan.
4. Metric switching. The design doc names one primary metric. A win on a secondary metric after a primary null is a fresh hypothesis, not a result.
5. Sample ratio mismatch. If the split deviates from plan (evaluate flags p < 0.001 on the split), assignment or tracking is broken and the whole test is void, even a "significant" one. Common causes: redirect variant loses slow connections, bot filtering hits one arm, QA traffic in one arm.
6. Winner's curse. Even a clean significant result overstates the true effect, because tests that get lucky are the ones that cross the line. Project business impact from the low end of the confidence interval, not the observed lift.
7. Mid-test changes. Any change to targeting, budget (on ad platforms), page content or audience mid-test restarts the clock. Data from before the change is a different experiment.

Bayesian vs frequentist, and what tools report

Plain terms:

• Frequentist p-value: how surprising this data would be if there were truly no difference. It is NOT the probability that B is better. p = 0.04 does not mean 96% chance B wins.
• Bayesian probability to beat: the probability B is better than A given the data and a prior. More intuitive, no fixed horizon needed, but the number depends on the prior, and "95% chance to beat" on day 2 with 30 conversions still just means "small noisy sample leaning B".
• With big samples and neutral priors the two agree on the decision. Every fight about which is better matters less than sample size, pre-registration and honest stopping.

What tools run (verified Aug 2026): VWO's SmartStats engine is Bayesian, reporting outcome probabilities such as better than control, worse than control and practical equivalence, and handles peeking within its own framework. Optimizely's Stats Engine is frequentist sequential testing with false discovery rate control: its displayed significance is always-valid, built so users can peek. Rule: identify the user's tool first. Numbers from a sequential or Bayesian engine are peek-safe but not comparable to a fixed-horizon p-value; numbers from a plain significance calculator applied to a live dashboard are fixed-horizon stats being peeked at, which is the classic error.

Evaluating a test: workflow

Given test data, in order:

1. Get raw counts per arm (visitors or sessions, conversions), runtime in days, and the design doc numbers (planned n per arm, planned days, primary metric, MDE, variant count, pre-registered segments).
2. If no design doc exists, reconstruct honestly: ask what the stopping rule actually was, how many variants and metrics were looked at, and compute what MDE the achieved sample could detect (run size backwards by trying MDEs until n matches). Report that detectable-MDE alongside any verdict.
3. Run evaluate with all flags. Check SRM first; if flagged, stop, the test is void.
4. Run the lies checklist above.
5. Deliver one of four verdicts, with the confidence interval always quoted:
   • SHIP: significant, plan complete, checklist clean. Project impact from the low end of the CI.
   • NO SHIP (as a win): plan complete, not significant. If the CI excludes the MDE, say clearly: this change does not do what we hoped, decide on other grounds. If the CI still contains the MDE, the test was underpowered in practice; say that instead of "no effect".
   • KEEP RUNNING: not at planned sample or runtime, or significant-but-early. State the finish date and forbid further peeks.
   • INVALID: SRM, mid-test changes, attribution mismatch, or a stopping rule of "when it looked good". Name the flaw, design the rerun.

Output template: test design doc

# Test design: [name]
Date: [date]   Owner: [name]
Hypothesis: [change] will increase [primary metric] because [reason].
Primary metric: [one metric, e.g. signup rate per visitor]
Baseline: [x%] (source: [analytics, last 30 days])
MDE: [+y% relative], the smallest lift worth shipping for
Alpha 0.05 two-sided, power 0.80
Sample: [n] per arm, [2n] total (script output attached)
Traffic: [z]/day into test, split 50/50
Runtime: [d] days ([whole weeks]), ends [date]. Ends only when BOTH sample and runtime are met.
Stopping rule: no result peeks before end date. Mid-test looks are for tracking QA only.
Variants: control + [k]. If k > 1: significance threshold 0.05/[k].
Guardrails: [metric, e.g. AOV, refund rate] must not drop more than [x%].
Pre-registered segments: [none, or list]
Outlier rule (revenue only): winsorise per-visitor revenue at 99th percentile, both arms.

Output template: test readout

# Test readout: [name]
Ran [start] to [end], [d] days of [planned]. Sample [n1] vs [n2] (planned [n] per arm).
SRM check: [pass/fail, p=]
Result: A [c1]/[n1] = [r1%], B [c2]/[n2] = [r2%]
Lift: [+x.x%] relative ([+x.xx] points). p = [ ]. 95% CI: [lo, hi].
Checklist: peeking [ok/violated], multiple comparisons [k variants, threshold used],
segments [pre-registered only], metric [as designed], outliers [rule applied],
mid-test changes [none/listed].
VERDICT: [SHIP / NO SHIP / KEEP RUNNING / INVALID]
Basis: [two sentences: the statistics and the plan compliance]
Expected impact if shipped: [low end of CI] x [volume] = [conservative estimate]
Next: [ship and monitor guardrails / retest with bigger MDE / rerun clean]

Gotchas

• As of Aug 2026, Meta's public help pages for Ads Manager A/B tests publish no numeric minimum duration or confidence threshold: the docs describe the winner simply as lowest cost per result, recommend equal budgets, and suggest extending tests that lack data. Older advice quoting exact Meta-mandated day minimums or confidence percentages is not in the current docs; set your own standard (7+ days, 50+ conversions per cell, verify with your own significance run).
• Google Optimize was discontinued on 30 September 2023. Any workflow or content recommending it as the free testing tool is stale.
• Optimizely has reported always-valid sequential results with false discovery rate control since its 2015 Stats Engine. Applying peeking corrections or Bonferroni on top of its displayed significance double-counts the protection.
• VWO reports Bayesian probabilities (SmartStats), not p-values. Do not ask "is it significant at 0.05" of a VWO readout; judge its probability of best plus potential-loss style outputs against the decision's cost.
• Models trained on older CRO content over-recommend micro-tests. At 500 visitors/day and a 3% baseline, detecting a +5% relative lift needs about 208,000 visitors per arm, which is roughly 832 days: at founder traffic, button-colour testing is innumerate, not conservative.
• The learning phase figure used above (about 50 optimisation events in the week after the last significant edit) is Meta's current documented guidance as of Aug 2026; older sources citing different counts or windows are out of date.

Sources

• https://www.evanmiller.org/how-not-to-run-an-ab-test.html (peeking false positive inflation, 26.1% example, per-peek thresholds, 16 * variance / delta^2 rule)
• https://arxiv.org/abs/1512.04922 (Johari, Pekelis, Walsh: always-valid inference, mSPRT)
• https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/ (Stats Engine: sequential, FDR control)
• https://vwo.com/testing/ (SmartStats Bayesian engine and reported outcomes)
• https://www.facebook.com/business/help/1738164643098669 (about A/B testing, equal budgets)
• https://www.facebook.com/business/help/1376548572415613 (A/B results: lowest cost per result winner, extend recommendation)
• https://developers.facebook.com/docs/marketing-api/guides/split-testing/ (winner by efficiency metric, reach and budget vs significance)
• https://www.facebook.com/business/help/112167992830700 (learning phase, about 50 results in a week)
• https://support.google.com/optimize/answer/12979939 (Google Optimize sunset date)