A/B testing: the stat-sig problem

Post source: https://4thwithme.dev/blog/ab-testing-stat-sig-problem/

[ THE SETUP ]

Generic scenario, no company specifics: you run an ecommerce storefront. Someone ships a change, a re-ranked product grid, a new checkout button color, doesn't matter. You split traffic 50/50, wait for data, and want to know: did this actually help, or did it just look like it helped?

That's the whole job of an A/B test. It answers exactly one question: is there a difference between the two groups, yes or no. It does not tell you why. It does not reliably tell you how much the metric will move once you roll out to everyone. It rejects a null hypothesis or it fails to reject it. That's the entire output. Everything else people read into a test result, they're reading in themselves.

[ DEF.1 ]

Null hypothesis (H₀) — the default, skeptical assumption that nothing changed: there is no real difference between A and B. An A/B test is built entirely to argue against this one claim. It either rejects it (evidence a difference exists) or fails to reject it (not enough evidence either way). It never proves the opposite outright, it just runs out of reasons to doubt it.

the assumption every A/B test starts by trying to disprove

[ WHAT "STAT SIG" ACTUALLY MEANS ]

Here's the misunderstanding almost everyone ships with: a p-value is not "the probability that B is better than A." It's the probability of seeing data this extreme (or more extreme) if there were actually no difference at all. It's a statement about the data given no effect, not a statement about the effect given the data. Those sound similar. They are not the same claim, and the difference matters every time you're deciding whether to trust a result.

Put another way: a p-value doesn't tell you "B is probably better." It tells you "if B and A were actually identical, how surprising would this result be?" A low p-value means the result would be surprising under the assumption of no real difference — that assumption starts to look shaky, so it's worth trusting. It does not tell you the odds that B is actually better; that's a different question the test never answers. It's the same logic as saying "if this coin were fair, getting 9 heads in 10 flips would be surprising" — that doesn't prove the coin is rigged, it just means the fair-coin explanation is hard to believe.

[ DEF.2 ]

P-value — the probability of seeing a result at least this extreme, assuming the null hypothesis is true, assuming there's actually no difference between A and B. It is not the probability that B beats A, and it is not the probability that the null hypothesis itself is true. A small p-value just means "this would be a strange coincidence if nothing had actually changed," nothing more.

a statement about the data given no effect, not about the effect given the data

To get to a p-value you first have to set up two competing claims:

• Null hypothesis (H₀): there is no difference between A and B. This is always the same claim, every single time.
• Alternative hypothesis (H₁): there is a difference.

An A/B test never proves H₁. It only ever rejects H₀ or fails to reject it. Failing to reject H₀ doesn't mean "there's no effect," it means "we didn't detect one at the sensitivity we set up for." That distinction alone would save a lot of bad blog posts about "our test failed, this feature doesn't work."

Then there are two ways to be wrong, and they have genuinely useful mnemonics:

[ DEF.3 ]

Type I error (false positive) — you see a difference that isn't really there. Telling grandpa he's pregnant.
Type II error (false negative) — you miss a difference that is really there. Telling a pregnant woman she isn't.

the two ways an A/B test can lie to you

The industry default is 5% tolerance for Type I (α = 0.05) and 20% tolerance for Type II (β = 0.20, i.e. 80% power). Neither number is handed down by nature. They're both choices you make based on the cost of being wrong. A drug trial might demand α with eight zeroes before the first significant digit. A button-color test where you genuinely don't care much either way could reasonably run at α = 0.20. The 5% convention is a convention, not a law.

Here's how the p-value and alpha actually connect: alpha is the surprise threshold you commit to before you look at any data — your pre-agreed tolerance for crying wolf. The p-value is how surprising your actual result turned out to be. The decision rule is just a comparison: p-value < alpha means "significant." That's it. And alpha isn't just a cutoff, it IS your Type I error rate — a 5% alpha means you've pre-accepted a 5% chance of telling grandpa he's pregnant, across all the times you'd run this test. Type II (missing a real difference) isn't controlled by alpha at all — that's a separate knob, beta, driven mostly by how much data you collect.

Beta works the opposite way from alpha: it's the rate at which you fail to catch a difference that's actually there — telling the pregnant woman she isn't. Beta isn't a tolerance you set directly like alpha; it falls out of how much data you collect, given the effect size you're trying to detect. Small sample, small true effect: beta is high, meaning you'll miss it most of the time and walk away concluding "no difference" when there really was one. More data pulls beta down. Power is just the flip side of the same number: power = 1 − β, so the industry-default β = 0.20 is the same fact stated as "80% power" — an 80% chance of actually seeing the effect if it's real, and a 20% chance of missing it outright.

[ THE FOUR NUMBERS THAT SET THE BAR ]

Before you run anything, four inputs determine how much data you actually need: your alpha threshold, your power, your baseline conversion rate, and your MDE, minimum detectable effect, the smallest lift you actually care about being able to see.

[ DEF.4 ]

MDE (minimum detectable effect) — the smallest lift you actually care about being able to see. It's not a property of the data, it's a choice you make going in: below this size, a real effect might exist and your test still won't reliably catch it. Set it too small and your required sample size balloons past what your traffic can deliver in a reasonable timeframe; set it too large and you'll miss smaller wins that were real.

the smallest lift you've decided is worth being able to see

Change any one of these and your required sample size moves, usually a lot more than intuition suggests. Sample size explodes as your baseline moves away from 50%, or as your MDE shrinks. Concretely, at a 10% baseline conversion rate with the standard 5%/80% thresholds:

FIG.1 — going from "detect a 10% lift" to "detect a 1% lift" costs you 100x the traffic

Going from detecting a 10% lift to detecting a 1% lift costs you roughly one hundred times the traffic. This is why small-traffic products shouldn't be running color-tweak A/B tests: they simply don't have the volume to detect small effects reliably, and running one anyway doesn't make the math work, it just produces a noisy coin flip dressed up as a decision. Where does MDE actually come from in practice? There's no clean formula. In order of how people actually do it: gut feel on plausibility, historical results from similar past tests, or, the more honest method, work backwards from your constraints. How much traffic do you actually have, how many hypotheses are sitting in the backlog, how fast do you need to decide. Set your MDE from what's actually testable in your time and traffic budget, not from an abstract target someone wrote on a roadmap slide.

[ USING THE CALCULATOR: TWO EXAMPLES ]

You don't need to memorize the sample size formula to use any of this. Evan Miller's sample size calculator does the arithmetic for you: plug in your baseline conversion rate and your minimum detectable effect, and it hands you a number per variation. The one setting that trips people up is the toggle between Absolute and Relative. Absolute means "detect a change of this many percentage points" (a 20% baseline with a 5-point absolute MDE means detecting 20% → 25%). Relative means "detect a change of this many percent of the baseline itself" (a 20% baseline with a 10% relative MDE means detecting 20% → 22%, since 10% of 20 is 2). Relative is almost always the more honest way to think about it, because the same absolute point-move means something very different at a 2% baseline than at a 50% one. Here are two examples run through the same tool, same 5%/80% thresholds, to show how differently the same math treats two real spots on a storefront.

Example 1: top of the funnel. Say you're changing the color of the main call-to-action button on the homepage, high-traffic real estate that basically every visitor sees. Baseline click-through is 20%, and you want to detect at least a 10% relative lift, meaning you'd notice a move to roughly 22% or higher.

FIG.2 — 20% baseline, 10% relative MDE → 6,347 per variation

That's 6,347 per variation, 12,694 total. If the homepage does something like 3,000 visits a day, split 50/50, that's 1,500 per arm per day, so you'd clear the bar in well under a week. The strategy here is straightforward because the traffic supports it: pick a real, meaningful MDE, run a standard fixed-horizon test, wait for the number, done.

Example 2: deep in the site, low traffic. Now say the change is on an advanced export feature buried three clicks into account settings, the kind of thing only a small slice of engaged users ever reaches. Baseline conversion on the action you're watching (say, clicking "export") is 3%, and because the effect would need to be large to matter at this baseline, you set a generous 20% relative MDE, detecting a move to roughly 3.6% or higher.

FIG.3 — 3% baseline, 20% relative MDE (already a generous ask) → 13,050 per variation

Even with an MDE more than double the first example's, you need 13,050 per variation, more than double the homepage number, because the low baseline is working against you the whole time. If that settings page gets 300 visits a week, that's 150 per arm per week. Reaching 13,050 per arm at that rate takes roughly 87 weeks, about a year and a half. That's not a test, that's a career milestone.

The two examples share a calculator and a formula. They don't share a strategy. For the homepage button, run the standard test, it's cheap and fast. For the deep feature, a fixed-horizon 50/50 split is very likely the wrong tool entirely: either loosen your MDE further until the sample size matches your actual traffic (accepting you can only detect huge swings), or stop trying to force a classic A/B test onto traffic that can't support one, and reach for one of the alternatives from the next post in this series instead, a long-running hold-out, a qualitative read, or a Bayesian approach that doesn't demand a pre-fixed sample size to say something useful early.

[ THE PROBLEM: PEEKING ]

Here's where the dashboard-refreshing habit turns into an actual statistical problem, not just an impatience problem. Refreshing the page doesn't corrupt the underlying data, the numbers are whatever they are regardless of who's watching, but it does change your stopping rule. "Stop and declare a winner the moment p < 0.05" isn't one test, it's a test you get to retry every time you check.

And "check" has a specific meaning here: computing the p-value and using it to decide whether to stop, not just glancing at a chart. Opening the dashboard to see where the numbers stand is harmless on its own. It's treating that p-value as a decision point, "is this under 0.05, should I call it," that counts as a look.

Your 5% alpha threshold is a guarantee about exactly one such decision point, made after your pre-calculated sample size is reached. Every time you check the result before that point and treat "p < 0.05 right now" as a stopping signal, you're giving yourself another shot at the threshold, another roll of the dice for a false alarm, no new data required. Do that ten times over the course of an experiment and your real false-positive rate isn't 5% anymore. It's roughly 4x that, around 20%.

Here's the simple version of why. A single 5%-threshold look means a 5% chance of a false alarm, and a 95% chance of correctly seeing nothing when there's nothing there. That 95% is your "safe" probability for one look. Now take another look. If those looks were fully independent, both looks correctly showing nothing would need 95% and then 95% again, multiplying down toward zero the more times you check, ten independent looks would already put you near a 40% false-alarm rate.

Real peeking lands lower than that naive multiplication, because your looks aren't independent, each one is checking the same accumulating data as the last, just with a few more rows added. But the direction holds and the size is still large: more looks, more chances to get unlucky, and nobody told the dashboard to warn you which chance you're on.

FIG.4 — your chosen 5% only holds if you look exactly once

Intuitively: a p-value during a running test isn't a fixed property of your data, it's a noisy quantity that jumps around as samples accumulate. It can dip under 0.05 early by pure chance, climb back over, dip again. If you stop the moment it happens to be favorable, you're selectively sampling the noise, not the effect. The p-value only reflects the error rate you actually chose once the pre-calculated full sample size is reached. Everything before that is a preview, not a verdict, no matter how green the dashboard looks.

This doesn't mean the textbook rule, "never look, always wait for full sample size," is the only correct answer. It's correct if you need that specific 5%/20% guarantee, which you usually do in regulated or high-stakes contexts. It's adjustable if you're willing to trade away some of that precision on purpose: agree on a looser threshold up front, or use a testing method actually designed to handle repeated looks (more on that in a minute). What's not adjustable is pretending you didn't peek when you did, and reporting the original 5% anyway.

[ WHY IT KEEPS HAPPENING ]

This isn't a knowledge problem. Most people who work near experimentation can recite "don't peek" if you ask them directly. It keeps happening anyway, because the incentives in the room are almost never aligned with statistical patience.

A test trending toward a strong result after one week, against a five-week pre-registered horizon, puts real pressure on the room: waiting the textbook full duration for rigor has an actual cost in forgone revenue or forgone learning speed. A test trending the wrong direction creates the mirror pressure: cutting losses early feels responsible, and also inflates your error rate exactly the same way a favorable early peek does. Shipping pressure doesn't care which direction the needle is moving. It pushes toward stopping early either way.

The organizational failure mode compounds from here. A single test tells you a difference was detected, not the true magnitude of that difference. Teams that build quarterly OKRs by summing the observed uplifts of "winning" tests are stacking a series of noisy, possibly-inflated point estimates and treating the sum as a forecast. It rarely survives contact with the next two quarters. The statistics didn't fail here. The organization asked the statistics a question they were never built to answer.

[ ALTERNATIVES, BRIEFLY ]

None of this means fixed-horizon testing is broken or that peeking is unforgivable. It means the standard frequentist test answers one specific question, "is there a detectable difference at a preset error tolerance," and if that's not actually the question you're trying to answer, there are other tools built for the question you do have.

Sequential testing and always-valid p-values (mSPRT, group sequential designs) are built specifically to let you look as often as you want without the error-rate blowup, by spending your error budget across looks instead of assuming a single look. Bayesian A/B testing reframes the whole question away from p-values entirely, toward "what's the probability B is actually better, and by how much," and it doesn't require a fixed sample size to interpret at all. Both of these are real, useful, and deserve more than a paragraph each, which is exactly why they're getting their own posts in this series rather than a rushed footnote here.

[ THE MANAGERIAL PROBLEM ]

Even once the math is right, there's a second problem that's arguably harder: communicating a probabilistic result to a stakeholder who wants a yes-or-no answer by end of day.

"We're 87% confident this is an improvement, with a plausible range of 0 to 4%" is a true, useful sentence. It is also not the sentence most rooms want to hear. The fix isn't to round it down into a fake yes-or-no, it's to make the underlying decision structure explicit before you ever run the test, so uncertainty has somewhere to land. A workable hypothesis has four parts: if we do X, then Y will change by Z, because W. X is the change, Y is the metric you're watching, Z is the effect size you actually expect (which is also what feeds your sample size calculation), and W is your causal reasoning for why you expect it. W is the part almost everyone skips, and it's the part that compounds: a single test only tells you that something changed. Your working theory of why is what turns one test into a body of knowledge instead of an isolated anecdote you can't reuse.

[ WHAT WE ACTUALLY DO NOW ]

The practical version of all of this, generalized, not tied to any one team's exact setup:

• Pre-register your sample size before you look at results, driven by your actual alpha, power, baseline, and MDE, not by vibes
• If you must look early, decide that up front and either use a method designed for it (sequential testing, Bayesian) or explicitly accept a looser real error rate
• Run continuous A/A tests on any traffic that isn't allocated to something real, they're nearly free and they catch splitter and tracking bugs before those bugs corrupt an actual test
• Write the X/Y/Z/W hypothesis down before launch, so "why" survives even when the test result is ambiguous
• Treat a null result as information, not failure, log it and move to the next hypothesis instead of quietly re-running until something turns green

None of this eliminates uncertainty. It just makes sure the uncertainty you're shipping with is the uncertainty you actually chose, instead of one that quietly grew while nobody was looking.

[ USEFUL LINKS ]

• Evan Miller's A/B testing sample size calculator — the fastest gut-check on whether you have anywhere near enough traffic before you run anything
• "The ASA Statement on p-Values: Context, Process, and Purpose" (Wasserstein & Lazar, The American Statistician, 2016) — the closest thing statistics has to an official correction of the p-value misunderstanding in this post, worth searching up directly
• mSPRT / "always-valid p-values" (Johari, Koomen, Pekelis, Walsh) — the paper behind modern sequential testing at scale, more detail in the next post in this series

[ ELSEWHERE ]

GitHub · LinkedIn · Dev.to · Substack · 4thwithme.dev/blog

May the --force be with you. See you next week.