I ran 200,000 tests with no winner in them. A daily peek found a winner in 27 of every 100.

Last spring I told a client to ship a checkout change on the fourth morning of a test. The dashboard said the lift was significant, the number was fat, the room had been waiting a fortnight for something to celebrate, and I was the only person in the building whose job it was to say that four days of data is not enough to know anything, which is a thing you can say once before people start scheduling meetings without you. So I did not say it. Six weeks later the quarterly figures came back flat.

Nobody blamed me. That was the worst part, because it meant nobody learned anything either.

Here is the advice up front, so you do not have to read to the end for it. You should never let the daily look decide when a test stops. Always write the stopping day into the ticket before the code exists. Make sure the person who wants a daily update gets an interval and not a green tick. Do not treat a weekly check as the safe compromise, because it is not one.

That is the whole recommendation. Everything below is me finding out what the old habit was costing, because I did not want another argument about statistics. Those get settled by whoever talks longest and sounds most tired.

I wanted one number. So I built the smallest thing that could produce one, which took most of a Tuesday afternoon.

What I actually ran

The simulation contains no winner. Both variants convert at exactly 3.2%. There is nothing to find, because I did not put anything there. Each variant gets 5,800 visitors a day, the test runs 28 days, and I repeated the whole thing 200,000 times with a fixed seed so anyone can rerun it and land on the same figures.

Every simulated day I ran the same two sided z test on two proportions that your testing tool runs. Pooled variance, threshold at 0.05, nothing clever about any of it.

Then I asked the same data what happens under different habits of looking at it.

The control came first. Look only once, on the final day, and a test with no effect in it came back significant in 4.98% of runs, which is the 5% that the whole method has been promising you since somebody first put a green tick on a dashboard. That is the number working correctly. Getting it out of my own harness is the only reason I believe anything else the harness told me afterwards.

Now the habit everybody actually has. Look every morning, stop at the first significant reading, and the rate went to 27.44%.

The compromise most teams think they already made is a weekly look. That produced 12.60%.

An aside about that seed, which has nothing to do with any of this. It is the date I ran the job. I used to pick seeds at random. Then I would lose track of which run made which chart. A colleague at the next desk watched me do that twice. She said “just use the date, you are not a cryptographer”. Anyway. Back to the numbers.

The theory I held for about an hour, which was wrong

My first assumption was that this is a small site problem. Low traffic means a jumpy estimate, a jumpy estimate means more crossings, and the fix is simply more visitors. That felt obviously true. It also let me keep believing that the larger accounts I work on were mostly fine.

It is wrong. I reran everything at four traffic levels and the daily peek rate barely moved.

Sixty thousand visitors a day per variant is a serious business. It buys nothing here.

Traffic tightens the interval. That is all it does. A tighter interval does not reduce the number of separate occasions on which the running line can wander across the threshold, and the number of occasions is the thing that decides how often a test with nothing in it eventually produces a result somebody is willing to ship. You get a calmer test. Then you interrupt it just as often. Twenty eight chances stay twenty eight chances.

That result changed how I talk about this. It is not a startup problem. It is an everybody problem, and the big accounts are arguably worse off, because their tests feel more trustworthy while carrying an identical error rate.

Why the number moves so far

The 5% is not a property of your data. It belongs to the procedure, and the procedure includes the moment you decide to stop.

The running estimate wanders. Early on it wanders a long way, because the sample is thin and the interval around it is enormous. One look gives that wandering line a single chance to sit past the threshold on that particular day. Twenty eight looks give it twenty eight chances, and the line only needs to clear the bar once, because that is the morning somebody screenshots the dashboard and drops it into the channel with three exclamation marks.

None of this is new. Walsh, Johari and Pekelis wrote the paper for KDD in 2017. They open with the flat observation that users of A/B testing software are, in their words, “known to continuously monitor these measures as the experiment is running”. Their answer was to publish p-values that stay valid under exactly that behaviour, which Optimizely then shipped to customers.

I did not want their fix. I wanted the size of the hole the fix was built to plug, measured on numbers I generated myself, because a borrowed number never survives a meeting.

The second bill, which is bigger than the first

The error rate is only half the cost. The other half is what the false winner claims to be worth.

Among the runs where daily looking produced a false winner, the median reported lift was 9.7%. The ninetieth percentile was 23.1%. The true effect in every one of those runs was zero.

The reason is close to mechanical. You stop at the first moment the gap is wide enough to clear the bar. Wide gaps early in a test are noise. So the figure you write down is the top of a swing rather than a settled value, which means the decision to stop and the size of the effect are both being chosen by the same coin flip, and the one that survives into the slide deck is always the flattering half of that pair. The unflattering half never gets screenshotted.

Then somebody multiplies that 9.7% by annual revenue. It goes into a roadmap. Two quarters later it becomes a number that everybody has forgotten the origin of, and it is still sitting in the plan, quietly justifying the next thing.

Timing made it worse than I expected. Of those false winners, 60.6% showed up inside the first week and the median first firing was day five. The quarter that fired earliest did so by day two. Day five is exactly when a test starts to feel ready to a room that has been waiting for it.

I keep thinking about the day five figure. It lines up with almost every premature rollout I have argued against, including the one I lost. The head of product in that meeting told me the direction was obvious by then. He was reading day four of a twenty eight day test, and I did not have this table to put in front of him.

What changed in how I work

The stopping day goes into the ticket before any code exists. Sample size gives a count of visitors per variant, current traffic turns that count into a date, and the date sits beside the metric. It takes ten minutes. It also removes the most expensive decision in the process from the one moment when everybody is emotionally invested in a particular answer.

Tests run at least one full week, then whole weeks after that. Kohavi and Longbotham put this plainly in 2015, calling one full week a best practice so the test catches a whole weekly cycle, with multiple weeks beyond that. A test that ends on a Thursday has handed Thursday a vote it never earned.

Traffic gets split evenly. The same paper notes that a 50% split gives an experiment its maximum statistical power. Ramping to 10% on day one is sensible. Leaving it there for three weeks buys a longer test and a worse mood.

The health check and the decision are now separate jobs on separate screens. Somebody does need to watch a live test, so error rates, sample ratio, page timing and the revenue guardrail get looked at daily. The primary metric gets looked at on the agreed date.

The daily look was never the sin. Letting the daily look choose the stopping point is the sin, and once the two screens are apart the temptation mostly evaporates.

When a stakeholder wants a number before the date, they get the interval instead of the point estimate. No green tick. On day three that interval is embarrassingly wide, and the picture ends the conversation faster than any explanation I have ever tried, partly because it does not ask anyone to understand a p-value and partly because a bar that stretches from minus nine to plus twenty two is self evidently not a decision. One head of growth looked at it. She said “so we genuinely do not know yet”. That is the lesson in six words.

If early stopping is genuinely required, I change the method rather than leaning on discipline. Sequential designs and always valid p-values exist for this. They are deliberately conservative early and they keep their promise under continuous monitoring. Peeking at a fixed horizon p-value is not a shortcut. It is a different test with a worse error rate, and it will not tell you that it has become one.

The bit I have not settled

One result still bothers me. Twelve point six percent is roughly what a careful weekly checking team is running at, and every one of those teams believes the number is five. Nothing in the tooling says otherwise.

I do not know what honest disclosure would look like. A tool could count the looks taken and widen the threshold to match, and a handful now do. Most still print one p-value as though nobody in the building has ever refreshed the page.

What I still want is the real distribution of looks per test inside an ordinary marketing team. My simulation assumes a perfectly regular peeker who checks once a day at the same hour. I have never met that person. The real habit is bursty, it clusters around launch day and around whenever the weekly meeting falls, and my guess is that the true rate sits above 27.44% rather than below it.

That is the next thing I plan to measure. I still do not know how far off my 27.44% is. I would rather measure it than keep guessing out loud.

Questions people ask about this