Everybody has a guess about that step, and the guess is usually the part of the site that is easiest to redesign. We measure which step is actually costing you, then run the tests that move it, and we tell you in writing when a test cannot answer the question.
Simulated run, not a client result. Baseline 3.24%, true lift 6.8%, 5,800 visitors per variant per day, two sided 95% interval on the relative difference. The interval is recalculated from the sample collected so far, exactly as it would be in a live test.
Every number below comes from people who ran experiments at scale and wrote down what happened. None of them come from us, and every one has a link.
The fourth number is the one that costs money. If you run twenty tests where nothing is really happening, one of them will still look like a win at the usual threshold. Peek at the dashboard every morning and that rate climbs well past one in twenty.
Three ways we work, depending on whether the problem is what you are testing, whether the numbers can be trusted, or whether one particular result is about to be believed.
A sized backlog, a stopping day agreed before the code exists, and one page per readout. Including the readouts that say no.
Most disputed results are a tracking problem wearing a statistics costume. We rebuild the event layer so the arguing stops.
A 12% lift is about to become a roadmap. We check the sample, the segments and the stopping point first.
Move the four sliders. The tool gives you the sample you need per variant and how long your traffic takes to produce it. Nothing is sent anywhere until you ask for the plan.
Two proportion comparison, two sided. Per variant sample is n = (zα/2·√(2·p̄·q̄) + zβ·√(p₁q₁ + p₂q₂))² / (p₂ − p₁)², where p₁ is your current rate, p₂ is that rate lifted by the effect you chose, and p̄ is their average. Traffic is split evenly across variants. Duration rounds up to whole days, and we round again to whole weeks in the advice because partial weeks bias a test toward whichever days it happens to catch. This is our reading of standard practice, not advice for your specific case, so check it with your own analyst before you commit a quarter to it.
eBay ran the experiment, published it, and the gap between the dashboard and the truth was about forty times. Then I sized a holdout for a normal advertiser and found most of them cannot run one.
A 5% lift on checkout completion needs about 15,000 sessions per variant. At a thousand checkout starts a month that is two and a half years, which is longer than the checkout will exist.
The false positive rate is supposed to sit at 5%. I measured what happens to it when somebody opens the dashboard every morning, and the honest answer is 27.44%. Then I found out that a bigger site does not help.
Send the change you are planning and the traffic you get. We will tell you whether the test can answer the question at all, and we will say so plainly when it cannot.