We write these for ourselves first. Every number has a source, every method has the workings attached, and when we got something wrong we say so in the note rather than quietly editing it.
eBay ran the experiment, published it, and the gap between the dashboard and the truth was about forty times. Then I sized a holdout for a normal advertiser and found most of them cannot run one.
A 5% lift on checkout completion needs about 15,000 sessions per variant. At a thousand checkout starts a month that is two and a half years, which is longer than the checkout will exist.
The false positive rate is supposed to sit at 5%. I measured what happens to it when somebody opens the dashboard every morning, and the honest answer is 27.44%. Then I found out that a bigger site does not help.