Small Lift Labs — field notesWorking notes on experiment design, sample size, measurement and the ways a test result goes wrong. Every number has a source and every method has its workings attached.https://smallliftlabs.com/en-usAttribution said the ads returned 4,173%. The experiment on the same spend said minus 63%.https://smallliftlabs.com/blog/attribution-said-4173-percent/https://smallliftlabs.com/blog/attribution-said-4173-percent/eBay ran the experiment, published it, and the gap between the dashboard and the truth was about forty times. Then I sized a holdout for a normal advertiser and found most of them cannot run one.Fri, 14 Aug 2026 00:00:00 GMTMeasurementDana WhitlockI sized a checkout test for a client and the answer was thirty months. Most shops cannot test their checkout at all.https://smallliftlabs.com/blog/checkout-test-you-cannot-run/https://smallliftlabs.com/blog/checkout-test-you-cannot-run/A 5% lift on checkout completion needs about 15,000 sessions per variant. At a thousand checkout starts a month that is two and a half years, which is longer than the checkout will exist.Thu, 13 Aug 2026 00:00:00 GMTCheckoutDana WhitlockI ran 200,000 tests with no winner in them. A daily peek found a winner in 27 of every 100.https://smallliftlabs.com/blog/peeking-costs-27-percent/https://smallliftlabs.com/blog/peeking-costs-27-percent/The false positive rate is supposed to sit at 5%. I measured what happens to it when somebody opens the dashboard every morning, and the honest answer is 27.44%. Then I found out that a bigger site does not help.Wed, 12 Aug 2026 00:00:00 GMTMethodDana Whitlock