The ultimate guide to A/B testing

Ronny Kohavi, a longtime experimentation leader who has run experimentation at Amazon, Microsoft (Bing), and Airbnb, and author of the book Trustworthy Online Controlled Experiments, discusses how to run A/B tests well. The conversation covers surprising wins and failures, typical failure rates, choosing an overall evaluation criterion, statistical pitfalls like sample ratio mismatch and misreading p-values, and how to build an experimentation culture and platform.

01Key learnings

• Test every code change, since even small bug fixes or tiny UI tweaks can have surprising, large impact on key metrics.
• Expect most ideas to fail: roughly two-thirds to over 90% of experiments in his experience did not move the target metric, so plan accordingly.
• Allocate a portion of experiments to high-risk, high-reward bets, accepting that about 80% will fail, while keeping most work incremental.
• Define an overall evaluation criterion that pairs the business metric with guardrail metrics and long-term value, so short-term gains that hurt the user experience don't win.
• Check for sample ratio mismatch on every experiment; he found around 8% of experiments at Microsoft were invalid, so a mismatch is a red flag to investigate.
• Treat unexpectedly large positive results skeptically (Twyman's law) and replicate and double-check before celebrating, since most such outliers turn out to have flaws.
• Avoid stopping experiments early on a real-time significance threshold, which inflates false positives; interpret p-values carefully, since a p of 0.05 can still carry a high false-positive risk.
• Document surprising wins and losses in a searchable institutional memory and review them regularly so the organization learns and doesn't forget.

02Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.