Lenny’s Podcast96 min
The ultimate guide to A/B testing
Ronny Kohavi · Jul 27, 2023
Key learnings
- Test every code change, since even small bug fixes or tiny UI tweaks can have surprising, large impact on key metrics.
- Expect most ideas to fail: roughly two-thirds to over 90% of experiments in his experience did not move the target metric, so…
- Allocate a portion of experiments to high-risk, high-reward bets, accepting that about 80% will fail, while keeping most work…