Read the original at Lenny’s Newsletter ↗lennysnewsletter.com · subscriber post
By Lenny Rachitsky · lennysnewsletter.com · @lennysan on X · YouTube · LinkedIn
The post argues that experiments are usually the right default for product changes, but describes when to skip them. It covers the benefits and downsides of A/B testing, then three cases to ship without testing: when results would take too long to reach, when downside risk is low and setup effort is high, and when launching something with no control to compare against.
Subscriber post — summary only01Key takeaways
- Experiments give precise impact measurement, but they cost setup time and can encourage short-term, narrow thinking.
- Before testing, estimate how many users are needed to detect your expected change; at startup scale this is often too long.
- Lowering the required confidence level can shorten tests when speed matters and shipping an occasional slightly negative change is acceptable.
- Skip experiments for well-established best practices with low downside, especially when running them is slow and cumbersome.
- Making experimentation faster is often a better investment than running many slow tests.
- For brand-new products with no control, set independent success criteria instead of forcing an awkward experiment.
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.