By Teresa Torres · producttalk.org · @ttorres on X · LinkedIn
Teresa Torres argues that a single A/B test rarely settles a question; instead, teams should string related hypotheses together around one underlying insight. Using an email subject-line example about urgency, she shows how varied tactics (expiration dates, limited rewards, calls to action, missed opportunities) each test the same core idea. Passing and failing results reveal nuances across segments and over time, moving teams from knowing a tactic to understanding the why. This matters because it turns experimentation into a compounding body of knowledge specific to a product's context and audience.
01Key takeaways
- Test a family of related hypotheses around one insight rather than a single tactic.
- Use passing and failing tests to map where an insight holds across segments.
- Prioritize understanding the why behind results over memorizing individual tactics.
- Expect tactics to lose potency with repetition and validate that in your own context.
- Build a cumulative knowledge base about your audience through ongoing, connected experiments.
02Key sections
- Testing related hypotheses
- Torres shows that one insight, such as urgency boosting open rates, can be tested through many different tactics. Each variation probes the same underlying idea from a different angle.
- Learning the nuances
- Results differ by segment, over time, and by tactic, so testing a family of hypotheses reveals when and for whom an insight holds. This is where real learning happens.
- From tactics to the why
- Knowing that a tactic works is less valuable than understanding the reason it works. Deeper understanding comes from exploring what works and what doesn't across contexts.
- Run your own tests
- The conclusions in the example are illustrative, not universal. Every context and segment is different, so teams must validate patterns with their own experiments.
- Intellectual honesty
- Torres closes with a Feynman quote on the rigor needed to truly know something and the ease of fooling oneself. She asks readers what they actually know about their audience.
03From the post
“In the last post we ended by concluding that the results of many A/B tests lead to further questions. This is a good thing. Let's take a look at how you can string A/B tests together to build up a knowledge base over time. Let's continue”
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.