Why You Aren’t Learning As Much As You Could From Your Experiments

Teresa Torres argues that teams use A/B tests poorly, mainly to confirm favored ideas or to blindly try variations, which teaches them little. Drawing on John Dewey's account of reflective thinking, she describes a double movement between inductive discovery (forming a general theory from observed facts) and deductive testing (checking that theory against new cases). She illustrates this with Katy Milkman's 'Fresh Start' theory, which explains New Year's goal-setting and predicts similar behavior at other milestones. The essay matters because it shifts experimentation from scattered tests toward a systematic, theory-driven search that produces reusable learning.

01Key takeaways

• Form an explicit theory from observed facts before designing experiments, rather than jumping straight to a tactic.
• Question your first idea and look for additional factors before accepting a conclusion.
• Derive what else your theory should predict, then design tests for those predictions.
• Use experiment results to revise the general theory, not just to judge one specific tactic.
• Repeat the cycle of inductive discovery followed by deductive testing to build a systematic search.

02Key sections

The problem with how we experiment

Teams either use A/B tests only to gate launches or test variables at random, and both approaches limit what they learn. Most experiments end up validating a favored idea instead of testing it.

Dewey's double movement of reflection

Dewey describes thinking as moving from raw facts to a suggested meaning (induction), then back to particular facts to test that meaning (deduction). Good thinking requires both movements and sustained doubt.

Applying it to product experiments

Using a New Year's app-marketing example, the author shows that jumping straight to tactics like emails and ads teaches only whether those specific tactics worked.

Milkman's Fresh Start theory

Milkman moved from the observed New Year's pattern to a general theory of fresh starts, then deduced and tested predictions using Google search and gym attendance data at other milestones.

Making theories explicit

Teams rarely state their theories, so they cannot revise them when data arrives. Making the theory explicit lets experiments refine it and reduces wasted effort.

03From the post

“I love that A/B testing has become so popular. It’s an important tool in our toolbox. But it frustrates me that we use it so poorly. Most teams make one of the following two mistakes: They either gate releases with an A/B test trying to understand whether”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.