How to Choose Your Next Experiments

Teresa Torres argues that a common mistake among teams new to Lean Startup methods is substituting experimentation for judgment, testing options like folders versus tags without first reasoning about which is most likely to work. She contends that Lean Startup tools exist to test and refine judgment, not replace it. Product people bring knowledge and experience, and should reason toward strong solutions while staying appropriately doubtful until assumptions are tested. Like scientists who use insight to pick the most promising experiments rather than testing every permutation, teams should select experiments deliberately. The piece closes by previewing a session on a 'mapping the challenge' exercise for exploring the problem space and its goals and constraints to choose the next experiment.

01Key takeaways

• Don't default to running an experiment when a reasoned decision is available; use judgment first.
• Use your domain knowledge to propose the most plausible solution before testing it.
• Stay appropriately doubtful about your assumptions long enough to design a meaningful test.
• Prioritize experiments based on insight rather than testing every option you can think of.
• Map the problem space, including goals and constraints, to decide which experiment to run next.

02Key sections

The common trap

Teams new to Lean Startup often reach for an experiment immediately when two options are debated. The author calls this an expensive way to avoid making a reasoned choice.

Experimentation tests judgment

The Lean Startup should sharpen reasoning rather than replace it. Practitioners still bring domain knowledge and should use it to propose good solutions.

Maintaining productive doubt

Rather than charging ahead with certainty, teams should pause and test their assumptions before committing to a solution.

Choosing the most promising experiments

Like scientists who use insight to narrow the field, product teams should pick experiments strategically instead of testing every possibility.

Mapping the challenge

The author previews an exercise for exploring the problem space, clarifying goals and constraints so teams can better choose their next experiments.

03From the post

““I think we should use folders.” “I think we should use tags.” “I know. Let’s test them both.” Does this sound familiar? I bet it does. By far the most common mistake teams make when they are new to Lean Startup methodologies is they fall into the trap of”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.