The Dos and Don'ts of Hypothesis Testing

Teresa Torres argues that many product managers run experiments poorly: they struggle to choose what to test, decide how long to run tests, lack scientific training, find tools hard to use, and overgeneralize results. Although experimentation is spreading, her point is that research methods must improve for product decisions to get better. The piece is a short introduction to a talk she gave at a Lean Product and UX meetup, drawing on her coaching work with product teams. It matters because it frames hypothesis testing as a skill that many teams are still learning, not just a practice they have adopted.

01Key takeaways

• Experimentation adoption alone does not guarantee better product decisions; research methods need improvement.
• Decide deliberately what to test, since choosing the wrong experiment wastes effort.
• Set a reasoned duration for tests rather than stopping arbitrarily.
• Build basic scientific thinking, because most product managers are not trained as scientists.
• Avoid overgeneralizing from a single experiment's results.
• Learn from the common mistakes other teams have already made.

02Key sections

The growing popularity of experimentation

Experimentation is becoming standard practice among product teams. The author notes that adoption alone has not made decisions better.

Common struggles with experiments

The piece lists recurring problems: unclear test choice, uncertain duration, limited scientific training, difficult tooling, and overgeneralized conclusions.

Where the material comes from

The talk content is drawn from the author's coaching of product teams, so the lessons reflect real mistakes teams made.

What the talk covers

The presentation runs about 40 minutes with a 20-minute Q&A, framed as a way for readers to learn from other teams' errors.

03From the post

“I see many product managers struggling with how to design and run good product experiments. It’s hard to know what to test and for how long. Few of us are trained to think like scientists. The tools are hard to use. We often overgeneralize our conclusions. It’s great”

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.