Hypothesis Testing Mistake #1: Not Knowing What You Want To Learn

Teresa Torres argues that many teams run A/B tests without first knowing what they are trying to learn, which produces wasted effort and unclear results. Before testing, a team should define a specific hypothesis and decide in advance what it will do if the test passes, fails, or shows no effect. Failures are often misread as proof that an idea is bad, when the real problem may be the chosen design or the wrong feature. She proposes four levels of product analysis (value, features, design, feasibility) and says each test should be tied to one of them so conclusions are drawn correctly.

01Key takeaways

• Before running any experiment, write down the specific question you are trying to answer.
• Make hypotheses measurable, with a target effect size and a time window.
• Decide in advance what you will do if the hypothesis passes, fails, or has no impact.
• Don't reject an idea based on one failed design; the implementation may be the problem.
• Validate that the problem and value exist before testing specific features or designs.
• Label each test by the level of analysis it addresses to draw accurate conclusions.

02Key sections

Start with a clear learning goal

Testing everything is slow and frustrating, so teams should begin by asking what they want to learn and how an experiment can reveal it. Tools make testing easy, but without a clear goal the data is rarely actionable.

Make the hypothesis specific

A vague hypothesis like adding search filters helps shoppers find products faster should include a measurable target and timeframe. Precision makes it clear whether the result counts as a pass or a fail.

Plan for pass, fail, and no impact

Decide ahead of time what action each outcome triggers. A failed result should not be generalized to reject the whole idea, since the design or feature choice may be the cause.

Four levels of product analysis

Move through value (the problem), features (how to solve it), design (how to implement the feature), and feasibility (whether it can be built). Testing a design before validating value wastes effort and muddies results.

Match each test to its level

Identify which level a test addresses before running it, so conclusions match what was actually tested. This leads to sounder decisions about what to change next.

03From the post

“Have you ran an experiment in the past day, week, month? Can you clearly articulate your hypothesis? Do you know under which conditions your hypothesis will pass or fail? Do you know how to interpret the data in a way that drives successful product decisions? Many of us are new”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.