The 5 Components of a Good Hypothesis

Teresa Torres argues that product teams should test ideas with experiments before building them, and that this starts with a well-formed hypothesis. Vague statements like 'a redesign will improve usability' cannot be proven or refuted, so a testable hypothesis must be specific. She proposes five components: the change, the expected impact, who will be affected, by how much, and after how long. Each component removes ambiguity and guards against post-hoc rationalization. The article matters because it turns experimentation from a slogan into a concrete discipline that prevents teams from fooling themselves with their data.

01Key takeaways

• Write hypotheses specific enough that an experiment could clearly support or refute them.
• Name the exact design or feature you will test, not a broad idea like 'a redesign'.
• Pick a measurable metric that the change is expected to move.
• Test only the population the change actually affects, excluding users for whom it cannot matter.
• Define the expected size of the effect in advance so results cannot be rationalized afterward.
• Calculate test duration from your expected effect and avoid stopping tests early when one version looks ahead.

02Key sections

Why vague hypotheses fail

Early hypotheses like 'fixing the comment form will increase engagement' sound reasonable but cannot be clearly supported or refuted by an experiment. Torres shows that specificity is the first test of a usable hypothesis.

The change and the expected impact

The change must name a concrete solution rather than a broad idea like 'redesign', and the impact must be a measurable metric. Each specific variant becomes its own testable hypothesis.

Choosing who to test with

Using an example of a newsletter popup shown to existing subscribers, Torres shows how including the wrong population dilutes results. Restricting the test to the relevant audience yields a much clearer conversion picture.

Setting the magnitude and duration

Defining the expected size of the effect draws a line in the sand that prevents moving goalposts after results arrive. Estimating the effect size also lets teams calculate how long the test must run to avoid false positives.

Putting it together and checking it

Torres provides template sentences and a checklist that asks whether the impact is measurable, the reasoning is clear, the population is right, and the duration was calculated.

03From the post

“Update: I've since revised this hypothesis format. You can find the most current version in this article: * How to Improve Your Experiment Design (And Build Trust in Your Product Experiments) “My hypothesis is …” These words are becoming more common everyday. Product teams are starting to talk like scientists. Are”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.