How to Improve Your Experiment Design (And Build Trust in Your Product Experiments)

Teresa Torres argues that the popular "We believe... will result in... we will have confidence when..." hypothesis format is quick but insufficient for sound experiment design. She explains that teams should test the individual assumptions behind an idea rather than the idea itself, since that surfaces faulty beliefs before anything is built. She also shows that without alignment on the design before running a test, teams reinterpret disappointing results to fit their preferences and reach no decision. Her remedy is to specify the assumption, participants, metrics with thresholds, rationale, and planned actions for each possible outcome before experimenting. This matters because it targets a common failure mode in experimentation: running tests that cannot change anyone's mind.

01Key takeaways

• Separate the hypothesis from the experiment design; a well-phrased belief does not guarantee a sound test.
• Test the individual assumptions an idea depends on instead of building the full capability first.
• Frame decisions as comparisons among ideas to account for opportunity cost, not as isolated whether-or-not questions.
• Agree with stakeholders on the design and success threshold before running the experiment, not after results arrive.
• Define in advance what you will do if the assumption is supported, refuted, or flat; if the action is the same, skip the test.
• Avoid over-testing trivial variations without a clear rationale, since it raises false positive rates and wastes effort.

02Key sections

Hypothesis is not experiment design

A hypothesis is a supposition to investigate, while experiment design is the plan for testing it. The popular format helps teams commit beliefs to paper but does not guarantee a sound test.

Test assumptions, not ideas

Testing whether a feature works forces building it first and frames decisions narrowly. Testing the assumptions each idea depends on is faster and reveals which ideas share failing beliefs.

Facebook dislike button example

Testing a single capability yields an ambiguous result, while testing each underlying assumption, such as whether people have one emotional response to a story, exposes the faulty belief before building anything.

Align on design before running the test

When objections arrive after results are in, confirmation bias and escalation of commitment make them hard to judge. Teams should agree on the design upfront so a missed threshold means the hypothesis is false.

Elements to define upfront

Specify the assumption, the stimulus or data, the participants and sample size, precise metrics with thresholds and timing, a rationale for the metric, and what action each outcome will trigger.

03From the post

“I’ve got a pet peeve to share with you. If you’ve been following along with the growth of the Lean Startup and other experimental methods, you’ve probably come across this hypothesis format: * We believe [this capability] * Will result in [this outcome] * We will have confidence to proceed”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.