Why Testing Everything Doesn't Work

Torres argues that teams should stop testing ideas at random and instead be strategic about what they experiment on. Rather than prioritizing a list of ideas by gut feel, teams should estimate expected outcomes, recognize those estimates are likely wrong, and focus on testing the underlying assumptions that must hold for an idea to work. She stresses choosing experiment types that fit each assumption, and deciding in advance what result would count as success so that confirmation bias does not rationalize mediocre outcomes. The piece matters because it turns experimentation from busywork into a disciplined way to accelerate real learning and build the right product faster.

01Key takeaways

• Estimate expected outcomes for ideas, but treat those estimates as likely wrong and look for disconfirming evidence.
• Identify the assumptions behind each idea and test those assumptions rather than only the idea itself.
• Choose the experimental method that fits the specific question, rather than relying on a single favorite technique.
• Define in advance what result will count as success and what action you will take for each outcome.
• Enlist someone to hold you accountable to your threshold so you act on results instead of rationalizing them.

02Key sections

Be Strategic About What You Test

Teams should not simply prioritize and build their idea list. Estimate expected outcomes, assume you are wrong, and use informed judgment rather than personal preference to rank what to test.

Test Your Assumptions, Not Your Ideas

Identify the assumptions that must be true for an idea to work, such as more traffic leading to more signups. Testing the idea alone can miss whether those underlying assumptions hold.

Run the Right Experiments

Throwing something at the wall is not an experiment. Match the method to the question, such as interviews for motivation, observation for behavior, split tests for layout, and surveys for prevalence.

Draw Lines in the Sand

Before running an experiment, define what a good result looks like and what action each outcome triggers. Otherwise the brain will rationalize whatever result appears.

Be Honest With Yourself

Once a threshold is set, teams must act on results even when they are disappointing. Accountability partners and systems help avoid second-guessing and questioning the experiment itself.

03From the post

“Please stop throwing spaghetti at the wall. It doesn’t work. I know you can launch a landing page in five minutes. I know it’s easy to toss up a quick A/B test. And I know how easy it is to chat with the 3 people sitting next”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.