Just Enough Statistics to Get the Product Job Done

Teresa Torres argues that product leaders need only a basic grasp of statistics to avoid being misled by A/B tests. She explains statistical significance using a conversion example and warns that many web testing tools accept as low as 80% confidence, which produces many false positives. She also shows that stopping a test whenever results look significant inflates errors, so duration should be fixed in advance from the smallest worthwhile improvement. Finally, she notes that testing many variations raises the chance of false winners, so teams should test only their best options.

01Key takeaways

• Check statistical significance before trusting any A/B test result.
• Verify your testing tool uses at least 95% confidence rather than a lower default.
• Define the test duration or sample size before launching and do not stop early.
• Base the expected improvement on the smallest change that would be worth implementing.
• Limit tests to your best options, since many variations create false winners.
• Use judgment alongside data; A/B testing does not replace thinking.

02Key sections

Why A/B test results can mislead

Running split tests isn't enough; teams must know when results are trustworthy. Torres uses a coin-flip analogy to show that small samples can produce misleading patterns.

Statistical significance and tool thresholds

Significance measures the likelihood that observed differences are real rather than chance. Many testing tools use weak thresholds, so teams should verify tools use at least 95% confidence.

When to stop a test

Stopping as soon as results look significant inflates false positives because significance fluctuates daily. Set a fixed duration upfront, based on the smallest improvement worth implementing, and ignore interim results.

Test fewer, better options

Testing many variations increases the odds that one looks like a winner by chance. Focus tests on the strongest candidates and apply judgment rather than relying on data alone.

03From the post

“Math is hard. I get it. Many of us continue to suffer from the false belief that we are incapable of doing math. I’m going to leave that alone, as this is a product blog and not a blog about the sad state of K-12 education or more”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.