Q:How do I run A/B tests well, and when should I skip them?

Running A/B tests well means treating them as scientific experiments with a clear hypothesis and fixed rules, and knowing that they are only one of several ways to validate an idea. A/B testing is not always the right tool, and Teresa Torres stresses that experiments in general are simply procedures for testing a hypothesis¹. The core discipline is to start from a real insight, change one variable, and fix the test rules in advance³⁵.

01Design the test properly

• Begin with an insight about user behavior, not a long list of random variations, and write a hypothesis that predicts a measurable change³.
• Change only one variable and use a true control group, so you can attribute any difference to that change³.
• Count unique people taking action rather than repeated actions by the same person³.
• Test only your best options, since many variations raise the chance of false winners⁵⁹.

02Avoid stopping too early

• Torres warns that stopping whenever results look significant inflates false positives, so set the duration or sample size before launch and ignore results until it ends⁵.
• Trust only results around 95% confidence or higher; check that your testing tool is not defaulting to a lower threshold³⁵.
• Base the expected improvement on the smallest change that would be worth shipping, and remember small samples only reveal large differences³⁵.

03When to skip A/B testing

• Torres argues that enterprise teams with onsite deployment or infrequent releases still can and should experiment, using interviews, prototypes, and surveys instead¹.
• Marty Cagan separates discovery from optimization: A/B tests suit optimizing an existing product with live data, while new functionality calls for prototyping and user testing⁸.
• Skip the test if you have already decided what you would do under every outcome, since the answer would not change⁹.