What to Do When You Don't Have Enough Traffic To A/B Test

Teresa Torres argues that a lack of web traffic should not stop product teams from learning, even if A/B testing is their only experimentation tool. Using a worked example of testing testimonials on a blog's about page, she shows that a duration calculator can reveal a test would take over 100 days, which is too slow. She then walks through the four levels of product analysis, asking what must be learned at each stage. Her point is that interviews, paper prototypes, competitive research and temporary paid traffic can answer those questions well before any split test is viable.

01Key takeaways

• Formulate a specific, measurable hypothesis before deciding how to test it.
• Use a duration calculator with your real traffic to see whether an A/B test is practical.
• Use customer interviews to validate whether the underlying problem is worth solving.
• Paper prototypes and think-aloud sessions reveal design insights without statistically significant traffic.
• Paid search ads can temporarily boost traffic when a test must run sooner.
• A/B testing is one tool among many, so do not let limited traffic stop learning.

02Key sections

Check whether traffic supports an A/B test

Start with a clear, measurable hypothesis and run it through a duration calculator. If the test would take too long at current traffic levels, A/B testing is the wrong tool for now.

Finding problems worth solving

Interview target users about how they discover and choose products, such as blogs, to test whether the underlying need, like social proof, actually matters to them.

Identifying the right feature

Compare alternative solutions to the same problem, such as testimonials versus social widgets, using paper prototypes and think-aloud feedback sessions.

Optimizing the design

Settle detailed design choices through paper prototyping, borrowing from successful peers, or temporarily buying traffic with paid search, accepting less statistical rigor.

Staying within what is feasible

Assess whether the best-performing solution and design can realistically be built, and plan any prerequisites such as collecting enough testimonials.

03From the post

“You’ve read the Lean Startup. You’ve drank the Kool-Aid. You’re ready to start experimenting. There’s only one problem. You don’t have any web traffic. If the only tool in your toolbox is A/B testing, then this might be a problem. Fortunately, there are”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.