Why You Can (And Should) Experiment When Building Enterprise Products

Teresa Torres argues that product teams building enterprise software can and should experiment, even though A/B testing is often impractical for them. She points out that experiments are simply procedures for testing a hypothesis, and that many common activities like interviews, usability studies, and surveys already qualify. The objections enterprise teams raise to A/B testing, such as onsite deployment, infrequent releases, and customer resistance to change, are actually stronger reasons to validate ideas before building. Teams with few chances to get it right need experimentation even more. The piece encourages product managers to widen their toolbox rather than dwell on the experiments they cannot run.

01Key takeaways

• Treat any procedure that tests a specific hypothesis as an experiment, not only A/B tests.
• Start with a clear, refutable hypothesis, since weak hypotheses are why many methods fail to produce knowledge.
• Recognize that infrequent releases and onsite deployment make pre-build validation more important, not less.
• Use interviews, prototypes, surveys, usage analysis, and diary studies to test ideas before committing engineering effort.
• Focus on the experiments you can run rather than being blocked by the ones you cannot.

02Key sections

Experiments are more than A/B tests

An experiment is any procedure designed to test a specific, falsifiable hypothesis. Interviews, surveys, and usability studies can all serve this purpose when framed well.

Why A/B testing often fails in enterprise

Onsite deployment, infrequent releases, and customer preference for stability make continuous A/B testing difficult. These constraints are real but do not rule out experimentation.

Constraints increase the need to experiment

Disruptive, infrequent updates mean each release must deliver value, so teams should test ideas before building them. Customers controlling upgrades adds pressure to create compelling changes.

Expanding the experimentation toolbox

Teams can use customer interviews, usability studies, usage analysis, prototypes, surveys, diaries, and observation. The goal is to focus on experiments that are actually possible.

03From the post

“Every time I speak about hypothesis testing, I get a series of questions about how to run experiments from people working on enterprise products. The assumption behind these questions is that hypothesis testing only works with consumer Internet products. And the assumption behind that belief is that the only way”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.