By Teresa Torres · producttalk.org · @ttorres on X · LinkedIn
Teresa Torres recounts how her team at AfterCollege nearly celebrated a false win: a report claimed their new Explore feature beat the old Search, but the funnel had been misread as page-to-page when it was actually a goal funnel, crediting Explore for Search-driven success. She argues that no amount of research or experimentation matters without intellectual honesty, which means rigorous methods, reliable data, unfiltered evidence, separating interpretation from data, seeking dissenting views, and admitting error. Because our brains push us to protect our sense of being right, she recommends assuming you are wrong and building a habit of reflection to develop that honesty.
01Key takeaways
- Scrutinize surprising results, especially outliers that look too good, before celebrating them.
- Confirm how your metrics are actually calculated, such as the funnel type, before trusting a report.
- Show all the data, including the parts that contradict your preferred story.
- Keep observations separate from interpretations, and get judgments from people who disagree with you or were not involved.
- Assume your idea is probably wrong and design tests that could prove it so failure becomes a planned step.
- Reflect regularly on what you did and learned, and share those insights with others.
02Key sections
- A near-miss with a too-good report
- A celebrated analytics report showing the new product outperforming the old one turned out to misread the funnel type. The author's suspicion, triggered by an outlier that looked too good, led to investigation that corrected the story.
- Why research alone is not enough
- Experimentation, customer interviews, and data-driven practices only help when paired with intellectual honesty. The author defines this as a discipline spanning methods, data reliability, and honest interpretation.
- Practices of intellectual honesty
- Concrete habits include keeping all data including outliers, separating raw observations from interpretation, getting disagreeing perspectives, looking for missing data, and having uninvolved people evaluate results to counter confirmation bias.
- Why we resist the truth
- The desire to be right, reinforced by cognitive biases and ego, makes us notice only what we want to see and react defensively. Fighting this requires deliberate effort.
- Assume you are wrong and reflect
- Framing ideas as probably wrong and planning next tests makes failure expected rather than painful. Regular reflection, even brief daily or weekly writing, builds the self-awareness that intellectual honesty depends on.
03From the post
“The report is wrong. This was the last thing I wanted to hear. It was March or April of 2013. We had launched Explore at AfterCollege a couple of months earlier. In the past week, we had circulated and celebrated a report that indicated that Explore (our new product) was”
04Frameworks mentioned
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.