By Teresa Torres · producttalk.org · @ttorres on X · LinkedIn
Teresa Torres synthesizes conversations with nine product teams building AI features in production, highlighting recurring patterns across their work. She argues that small cross-functional teams, domain expertise, and narrow initial scope are driving success, while many teams start with no AI background. The piece also covers evolving architectures and eval practices, choosing the right tool rather than the trendiest one, handling integration sprawl, and teaching models to admit uncertainty. It matters because it offers concrete, experience-based guidance for PMs building their first AI products.
01Key takeaways
- Pair a product person with an engineer, or someone who has crossed disciplines, to make fast real-time tradeoff decisions.
- Start with the narrowest high-value use case, then expand based on observed user behavior.
- Bring domain experts in to define what good output looks like, including schemas and rubrics.
- Begin measurement immediately with simple tools like spreadsheets, then add sophistication as failure modes become clear.
- Prefer debuggable approaches over trendy ones when they perform comparably.
- Design systems to express uncertainty and say 'I don't know' rather than fabricate confident answers.
02Key sections
- Small, cross-functional teams
- Teams are small product-engineering pairs or trios where members span disciplines and move quickly. Boundary-spanning people with technical or product experience make the partnership effective.
- Starting from zero
- Most teams had no AI expertise at the outset and learned through time-boxed experiments, sharing findings regularly, and using LLMs themselves to learn.
- Domain expertise and narrow scope
- Former teachers, government officials, and SREs shaped schemas, rubrics, and workflows. Teams starting with one narrow use case and expanding based on usage outperformed attempts to do everything at once.
- Architecture and evals
- Architectures evolved from constrained workflows to plan-first and multi-agent systems, while evals progressed from spreadsheets to LLM-as-judge and simulators, always tied to decision points and stakeholder concerns.
- Tool choice and the 'I don't know' problem
- Teams often walked back trendy choices like pure embeddings in favor of debuggable hybrids, and invested heavily in getting models to admit uncertainty rather than fabricate answers.
03From the post
“Stack Overflow built four versions of their AI-powered search, then killed it. Arize's team started building when "agent frameworks" didn't even exist. eSpark's former teachers are now writing eval code in Python. Over the past few months, I've had the privilege of sitting down with”
04Frameworks mentioned
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.