By Teresa Torres · producttalk.org · @ttorres on X · LinkedIn
Teresa Torres, host of Product Talk, is interviewed by Petra Wille about how product teams can move beyond casual use of chatbots to build AI-powered products. Drawing on months of building her own AI product, the Interview Coach, she argues that AI feature work resembles orchestrating a team of interns rather than writing one clever prompt. The conversation covers prompt decomposition, system design, observability through logging traces, and evaluation of non-deterministic outputs. It also addresses deciding when AI is the right answer to a customer problem and the often-overlooked ongoing maintenance burden. The episode is a practical starting point for product people working out where AI fits in their organization.
01Key takeaways
- Treat AI feature development like orchestrating a team of interns, with clear decomposed tasks and oversight.
- Break a complex AI task into smaller prompts and steps rather than relying on one massive prompt.
- Log full traces of LLM interactions so you can observe and debug how the system actually behaves.
- Replace simple thumbs up/down signals with structured evals that score specific quality dimensions.
- Confirm that AI is genuinely the right solution to the customer problem before building.
- Plan and budget for the ongoing maintenance cost of AI features, which is easy to underestimate.
02Key sections
- AI-powered vs. AI product managers
- The episode distinguishes PMs who use AI tools to do their own work from PMs who build products that embed AI features. The two roles require different skills.
- Prompting for products, not personal use
- Writing prompts that power a product differs from chatting with ChatGPT, because prompts must be decomposed and orchestrated to behave reliably at scale.
- System design and cross-functional collaboration
- Building AI features demands thinking about system architecture, risk mitigation, and closer collaboration across engineering, design, and other functions.
- Observability and evaluation
- Logging traces is essential for understanding LLM behavior, and evals must replace simple thumbs up/down feedback to judge non-deterministic outputs.
- Deciding fit and maintaining AI features
- Teams should first confirm AI is the right solution for a customer problem, and should budget for the ongoing cost of maintaining AI features.
03From the post
“Listen to this episode on: Spotify | Apple Podcasts In this episode of All Things Product, Petra Wille flips the script and asks Teresa Torres a big, timely question: How do product teams learn to build AI-powered products—beyond just dabbling with ChatGPT? Teresa shares fresh insights from her own”
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.