Lenny’s Podcast131 min
Hamel Husain & Shreya Shankar · Sep 25, 2025
Key learnings- Evals are systematic ways to measure and improve an AI application, essentially data analytics on LLM behavior, replacing…
- Start with error analysis by manually reviewing around 100 sampled traces and writing short notes on the first upstream failure…
- Product people with domain expertise should lead open coding; appoint one 'benevolent dictator' whose judgment you trust instead…
Product Talk · Teresa TorresSep 2, 2026
Sep 2, 2026 · 26 min
Teresa Torres argues that AI evals should be a core discovery habit for product teams, not just an engineering concern. Because LLM outputs are probabilistic and semantic, traditional pass/fail unit tests don't work…
Product Talk · Teresa TorresOct 2, 2025
Oct 2, 2025
“Listen to this episode on: Spotify | Apple Podcasts How do you know if your AI product is actually any good? Hamel Husain has been answering that question for over 25 years. As a former machine learning engineer and data scientist at Airbnb and GitHub (where he worked on research that”
Product Talk · Teresa TorresSep 23, 2025
Sep 23, 2025
Teresa Torres and Petra Wille explore AI evals: what they mean, why they matter beyond quality assurance, and how teams can build them. Drawing on Teresa's experience building the Interview Coach tool, the conversation…
Lenny’s NewsletterSep 9, 2025
Sep 9, 2025 · 20 min
Subscriber post — summary onlyHamel Husain and Shreya Shankar outline a practical playbook for building AI evaluation systems that improve a product rather than just producing dashboards. The core argument is to start with error analysis by a domain…
Product Talk · Teresa TorresSep 3, 2025
Sep 3, 2025 · 33 min
Teresa Torres describes building Product Talk's Interview Coach, an AI tool that gives students feedback on their customer interviews, and how she designed evals to measure its quality. She explains the core idea of…
@ttorres · Teresa Torres on X♥ 97
"The only way to know if our AI products and workflows are any good is with evals." 💡 If you're using AI to write PRDs, analyze customer feedback, or build customer-facing AI products, you need to understand evals. This guide breaks down what evals actually are and why product teams should be…