Lenny’s Podcast131 min
Why AI evals are the hottest new skill for product builders
Hamel Husain & Shreya Shankar · Sep 25, 2025
Key learnings
- Evals are systematic ways to measure and improve an AI application, essentially data analytics on LLM behavior, replacing…
- Start with error analysis by manually reviewing around 100 sampled traces and writing short notes on the first upstream failure…
- Product people with domain expertise should lead open coding; appoint one 'benevolent dictator' whose judgment you trust instead…