Q:How should product teams use AI evals?

Teresa Torres argues that AI evals should be a core discovery habit for product teams, measuring how often an AI behaves correctly rather than running pass/fail checks¹. Start with error analysis on real traces, then pick the simplest eval that works¹⁷.

01Start from observed failures

• Torres says to run error analysis first and categorize mistakes before deciding which errors deserve an eval¹.
• She warns that off-the-shelf eval tools measure generic things, so your product's specific failures need to surface first⁷.
• Torres and Petra Wille suggest turning recurring error modes seen in real traces into specific evals⁴.

02Choose the cheapest eval that works

• Torres prefers code assertions where possible because they are fast and cheap, using golden datasets for smaller tasks and LLM-as-Judge only when semantic judgment is required¹.
• Aman Khan recommends choosing an approach per step: human feedback for ground truth, code checks for objective rules, and LLM judges for scalable subjective grading⁸.
• Start with 10 to 100 human-labeled examples and aim for roughly 90% agreement before trusting an LLM judge⁸.

03Use evals as an ongoing loop

• Torres describes evals as a fast loop: analyze traces, measure failure modes, improve, and repeat, including A/B testing prompt and model changes⁷.
• Evals need upkeep because criteria drift over time⁴.
• Marty Cagan and Marily Nika note that PMs should define acceptable error rates before launch³.