Framework · 1 note · discussed by Dianne Penn

Evals as product requirements

Every note in the archive that names Evals as product requirements: summaries, key takeaways and links to the original essays, episodes and posts. Ask a question below to get a cited answer from just these notes’ authors.

Q:
Answers quote and cite the source notes
Lenny’s Podcast102 min

Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future

Dianne Penn · Jul 26, 2026
Key learnings
  • Treat evals as the new PRDs: turn vague user complaints into concrete, reproducible test sets that researchers can act on and…
  • Dig into user transcripts, not just pixels, to find where the model actually failed, such as wrong tool calls, bad retrieval, or…
  • Keep strong conviction about a theme or area while staying loose about the specific prototype, and revisit bets across one to two…