Framework · 6 notes · discussed by Teresa Torres, Cat Wu, Kevin Weil

Evals

Every note in the archive that names Evals: summaries, key takeaways and links to the original essays, episodes and posts. Ask a question below to get a cited answer from just these notes’ authors.

Q:
Answers quote and cite the source notes
Lenny’s Podcast116 min

OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more

Kevin Weil · Apr 10, 2025
Key learnings
  • Build products at the edge of current model capabilities, since models improve every few months and a barely-working product can…
  • Treat evals as unit tests for models: define hero use cases, write evals for them, and hill-climb on those evals to know whether…
  • Expect AI behavior to be fuzzy: a model right 60%, 95%, or 99.5% of the time implies very different product designs, so study…
Lenny’s Podcast128 min

Inside ChatGPT: The fastest-growing product in history

Nick Turley · Aug 9, 2025
Key learnings
  • Ship early to learn what people actually want; with AI, the valuable behaviors are emergent, so you won't know what to polish…
  • Treat the model itself as a product: iterate on it using real user use cases, personality/vibe checks, and new capabilities like…
  • Set the team's pace deliberately by asking whether work is 'maximally accelerated', separating fast product iteration from…
Lenny’s Podcast108 min

How Anthropic’s product team moves faster than anyone else

Cat Wu · Apr 23, 2026
Key learnings
  • Shorten the path from idea to user: AI-era timelines have shrunk from six months to days or a week, so PMs should focus on…
  • Ship features as clearly labeled research previews to lower the commitment of launching and get fast user feedback that can be…
  • Set a clear, specific goal for who the key user is and what success means, which rules out many approaches and speeds decisions.
Product Talk · Teresa TorresOct 22, 2025

AI Changes Everything (And Nothing At All)

Oct 22, 2025 · 12 min

Teresa Torres argues that generative AI changes many product-management practices (prompt engineering, context engineering, orchestration, and evals) but that core fundamentals still determine whether a product…

@ttorres · Teresa Torres on X♥ 9
💭 "I wish I could click on this card and be like, 'Clean this up.'" When a customer pointed out a flat, unstructured branch in her AI-generated opportunity solution tree, it sparked a three-week journey to fix the problem at its source. Here's what you'll learn from this article: How one customer…