Drawer 05 · 298 notes · 90 people · 2014–2026

Building AI products

AI-native products, evals, agents, and how AI changes product work

Q:
Answers from Building AI products notes, cited
Lenny’s Podcast131 min

Why AI evals are the hottest new skill for product builders

Hamel Husain & Shreya Shankar · Sep 25, 2025
Key learnings
  • Evals are systematic ways to measure and improve an AI application, essentially data analytics on LLM behavior, replacing…
  • Start with error analysis by manually reviewing around 100 sampled traces and writing short notes on the first upstream failure…
  • Product people with domain expertise should lead open coding; appoint one 'benevolent dictator' whose judgment you trust instead…
Lenny’s Podcast87 min

Netflix CPTO on AI and the future of product and tech roles

Elizabeth Stone · Jul 19, 2026
Key learnings
  • Expect a 'storming' phase with new technology; don't abandon AI, but be deliberate about capturing benefits while limiting costs…
  • Let product, design and data people prototype early against a clearly defined business problem, while still working with…
  • Invest in source-of-truth data, guardrails for shipping and testing, and review processes where AI output needs verification, and…
The Beautiful Mess · John Cutler♥ 83

TBM 425: AI and Agency

Jun 7, 2026

The piece argues that organizations asking their teams to find ways to use AI for company benefit are undermining that goal by stripping away the agency people need to experiment and decide. The core claim is that…

Lenny’s Podcast100 min

The non-technical PM’s guide to building with Cursor

Zevi Arnovitz · Jan 18, 2026
Key learnings
  • Start slowly with a ChatGPT project acting as a skeptical 'CTO' before moving to coding tools, gradually easing into code to…
  • Capture ideas quickly as Linear issues using reusable slash commands, so you can pick them up later with context already gathered.
  • Separate the workflow into explore, plan, execute, review, and document phases, and have the AI ask clarifying questions before…
Lenny’s Podcast108 min

How Anthropic’s product team moves faster than anyone else

Cat Wu · Apr 23, 2026
Key learnings
  • Shorten the path from idea to user: AI-era timelines have shrunk from six months to days or a week, so PMs should focus on…
  • Ship features as clearly labeled research previews to lower the commitment of launching and get fast user feedback that can be…
  • Set a clear, specific goal for who the key user is and what success means, which rules out many approaches and speeds decisions.
Lenny’s Podcast78 min

The Godmother of AI on jobs, robots & why world models are next

Dr. Fei Fei Li · Nov 16, 2025
Key learnings
  • Large labeled datasets were the missing ingredient that unlocked modern AI; big data, neural networks, and GPUs together formed…
  • Pursue a north-star problem and commit to it for years, as ImageNet's object recognition focus did for her lab.
  • Be willing to take intellectual and career risks, such as leaving tenure track or joining new ventures, when the mission and…
Lenny’s Podcast97 min

The 100-person AI lab that became Anthropic and Google's secret weapon

Edwin Chen · Dec 7, 2025
Key learnings
  • Define quality in rich, specific terms for each domain (e.g., what makes a poem Nobel-worthy, not just whether it has eight…
  • Gather thousands of signals on each worker and task, from background and expertise to performance, and use them like an ML…
  • Don't trust benchmarks at face value: many contain wrong answers, and optimizing for them can make models worse at messy…
Lenny’s Podcast102 min

Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

Chip Huyen · Oct 23, 2025
Key learnings
  • Focus on what actually improves AI apps, such as talking to users, improving data, writing better prompts, and optimizing…
  • Be cautious about committing to newly released technologies that have not been widely tested, since switching away from them…
  • Fine-tuning and post-training can shape model behavior a lot, and many teams now focus effort there because base pre-training…
Lenny’s Newsletter♥ 384

How to become a supermanager with AI

Nov 19, 2024 · 17 min
Subscriber post — summary only

Hilary Gridley, a WHOOP product director, argues that AI can help managers become "supermanagers" by scaling their coaching and feedback. She shares five practical strategies, from building custom GPTs that replicate…

Lenny’s Podcast123 min

The rise of the professional vibe coder (a new AI-era job)

Lazar Jovanovic · Feb 8, 2026
Key learnings
  • Start a project with several parallel attempts (brain dump, typed prompt, design reference, code template) to find the strongest…
  • Spend most of your time planning and chatting with the AI rather than executing, since clarity matters more than raw build speed.
  • Write a set of source-of-truth docs (master plan, implementation plan, design guidelines, user journeys, tasks.md) so the agent…
Lenny’s Newsletter♥ 497

How to use ChatGPT in your PM work

Apr 11, 2023 · 4 min
Subscriber post — summary only

Lenny Rachitsky argues that learning to work alongside AI tools like ChatGPT is quickly becoming a baseline skill for product managers, not something that replaces them. The post surveys a dozen concrete ways PMs…

Lenny’s Podcast102 min

Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future

Dianne Penn · Jul 26, 2026
Key learnings
  • Treat evals as the new PRDs: turn vague user complaints into concrete, reproducible test sets that researchers can act on and…
  • Dig into user transcripts, not just pixels, to find where the model actually failed, such as wrong tool calls, bad retrieval, or…
  • Keep strong conviction about a theme or area while staying loose about the specific prototype, and revisit bets across one to two…
Lenny’s Podcast102 min

What AI means for your product strategy

Paul Adams · Oct 26, 2023
Key learnings
  • Before redesigning a product for AI, start from the core premise: why people use it and what problem it solves, then map what AI…
  • Treat AI as a strategic shift rather than a feature; Intercom ripped up its strategy and rebuilt from first principles after the…
  • Build real machine learning depth on the team, but spread AI skills across product teams instead of isolating AI work in a…
Lenny’s Podcast148 min

Anthropic’s $1B to $19B growth run: how Claude became the fastest-growing AI product in history

Amol Avasare · Apr 5, 2026
Key learnings
  • Expect most growth time to go to 'success disasters' once growth is rapid; firefighting breaking systems matters as much as new…
  • In AI-first products, skew the growth portfolio toward larger swings, since future product value could be orders of magnitude…
  • Treat activation as a top lever in AI products: identify user traits that predict which feature fits them, then guide them there.
@joulee · Julie Zhuo on X♥ 228
Too many teams are doing AI transformation poorly, so I wrote down everything I have learned in my journey and in chatting with many other leaders. A long read, so get your coffees.
Lenny’s Newsletter♥ 528

An AI glossary

Jun 24, 2025 · 14 min
Subscriber post — summary only

A glossary explaining 20+ common AI terms in simple language, from models and LLMs to transformers, training, RLHF, RAG, evals, agents, and hallucinations. It aims to help readers follow AI conversations in meetings and…

Lenny’s Podcast73 min

How 80,000 companies build with AI: products as organisms, the death of org charts, and why agents will outnumber employees by 2026

Asha Sharma · Aug 28, 2025
Key learnings
  • Treat AI products as living systems: build the loop of collecting usage signals, defining rewards, A/B testing, and fine-tuning…
  • Invest in post-training and fine-tuning on your own data, not just off-the-shelf models; once models are large, adapting them to…
  • Watch for a shift from GUIs toward composable, code-native interfaces, since text streams and composability connect better with…
@joulee · Julie Zhuo on X♥ 284
4 tests any personal AI needs to pass to go big: 1. The mom test 2. The spare key test 3. The putting-it-off test 4. The bored-in-line test ...and my review of how @Muse does across each of them! Buckle up with a coffee and a snack, cuz this one goes deep.
Lenny’s Podcast100 min

How Block is becoming the most AI-native enterprise in the world

Dhanji R. Prasanna · Oct 26, 2025
Key learnings
  • Track AI impact with self-reported time savings plus validation metrics such as PR throughput and feature delivery, which Block…
  • Expect the current AI value baseline to keep rising; adopt tools continuously and re-evaluate where they add value as…
  • Non-technical staff building their own small internal tools with agents can compress weeks of waiting on engineering queues into…
Lenny’s Podcast118 min

The AI paradox: More automation, more humans, more work

Dan Shipper · May 24, 2026
Key learnings
  • Keep using the newest models and agent tools for your real work, and experiment on your own time if your employer restricts…
  • Expect most company AI to start as one shared agent that everyone can talk to, likely in Slack, rather than many personal agents…
  • Design products so agents can be users too: make them easy to drive via CLI and browser-friendly, because the customer's own AI…