OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more

Kevin Weil, Chief Product Officer at OpenAI and former product leader at Instagram, Twitter, and Facebook (including the Libra project), discusses how AI is changing product work. The conversation covers evals, fast-moving model capabilities, bottoms-up product teams, fine-tuned and ensembled models, the future of creativity and education, and lessons from past projects.

01Key learnings

• Build products at the edge of current model capabilities, since models improve every few months and a barely-working product can become excellent with the next model.
• Treat evals as unit tests for models: define hero use cases, write evals for them, and hill-climb on those evals to know whether a product is actually working.
• Expect AI behavior to be fuzzy: a model right 60%, 95%, or 99.5% of the time implies very different product designs, so study your use case in detail.
• Use models for pieces of a problem rather than one broad prompt: break work into specific tasks, use different model sizes and fine-tunes, and ensemble them, much like a team of people.
• Keep planning light and iterative: quarterly roadmaps are useful for alignment and checking dependencies, but expect them to change, and favor bottoms-up empowered teams that ship and learn in public.
• Consider prototyping by vibe coding demos and proofs of concept instead of slide decks or static mockups, since current tools make this feasible even for non-engineers.
• Make chat a baseline interface for open-ended needs, but pair it with more prescribed, faster interfaces for high-volume, specific tasks.
• Hire and build teams around high agency and comfort with ambiguity, keep PM headcount lean, and let PMs be decisive when no one else will make a call.

02Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.