Drawer 11 · 316 notes · 58 people · 2005–2026

Metrics, data & experimentation

North-star metrics, analytics, A/B tests, and data-informed decisions

Q:
Answers from Metrics, data & experimentation notes, cited
Lenny’s Podcast124 min

The ultimate guide to performance marketing

Timothy Davis · Jul 28, 2024
Key learnings
  • Paid acquisition suits most businesses because paid search is user-driven, so start by checking where users already find you and…
  • Run small signs-of-life tests first, starting with your own customer data for lookalike audiences at a 1% match, before…
  • Don't copy winning creative across platforms; user mindset and behavior differ, so adapt messaging for each platform.
Lenny’s Podcast115 min

How to find hidden growth opportunities in your product

Albert Cheng · Oct 5, 2025
Key learnings
  • Alternate between exploring for new insights and exploiting proven wins; when experiments stop reaching significance, return to…
  • Share experiment learnings across the company so adjacent teams can apply the same human-psychology insight to their own parts of…
  • Make a free tier a taste of the full product's value rather than a stripped-down version; sampling paid suggestions to free users…
Lenny’s Podcast69 min

An inside look at Mixpanel’s product journey

Vijay · Jan 26, 2023
Key learnings
  • Protect your core product: rather than moving people from the core to new ventures, invest the core's profits into new bets…
  • Be cautious adding adjacent products in new categories; bolt-on products that are only mid-tier in crowded categories rarely move…
  • When a product is losing customers to competitors, a focused 'speed' phase working directly from ranked churn reasons can close…
Lenny’s Podcast97 min

The 100-person AI lab that became Anthropic and Google's secret weapon

Edwin Chen · Dec 7, 2025
Key learnings
  • Define quality in rich, specific terms for each domain (e.g., what makes a poem Nobel-worthy, not just whether it has eight…
  • Gather thousands of signals on each worker and task, from background and expertise to performance, and use them like an ML…
  • Don't trust benchmarks at face value: many contain wrong answers, and optimizing for them can make models worse at messy…
Lenny’s Podcast118 min

Inside Google's AI turnaround: The rise of AI Mode, strategy behind AI Overviews, and their vision for AI-powered search

Robby Stein · Oct 10, 2025
Key learnings
  • Search demand is expanding rather than dying: AI lets people ask more and harder questions, so growth comes from new query types…
  • Build a product from a real user problem and stay dissatisfied with the status quo; keep making it better until it tips into…
  • Use a J-curve retention analysis (day 7, 30, 90) to tell whether a product is flattening into real usage or steadily losing users.
Lenny’s Podcast102 min

Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)

Chip Huyen · Oct 23, 2025
Key learnings
  • Focus on what actually improves AI apps, such as talking to users, improving data, writing better prompts, and optimizing…
  • Be cautious about committing to newly released technologies that have not been widely tested, since switching away from them…
  • Fine-tuning and post-training can shape model behavior a lot, and many teams now focus effort there because base pre-training…
Lenny’s Podcast74 min

How to grow a subscription business

Yuriy Timen · Sep 1, 2022
Key learnings
  • Give every channel a real test before judging it; many teams wrongly conclude YouTube or other tactics don't work because they…
  • Virality works best when the product has inherent network effects, where utility grows with more users, rather than trying to…
  • Treat SEO as a 3-month time-boxed experiment, favoring angles with a unique editorial, programmatic, or data advantage, and buy a…
Lenny’s Podcast93 min

Lessons from scaling Ramp

Sri Batchu · Jun 25, 2023
Key learnings
  • Sequence B2B growth roughly as founder-led sales, first salespeople, low-cost targeted marketing like content and community, then…
  • Use early investor and founder relationships as a growth channel: getting influential operators onto the cap table can drive word…
  • Make every growth channel driven by technology and data, such as dedicated growth engineers automating sales workflows and…
Lenny’s Podcast119 min

The one question that saves product careers

Matt LeMay · Aug 14, 2025
Key learnings
  • Ask whether you would fully fund your own team if you were the CEO; if you can't answer confidently, that signals a risk worth…
  • Set team goals no more than one step away from company goals, so leadership immediately understands how your work contributes to…
  • Keep impact first at every step, from OKRs and strategy to epics and daily backlog, rather than letting goals get cascaded into…
Lenny’s Podcast133 min

An inside look at X’s Community Notes

Keith Coleman & Jay Baxter · Feb 27, 2025
Key learnings
  • Surface notes only when raters who normally disagree both find them helpful; seeking cross-polarized agreement is what keeps…
  • Open participation to everyone with basic verification and random selection, rather than curating contributors, so the system…
  • Set the quality bar conservatively: showing a bad note damages trust far more than withholding a good one, so the team accepts…
Lenny’s Podcast88 min

Gibson Biddle on his DHM product strategy framework, GEM roadmap prioritization framework, 5 Netflix strategy mini case studies, building a personal board of directors, and much more

Gibson Biddle · Jun 20, 2022
Key learnings
  • Frame product strategy as a set of testable hypotheses about how to delight customers in hard-to-copy, margin-enhancing ways, and…
  • Use A/B tests to check what customers say they want; Gibson's new-release test showed that faster delivery barely moved…
  • Estimate the value of a feature by multiplying affected customers by lifetime value and a word-of-mouth factor, then weigh it…
Lenny’s Podcast159 min

Rethinking SEO in the age of AI

Eli Schwartz · Sep 19, 2024
Key learnings
  • Treat SEO as a product: start by understanding the user's search journey and what they would actually type before investing…
  • Skip SEO if you can't name the specific searches your target user would make; many SaaS products lack a real search journey and…
  • Expect AI answers to absorb top-of-funnel discovery queries, and focus SEO effort on mid-funnel, intent-rich queries where users…
Lenny’s Podcast108 min

Tobi Lutke

Tobi Lutke · Feb 2, 2025
Key learnings
  • Start new product decisions from first principles by re-deriving them from today's building blocks instead of copying what…
  • Be suspicious of pitches to build a better version of what everyone already does; doing the same thing cannot yield something…
  • Stay close to the details of the work, including the underlying technology, so decisions rest on real understanding rather than…
Lenny’s Podcast118 min

Unconventional product lessons from Binance, N26, Google, more

Mayur Kamat · May 22, 2025
Key learnings
  • Join fast-growing companies, since learning compounds much faster when you ship and learn daily rather than over multi-year…
  • Identify your genuine strengths, through structured assessments or peer feedback, and seek roles where success depends on using…
  • Early in your career, don't optimize for compensation; most earnings tend to come later, so prioritize learning and fit.
Lenny’s Podcast81 min

Why experts writing AI evals is creating the fastest-growing companies in history

Brendan Foody · Sep 18, 2025
Key learnings
  • Treat evals as the product requirement document for a model: if you cannot measure what success looks like for a task, you cannot…
  • Companies should build a systematic test of how AI automates their core value chain, since that measurement is the prerequisite…
  • Evals also serve as sales collateral, showing customers and researchers concretely which real-world capabilities a model or…
Lenny’s Podcast67 min

The future of AI in software development

Inbal S · Dec 1, 2023
Key learnings
  • Engineers using AI tools need to shift from writing code to systems and architecture thinking, which lets junior developers spend…
  • Don't adopt AI for its own sake; start from the customer problem and ask how AI can best solve it, rather than asking what to do…
  • AI tools must fade into the background: any extra setup, asking, or waiting adds friction and developers will abandon the tool.
Lenny’s Podcast111 min

How Snyk built a product-led growth juggernaut

Ben Williams · Nov 6, 2022
Key learnings
  • Start with a narrow, deep use case for one developer community (Snyk began with Node.js developers) to validate product-market…
  • Design acquisition loops into the product itself, such as Snyk-branded pull requests that educate other developers and drive them…
  • Build programmatic SEO assets, like Snyk Advisor's package health pages, that generate and index large volumes of pages…
Lenny’s Podcast115 min

Elena Verna on how B2B growth is changing, product-led growth, product-led sales, why you should go freemium not trial, what features to make free, and much more

Elena Verna 3.0 · Jun 23, 2022
Key learnings
  • Founders should drive the first stage of growth themselves; hire a growth leader only after product-market fit, strong retention…
  • A growth team cannot fix a declining core product or go-to-market; stabilize the core issues first before adding growth resources.
  • Homepage and brand redesigns rarely lift acquisition immediately; plan for three to six months of post-launch optimization or…
Lenny’s Podcast72 min

How to price your product

Naomi Ionita · Jan 12, 2023
Key learnings
  • Don't set pricing once and forget it; treat it like the roadmap and revisit monetization every 6 to 12 months, especially when…
  • Start monetizing earlier than feels comfortable, since delaying cheapens the product, removes pricing feedback loops, and creates…
  • Match price to value by choosing a value metric tied to how users get value, so revenue grows naturally as usage increases.
Lenny’s Podcast67 min

Teresa Torres on how to interview customers, automating continuous discovery, the opportunity solution tree framework, making the case for user research, common interviewing mistakes, and much more

Teresa Torres · Jun 30, 2022
Key learnings
  • Use the opportunity solution tree: put a measurable outcome at the root, map opportunities as customer needs and pain points…
  • Keep opportunities framed in the problem space rather than as disguised solutions; Torres notes most people mistakenly write…
  • Collect rich, behavior-based stories in interviews by asking about the last specific instance (e.g., 'tell me about the last…
Lenny’s Podcast110 min

Aishwarya Naresh Reganti + Kiriti Badam

Aishwarya Naresh Reganti + Kiriti Badam · Jan 11, 2026
Key learnings
  • AI products differ from traditional software mainly because user inputs and model outputs are both non-deterministic, so behavior…
  • Each time you grant an AI system more decision-making autonomy, you give up some human control; autonomy should be earned through…
  • Start with low-agency, high-human-control versions (e.g., suggestions to human agents) and log human corrections to build a…
Lenny’s Podcast145 min

5 questions to ask when your product stops growing

Jason Cohen · Jan 25, 2026
Key learnings
  • Compute your growth ceiling: new customers added per month divided by monthly cancellation rate gives the maximum customer count…
  • Ask 'what made you cancel?' rather than 'why did you cancel?' and use open-ended responses, since multiple-choice lists skew…
  • Be skeptical of 'too expensive' as a cancellation reason; customers who already bought didn't find the price too high, so dig…
Lenny’s Podcast81 min

OpenAI researcher on why soft skills are the future of work

Karina Nguyen · Feb 9, 2025
Key learnings
  • Build rigorous evals first: define deterministic pass/fail behaviors (e.g., correct time handling) and human win-rate comparisons…
  • Use prompting as a fast prototyping method to test product ideas before engineering investment, as Karina did with file uploads…
  • Synthetic data scales cheaply for teaching core product behaviors, but human expert data is still needed for specialized…
Lenny’s Podcast101 min

Building Anchor, selling to Spotify, and lessons learned

Maya Prohovnik · Sep 28, 2023
Key learnings
  • Dogfood your product by creating content with it yourself, since building creator tools without living the creator experience…
  • Ensure the team experiences the product's pain directly, for example by having engineers and PMs make their own podcasts, to…
  • Balance data and gut instinct by treating your intuition as a legitimate data point, explaining its reasoning with evidence…