Drawer 11 · 316 notes · 58 people · 2005–2026

Metrics, data & experimentation

North-star metrics, analytics, A/B tests, and data-informed decisions

Q:
Answers from Metrics, data & experimentation notes, cited
Lenny’s Podcast95 min

How to measure AI developer productivity in 2025

Nicole Forsgren · Oct 19, 2025
Key learnings
  • Before measuring anything, get crisp on the problem or goal, since teams often interpret vague mandates like 'improve developer…
  • Shipping smaller changes more often tends to improve both speed and stability together, because small batches have smaller blast…
  • DORA's four metrics (deployment frequency, lead time, time to restore, change fail rate) give a benchmark-style view of where you…
Lenny’s Newsletter · Free post♥ 14

This Week #10: Keeping designers and engineers excited about metrics + Transitioning from DS to PM 🕺

Jan 14, 2020 · 9 min

“Hello and welcome to my humble newsletter where I attempt to answer your questions about building product, driving growth, working with other humans, and anything else that’s stressing you out at the office. Send me your questions and in return, I’ll give you candid real-talk advice 🤝 If you’re returning from last week, thank you! If you’re new to this newsletter, nice to have you! 🤜🤛 This week (just two questions, an experiment)…”

Medium · Hiten ShahNov 21, 2025

The History of Website Heatmaps

Nov 21, 2025 · 25 min

This essay traces how website heatmaps emerged from a blind spot in early web analytics, which counted page destinations rather than the actual interactions visitors made. Two identically destined links were merged into…

Lenny’s Podcast88 min

Becoming evidence-guided

Itamar Gilad · Sep 21, 2023
Key learnings
  • Build product from evidence rather than opinion; a leader's conviction should be tested with customer data, interviews, and…
  • Define clear, user-facing goals first, and map them with a metrics tree linking a North Star metric (value created for users) to…
  • Use ICE (impact, confidence, ease) to score ideas, and be honest about confidence using the confidence meter: opinions and themes…
@ttorres · Teresa Torres on X♥ 97
"The only way to know if our AI products and workflows are any good is with evals." 💡 If you're using AI to write PRDs, analyze customer feedback, or build customer-facing AI products, you need to understand evals. This guide breaks down what evals actually are and why product teams should be…
Product Talk · Teresa TorresJul 17, 2013

Be Prepared To Be Wrong

Jul 17, 2013

Teresa Torres argues that product builders consistently overestimate how right they are, drawing on behavioral research and the Heath brothers' book Decisive, which describes four decision-making traps: narrow framing…

Lenny’s Newsletter♥ 17

How to run SEO experiments

Dec 15, 2020 · 16 min
Subscriber post — summary only

Brian Ta, a former SEO lead at Airbnb, Strava, and AngelList, explains how to run SEO experiments on sites with thousands of programmatic pages. He argues SEO tests must be bucketed at the page level rather than the…

Lenny’s Podcast107 min

Mastering paid growth

Jonathan Becker · May 7, 2023
Key learnings
  • Don't put all your marketing budget into one paid channel; treat spend like a portfolio and diversify across channels to avoid…
  • Judge paid lead-gen by downstream revenue, not cost per lead; a higher-cost lead that converts to valuable customers can…
  • Use a lead scoring model fed by CRM revenue data to bid more on audiences likely to become high-value customers, especially when…
Lenny’s Podcast103 min

Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers

Laura Schaffer · Mar 9, 2023
Key learnings
  • Build career momentum beyond your manager by learning your customers deeply and sharing those insights, which makes you a go-to…
  • Share customer insights regularly and make them visible; a recurring voice-of-the-customer report can grow into a forum that…
  • Expect many ideas to be rejected; treat each refusal as a chance to bring better information, since leaders value insights they…
Lenny’s Podcast102 min

Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future

Dianne Penn · Jul 26, 2026
Key learnings
  • Treat evals as the new PRDs: turn vague user complaints into concrete, reproducible test sets that researchers can act on and…
  • Dig into user transcripts, not just pixels, to find where the model actually failed, such as wrong tool calls, bad retrieval, or…
  • Keep strong conviction about a theme or area while staying loose about the specific prototype, and revisit bets across one to two…
Lenny’s Podcast106 min

The ultimate guide to adding a PLG motion

Hila Qu · Apr 2, 2023
Key learnings
  • Treat PLG as data-led growth: free products should yield broader reach plus usage data showing which features correlate with…
  • Most companies end up running both PLG and sales motions; start PLG early because retrofitting it into a sales-led organization…
  • Commit for a year or two and don't mistake simply opening a free trial for a PLG strategy; it requires redesigning activation…
Lenny’s Podcast118 min

Behind the product: Duolingo streaks

Jackson Shuttleworth · Dec 15, 2024
Key learnings
  • Streaks are an engagement amplifier, not a substitute: first make sure the core product is something users want to return to…
  • Choose a unit of use that matches real engagement; simplifying the streak to one lesson kept users, but dumbing it down to one…
  • Focus effort on the first seven days, since loss aversion appears to kick in once a streak reaches about seven days and retention…
Lenny’s Podcast98 min

This will make you a better decision-maker

Annie Duke · May 2, 2024
Key learnings
  • Make implicit judgments explicit (e.g., a hiring rubric) before using intuition; Duke cites Kahneman's work lifting hiring hit…
  • Gather opinions independently and asynchronously before meetings, then use the meeting only for discussion, focusing on where…
  • Reserve meetings for discussion; discovery of views should happen beforehand and decisions should be made by an accountable…
Lenny’s Podcast126 min

The original growth hacker reveals his secrets

Sean Ellis · Sep 5, 2024
Key learnings
  • Ask users how they would feel if they could no longer use the product; a high share saying 'very disappointed' is a leading…
  • Survey a random sample of users who have actually experienced the product and used it more than once recently, not just sign-ups…
  • Use the very-disappointed group as a filter: find their primary benefit and the context ('why is this important to you') to craft…