Drawer 11 · 316 notes · 58 people · 2005–2026

Metrics, data & experimentation

North-star metrics, analytics, A/B tests, and data-informed decisions

Q:
Answers from Metrics, data & experimentation notes, cited
Product Talk · Teresa TorresJun 10, 2020

Doing Discovery Well: How to Measure and Guide Your Team

Jun 10, 2020 · 7 min

“How do you know that you are doing discovery well? If you want to improve your discovery process, what outcomes help you track your progress? I get asked these questions often. I like that teams are trying to take an outcome-focused mindset to their discovery practice. The good news”

Product Talk · Teresa TorresMar 14, 2013

How To Run an A/B Test

Mar 14, 2013 · 11 min

Teresa Torres argues that A/B testing should follow the scientific method rather than being used for trivial tweaks like button colors or wording. She recommends starting from a real insight about why recipients ignore…

SVPG · Marty CaganMar 9, 2014

Flavors of Analytics

Mar 9, 2014 · 3 min

Marty Cagan argues that product teams often fixate on a single kind of analytics, usually web analytics, and miss the larger picture. He lays out four flavors that matter for product work: user analytics on how real…

Product Talk · Teresa TorresFeb 24, 2014

Why Testing Everything Doesn't Work

Feb 24, 2014 · 6 min

Torres argues that teams should stop testing ideas at random and instead be strategic about what they experiment on. Rather than prioritizing a list of ideas by gut feel, teams should estimate expected outcomes…

Product Talk · Teresa TorresJun 24, 2015

The Dos and Don'ts of Hypothesis Testing

Jun 24, 2015

Teresa Torres argues that many product managers run experiments poorly: they struggle to choose what to test, decide how long to run tests, lack scientific training, find tools hard to use, and overgeneralize results…

Product Talk · Teresa TorresNov 12, 2014

The 5 Components of a Good Hypothesis

Nov 12, 2014 · 6 min

Teresa Torres argues that product teams should test ideas with experiments before building them, and that this starts with a well-formed hypothesis. Vague statements like 'a redesign will improve usability' cannot be…

SVPG · Marty CaganJun 1, 2009

The Product Scorecard

Jun 1, 2009 · 5 min

Cagan argues that many companies lack a clear way to tell whether product managers' work actually aligns with business strategy, and that teams drift into chasing feature ideas. He proposes a 'Product Scorecard': a…

Lenny’s NewsletterAug 25, 2020

How to increase your retention

Aug 25, 2020 · 7 min
Subscriber post — summary only

This paid post explains how to measure retention using cohort analysis, including how to read cohort charts for trends and for signs of flattening that indicate product-market fit. It then previews a framework of seven…

Lenny’s Newsletter · Free post♥ 224

What is good retention?

Jun 9, 2020 · 12 min

Lenny Rachitsky and Casey Winters surveyed 20 experienced growth practitioners to define what good and great retention looks like across business types. They asked for benchmarks on user retention at six months and net…

SVPG · Marty CaganDec 11, 2011

Measuring Innovation

Dec 11, 2011

Marty Cagan argues that popular product measurement tools, such as product scorecards and Eric Ries's 'innovation accounting', are valuable for focusing on outcomes but are insufficient on their own. Measuring only…

SVPG · Marty CaganMay 12, 2007

Measuring Product Managers

May 12, 2007 · 3 min

Marty Cagan argues that the true measure of a product manager is the success of their product, but that picking the right success metric is hard. He proposes the Net Promoter Score (NPS), which asks customers how likely…

Product Talk · Teresa TorresJul 10, 2014

The Ethics of the Data We Collect

Jul 10, 2014 · 8 min

Teresa Torres argues that product builders rarely stop to consider the ethics of the data they collect, even though user tracking has grown far beyond the cookies that alarmed people in the late 1990s. She notes that…

Product Talk · Teresa TorresSep 20, 2012

Predicting The Impact Of Each User Story

Sep 20, 2012 · 5 min

Teresa Torres argues that predicting the expected impact of each user story before building it serves two purposes: it helps prioritize the backlog, and it surfaces the hidden assumptions behind what a team is building…

Lenny’s Newsletter♥ 135

How to determine your activation metric

Nov 8, 2022 · 17 min
Subscriber post — summary only

A follow-up on activation metrics explaining a three-step method: brainstorm candidate moments from usage data, test them with regression against retention, then run experiments to confirm causality. It adds real reader…

Product Talk · Teresa TorresMar 6, 2013

Measuring The Success Of Product Emails

Mar 6, 2013 · 4 min

This post sets out what product teams should realistically expect from email metrics: open rates, click-through rates, conversion rates, and unsubscribes. It explains that open rates are an imperfect approximation…