Framework · 1 note · discussed by Archie Abrams

Long-Term Holdout Experiments

Every note in the archive that names Long-Term Holdout Experiments: summaries, key takeaways and links to the original essays, episodes and posts. Ask a question below to get a cited answer from just these notes’ authors.

Q:
Answers quote and cite the source notes
Lenny’s Podcast94 min

Breaking the rules of growth: Why Shopify bans KPIs, optimizes for churn, prioritizes intuition, and builds toward a 100-year vision

Archie Abrams · Nov 7, 2024
Key learnings
  • Optimize for the absolute number of people reaching each funnel stage, not local conversion rates, since rate-chasing teams often…
  • Judge growth by the total long-term value of a merchant cohort (cumulative GMV over several years) rather than per-merchant or…
  • Keep long-term holdouts and revisit experiments at 3, 6, 12, and 24 months, because many short-term wins fade and some neutral…