Product Talk · Free post · Metrics, data & experimentation · Execution, roadmaps & process

Measuring The Impact Of Each User Story

Teresa TorresSep 18, 20123 min
SourceProduct Talk
KindFree post
PublishedSep 18, 2012
Originalproducttalk.org ↗
N:

Teresa Torres argues that choosing product themes is not enough; teams also need a way to predict and measure how each user story moves the product goal. Using an event-site example with the goal of increasing the number of events, she shows how to track two things for each story: whether people can complete the feature's funnel, and whether the feature actually changes the outcome it was meant to influence. The core method is comparing users who adopt a feature against those who don't, so teams only invest in features that demonstrably drive results rather than merely get used. This matters because it links day-to-day feature work to measurable product outcomes and prevents shipping features that look successful but do nothing for the goal.

01Key takeaways

  • Tie every user story to a specific measurable product goal before building it.
  • Measure the full funnel of each feature to find where users drop off.
  • Compare users who adopt a feature against those who don't to test whether it truly moves the goal.
  • Keep investing in a feature only if adopters outperform non-adopters on the target metric.
  • Track both whether people can use a feature as designed and whether the feature changes outcomes.

02Key sections

Linking stories to product goals
Themes keep the team focused, but each user story also needs a measurable connection to the overall goal. The example goal is increasing the number of events created.
Measuring the feature's mechanics
Track the funnel for each feature, from starting the flow through each step to completion, to see where people get stuck. This shows whether the design works as intended.
Measuring real impact
Test the assumption behind the feature by comparing outcomes, such as events started and completed, between users who used it and those who did not. Only features that improve the goal deserve continued investment.
Applying it to the venue lookup story
The same two-part approach applies: measure funnel completion, then check whether users of the lookup finish event creation more often than non-users.
Two kinds of measurement
Usability funnels are easier to measure and reveal problems, while impact measures are harder to define but explain why the feature exists.

03From the post

“On Thursday, I wrote about how to use themes to stay focused on your product goals. But using themes is not enough. You also need to identify how you are going to predict and measure the impact of each user story. Let's return to our previous example and suppose”

“You are looking to verify that this new feature is actually having an impact on your goals, not just that people are successfully using it.”Teresa Torres · Product Talk
“If and only if the people with imported contacts outperform people who have not imported contacts, do you want to focus on improving or keeping…”Teresa Torres · Product Talk
“The former is easier to measure and can tell you where people are running into problems, and while the second one might be harder to…”Teresa Torres · Product Talk

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.