Principles of product metrics
A lot of us, when asked about identifying good product metrics go to that one tool who knows the best.
Google knows a lot but unfortunately it does not know much about your product therefore, I don’t think the first 37 billion results will help.
What one really needs to understand is a structure or a set of principles that can be applied to any product and carve out its metrics easily. So here are 5 principles that you can apply before identifying metrics for any product in any stage of product development.
TL/DR
If you prefer watching a video instead of reading a blog, then you can watch this video on metrics. Watch from t=11:26 to 29:16
This blog is part of a 4 part series on product metrics.
Blog 1: Principles of product metrics
Blog 2: Thinking about lead indicators instead of lag indicators
Blog 3: Structure for product metrics
Blog 4: Graphs for product metrics
Principle 1: Metrics should be quantifiable & standardized.
Let us say I want to measure how much my customers love my product.
Philosophers would argue that love knows no measures but product managers are not philosophers. We need to find ways to quantify love.
Let’s say I can measure the love for my product by letting users rate the product on a scale of 1 to 10. Now this is a quantifiable metric that can be measured and is not subject to someone’s imagination.
These metrics should not only be quantifiable but also standardized. What this means is that we cannot have non-standardized metrics measuring the same thing.
I can’t have a few users rating my product on a scale of 1 to 10 and others clicking on smileys and frowns to establish their love for my product. As a user, I evoke a totally different emotion when I click on a rating of 1 to 10 versus clicking on a smiley or a frown.
In a nutshell, quantifiable and standardized metrics should answer WHAT should be measured.
Principle 2: Each metric should answer a question.
When I was a budding PM, whenever someone asked metrics, I used to get a foggy cloud of acronyms in my head.
It took me a while to realize that instead of searching for metrics on google I need to think of what are the questions that need to be answered to help me build the product better.
You do not start with metrics. You start with an unanswered question.
Let me explain using an example. Instead of straight away jumping into mau, dau, csat, nps, mau, cac, etc, one needs to think about what is the reason to capture a metric.
For example, my quest for a metric should start with few questions:
- How much time does it take for my customers to search for an item that they wish to purchase?
- How much time does it take for my customers to add an item to a shopping cart?
- How much time does it take for my customers to complete a purchase?
- What is the correlation between customers dropping from the website and the amount of time they spend on the website?
Now there are several metrics that can answer these questions.
- Cycle Time.
- Session based telemetry.
- Customer churn etc.
In a nutshell, metrics should answer a question, i.e., metrics should answer WHY you wish to measure something.
Principle 3: Each metric should elicit an action.
When I started my career, I used to think that as a PM my job was to measure as much as possible. I used to gather all possible telemetry without thinking whether I needed it or not.
It took me a while to realize that I need to think beyond WHAT I need to measure and WHY I need to measure.
I realized, I need to have a plan with respect to what do I do after I have captured these metrics. I realized, I need to think upfront about what action I need to take if the measurement is favorable or unfavorable. For example, I should be able to say that
- If “NPS” for my feature is < “0”, then I will …
- If “Customer Churn” > “10%”, then I will ….
- If “Cycle time” > 4 min, then I will …
In a nutshell, metrics that elicit an action should answer WHAT NEXT after you measure something.
Principle 4: Consequences of actions associated with a tracking metric should be evaluated.
We understand that metrics should have a business outcome. No wonder the proverbial phrase — you build what you measure is so popular.
When you measure wrong, you build wrong.
Metrics can have unintended consequences as well.
Imagine if I start rating doctors based on the number of lives that a doctor saves. Do you think the death rate will go high or low?
Share your thoughts in the comments.
For example, think about medium. If medium product managers focus on the following metrics, then what would happen?
- “Number of views.”
- “Read time of viewers.”
- “Number of claps.”
Medium product managers will only build products to make the viewing experience better. But is that the only persona that is important for medium? What about the editors? Would they have completely ignored the editor’s delight?
Here is an example of someone who did not think about metrics well and lost his job. The former CEO of Wells Fargo who had a perfectly good intention to grow the company customer base requested his sales team to increase the number of accounts by 8x using the catchy metric “”Going for Gr-Eight”. The sales team ended up opening fake accounts. The metric was met but the spirit of the metric was not. The CEO ended up losing his job since he did not foresee the consequences of the metrics.
In a nutshell, consequences of metrics, should answer WHAT IF the consequences of my metrics are not favorable.
To summarize, before you write metrics in your product spec, think of:
- What do you want to measure?
- Why do you want to measure?
- What will you do next?
- What if the consequences of the metric are not favorable?
If you’ve found value in my content and would like to support its continuation, I invite you to consider making a contribution at Patreon.
Your generosity enables me to keep creating and sharing meaningful content with you.
You can follow me on youtube or medium for regular updates on my own journeys of product management, business management, and leadership.
