Metric modeling for product managers
Florence Nightingale was an English nurse sent to Crimea to treat British soldiers who were wounded in the Crimean war. She observed that more soldiers were succumbing to non-war factors such as hygiene, shelter and nutrition. To highlight this, she created a graph (famously called the “rose diagram”) that succinctly conveyed this message to the British government. Not only did this diagram made the government take cognizance of unhygienic conditions of soldiers and act on it, it also established Florence Nightingale as a pioneer of data visualization. Such is the power of data and its representation.
The evolution of Data democratization has led to a steady decline in the contribution of “hunch” in building products. Data and metrics are central to decision-making and measuring the success of these decisions. Measuring a metric in an incorrect way will lead to Type 1 errors (commission errors) and tracking an incorrect metric will lead to type 2 errors (omission errors). Below are some thumb rules to help you get your metrics right…
1. Averages are deceiving: Most common rule and you would have heard it by now. Of course in a consistent performance setting the averages and percentiles may not be very different, but wherever there is variance it's better to look at percentiles because averages tend to hide the outliers. If you are looking to catch a very important flight, you better not go by the average time to reach the airport .
2. Don’t lose sight of counter metric: Let's take, for example, call handling time and customer satisfaction after the call. You must strive to reduce the call handling time by providing faster resolution to customers, however, tracked in isolation, this will lead to disaster. So when you are striving to improve a metric, ask yourself whats the counter metric.
3. Orthogonality of your metric: Sometimes there are “external” factors that can significantly improve or drop your key metric. When tracking a key metric make sure to define it as “orthogonal” as possible or at least document all those external factors for a quick check-in case of sudden movement.
4. Be sure to do sensitivity analysis: Improving a metric can become an obsession but after a point, the law of diminishing returns will kick in. So be sure that the connection of metric with the desired impact ( business value or customer experience) is analyzed for sensitivity. You must stop at the point of diminishing returns.
5. Maturity and time period: Some metrics will take time to mature, so its important to keep in mind the time period over which these metrics are getting tracked. For example, if your website has 10 day return policy, make sure to track all return metrics over rolling 15 day period.
6. Spot and separate the outliers: It's possible that one particular cohort of your customers or category is impacting your metrics ( either positively or negatively) its important to build those cuts, even if temporarily, so problems can be identified and isolated.
Other things to keep in mind…
1. Source of data: Be sure to know and track the health source of data that is driving your metrics.
2. Regular reviews: Do regular reviews of your definitions of metrics, changing market conditions may warrant changes to your definitions.
3. Customer experience vs efficiency: It is always good to prioritize customer experience metrics over efficiency metrics. Efficiency can be a by-product of a great customer experience.
4. Look at them every day: You must review your metrics every day — whether there is movement or not. Helps you keep the pulse.
