#43 - Metrics, Fast and Slow
The Hard Thing About Easy Metrics
Ask better questions, get better answers. Yet, we’re easily tricked into asking the wrong questions.
One of the key points of psychologist and Nobel Prize-winning economist Daniel Kahneman’s book Thinking, Fast and Slow is that people generally follow the path of least effort. To save time and energy, we develop heuristics or mental shortcuts. For example, Kahneman found that:
When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.
By mental sleight of hand, difficult questions are automatically transformed into easier ones when an answer can’t be found quickly. If the question is “Should I invest in Tesla stock?” the easier question to answer is “Do I like Teslas?” or “Do I like Elon Musk?” You don’t need to build a financial model, contemplate valuation, or wade through Tesla's 10K to answer them.
Kahneman calls this substitution. Below are a few examples. The hard question is the target and the easier to answer substitute is the heuristic:
Source: Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.
Business metrics are subject to substitution as well. Questions like “Are our clients satisfied?” or “How well are we allocating capital?” are hard to answer. Metrics like retention rate or the amount of capital returned to shareholders are much easier:
Like other heuristics, substitution works out fine most of the time, but occasionally causes trouble. Let's say you want to measure client engagement. Engagement is a hard concept, but the amount of time that your salesforce spends talking on the phone is easily measured. If this is your metric, you might end up with lengthier conversations about the weather.
Substitution happens unknowingly and the lack of scrutiny can be dangerous, as customers or problems to be solved become abstractions. As management guru Peter Drucker put it:
Securities analysts believe that companies make money. Companies make shoes.
It’s the same with metrics. Clients want solutions to problems, not sales calls. Your marketing department cares about net promoter scores, but your customers don’t.
Quality is a good example of a hard question. It’s tough to quantify or visualize in a Tableau dashboard. And yet it matters. As Shopify CEO Tobias Lutke notes on this excellent podcast:
Quality is a hard concept. It’s not quantifiable. This is why it's so hard to produce in the business world of today. There’s this disease of “everything needs to be quantifiable.” If it can’t be stated in an OKR then it apparently doesn’t exist.
It’s easy to spend a day in meetings discussing metrics and trends without thinking about harder questions like “What are our clients' problems?” and “How could we better solve them?” The path of least effort often leads to focusing on the easy question without scrutinizing if it’s really the question we want to answer.
Kahneman’s work suggests that we should be thoughtful about the metrics we focus on. Without meaning to, we could be focused on the wrong ones.
👉 If you enjoyed reading this post, feel free to share it with friends!
For more like this once a week, consider subscribing 👇
