February 23, 2024
For the last 8 years I’ve been helping businesses get value from their data. This has ranged from setting up a company’s first dashboard to building machine learning products but most recently I have been helping startups set up metrics to track. This is something that sounds simple but is actually quite easy to get wrong. After reading this blog you should know what kick-ass metrics look like and how to avoid the most common pitfalls.
One observation I have made in my career so far is that everyone talks about being data-driven but not many have an idea what it really means. Does it mean being like Spock from Star Trek? Is it possible to be too data driven? Before we tackle these questions let’s set up some fundamentals. It’s impossible to unlock the potential of your data if you aren’t tracking the right metrics. To make this clear, I want to start off with a real-world example.
Last week I read an interesting piece about the McNamara fallacy and the dangers of being too data-driven. The jist of the fallacy is that being solely data driven leads to bad outcomes with the core example being Robert McNamara’s direction of the United States army during the Vietnam war in the 1960s.
McNamara, business executive turned Secretary of Defense, served from 1961 to 1968 and led the American strategy in Vietnam. McNamara tracked the progress of the war by focusing on the ratio of enemy fatalities to American fatalities. As long as there were more enemy deaths than American deaths, McNamara concluded that the military was on the path to victory. History tells us this was not the case as ultimately the war ended in failure for the USA.
Hence, McNamara’s story is used today as a lesson in how to not be data-driven and that context and qualitative data are just as important as metrics. Whilst I agree with this as a concept, I think it’s important to consider how a common pitfall is actually choosing the wrong metrics rather than being too data-driven. Had McNamara had numbers on how the size of Vietcong was increasing through the course of the war despite its losses, perhaps he would have considered an alternative strategy.
First of all let’s be clear what we mean by metrics in the context of business. Metrics are quantifiable measures used to track processes to judge the performance level of your business. There a couple of key components of that statement:
Now we know what metrics are, let’s consider what good metrics look like:
The first common mistake I see in businesses is that they try to track too much and it becomes overwelming. With the explosion of Big data tech and the ability to track everything, many companies are doing just that. In reality however if you measure every data point in your business and create 100 dashboards with 20 charts on each, all you are creating is noise that only a computer or a rocket scientist can read. A study in 2018 showed that the most viewed dashboards have between 15–20 charts per dashboard. This might be what dashboard lovers like to build but it just not practical for the average decision-maker to have to decipher 20 dynamic charts in real-time to run their business.
Instead, I would encourage businesses to identify the handful (5–10) metrics that truly drive their performance and align the whole team in understanding and tracking those. Less really is more when it comes to analytics and when you get more people engaged in data you naturally become more data-driven.
One thing that McNamara did right is understand the power of ratios. He understood that doubling the number of enemy casualties was not a victory if it meant a tripling of your own casualties. Ratios are a powerful tool that can simplify the number of things you need to track whilst simultaneously improving insights from your metrics.
My go-to example here is one from a successful SaaS business I worked with a couple of years ago that had a solid analytics capability. They were trialling a new feature on one of their products and found that it increased user growth by 30% as well as bumped up revenue by 10%. This satisfied their two core Key Performance Indicators (KPIs) and so they decided to roll-out the features to all users. However, what they didn’t realise was that the revenue per new user coming from the new feature was half that of the average user. This killed their Customer lifetime value (LTV) and in-turn profit margin took a 5% hit. If they had focussed on the single ratio of revenue per new user that really mattered they would have seen the clear picture from the beginning.
There are many opportunities to implement ratio KPIs across businesses that are often not considered today and I would encourage decision-makers to think outside the box to implement ratios as often as possible. Cross-functional ratios are especially powerful and are something we love at Calliper. Cross-functional ratios are ratios that use metrics from 2 different functions / departments such as Finance and Product or Sales and marketing. Having north-star metrics that are cross-functional aligns different teams to the same goals and drives collaboration.
Here a few examples of cross-functional ratios you can use:
If all of this sounds quite complex and you want the right metrics to track straight out of the box it’s worth me mentioning Calliper. Calliper is a tool that sets up best-in-class metrics and alerts at the click of a button for SaaS companies and makes analytics accessible and actionable for the whole team.
Below is a recommended metrics template for a typical high-growth B2B SaaS business straight from Calliper. This “Mission Control” provides the health of the entire business on a single page highlighting the key ratios that matter: User growth rate, MRR growth rate, Revenue per active user and more. It opens up the power to quickly build an accurate and useful analytics capability to any operator instead of limiting it to data whizzes.
Do you know anyone that is currently going through the process of defining north-star metrics in their company? Please go ahead and share this post if you think they will find it useful.
Also, please comment below the metrics you have chosen for your business which are interesting and that allow you to ask the right questions as a company. We’d love to hear!
From Calliper’s work with +100 startups, we’ve seen first-hand that successful startups experiment aggressively with data. In this post, we’ll explore some useful frameworks to help you run better experiments at a startup.
Culture is a social framework of unwritten rules in a company. Exploring it is like exploring an iceberg – much is implicit and lies beneath the surface. In this post, we’ll explore data culture through the lens of Schein’s framework and explore what data culture looks like in practice.