The metric is dead, long live the metric

Perennial metrics

You can’t manage what you can’t measure, an idea I tend to agree with. Metrics are useful tools that are created to help us observe progress at a glance, and make informed decisions without having to dive deeply into specifics every time a fork in the road appears. We can use them to drive our teams towards a common set of objectives, measure our progress as we go along, make adjustments, and get feedback from those adjustments in a clear way. They provide a virtuous cycle of decision, activity, feedback, new decision, and so on, as long as our initial objective stays the course.

This is all fair and square, as long as the objectives, or even the operating model, don’t change throughout the journey. In today’s world, though, goals, objectives and operating conditions are fluid. Markets evolve, people rethink their strategies, customers demand changes and improvements more often, tools get better every day, and whatever metric was thoroughly designed at the start of the process will lose its value. Yet most of the metrics I know, and use, have this sticky quality to them which makes them linger around until they surpass their usefulness.

The cat in the box

As humans, and even more as decision makers, we strive to eliminate ambiguity. Metrics provide a useful proxy for decision making, because they translate complexity into a single number, which can guide our thinking and accelerate decisions.

They are used all around, they are followed as the value indicators that they were created for, almost as a mantra, and at some point we stop asking ourselves whether the metric is still pointing at the actual results we are trying to achieve. We are pursuing the numbers for the numbers’ sake, and don’t stop often enough to think about what the number actually means, and how it’s driving our objectives forward.

On the other hand, metrics can have a more pernicious effect, even considering that they are created with the best of intentions: they become the goal, rather than the measurement (Goodhart’s law). One can think about metrics the same way as Schrödinger thought about whether the cat was dead or alive inside the box: once you set a metric in place, it collapses the reality of what you set out to achieve. In other words, it becomes what you optimize for.

There’s at least one more situation where metrics deteriorate, and that has to do with operating models. We live in a world where strategic decisions have to follow the speed of the markets, leading to operating models that change much more often than the metrics built for them, with responsibilities changing across organizations, splits and merges of previous objectives following.

This is problematic for the stability of metrics, which were created to help those same organizations and departments achieve their objectives, but once the organization and underlying model evolve, many times we stick with the original metric to make sure we have a clear view of the progress (spoiler alert: most of the time, we won’t). Other times, the metric is still valid, but the organization has moved on and there’s no one entity accountable for the results the metric is optimized for, thus leaving us with an orphaned metric. And don’t get me started on change aversion, as I could probably write several theses on that subject alone.

I don’t move that fast, I don’t need to change a thing

Having lived in the world of software engineering and enterprise architecture for pretty much all my career, one might think that my view is biased, as this is a field that moves much faster than others, and that other, more stable industries, with less change in the way they operate, won’t fall into this kind of issue. A quick look at two of them shows us otherwise.

• Contact Centers

  • Metrics such as average handling time, built for phone queues where every contact went to a person and the mix of calls stayed roughly the same.
  • Then self-service took over the simple contacts, so people now handle only the hard ones.
  • Average handling time gets worse as the operation gets better. Example: 600 simple 3-minute calls and 400 complex 10-minute ones average 5.8 minutes. Move half the simple calls to self-service and the average rises to 7 minutes, with no agent doing worse.

• Produto Interno Bruto, aka GDP

  • GDP was originally built in the 1930s to measure the output of an industrial economy.
  • Then the economy moved to services and intangibles, and attention grew for unpaid household work and environmental cost.
  • It still steers policy as a stand-in for welfare, which Kuznets, who designed it, warned against.

This last example is probably the more obvious one. On the one hand, we live in an economy which is changing faster and faster. On the other hand, we are measuring it with a framework that, revisions aside, was designed almost 100 years ago. Perfect for comparison over time, but not exactly getting us a clear picture of where we stand today.

We come to the main tension then. We need faithful metrics in a world of moving targets and, at the same time, we need stable metrics that we can compare year-over-year. Both of these are pulling in opposite directions, and we default to the easiest way out: stability.

Getting the drift of it

We can’t move the metric, but we can’t live without it. How do we reconcile these two extremes? By going meta: we measure the measurement itself. We’ve seen so far that, the longer we try to stick with the same metric, the more likely it drifts away from its intended purpose. We need to track the drift of the metric itself, and update it as often as we can, so we know, and act, the moment that the original metric is no longer valid and needs to be updated or re-baselined.

Most metrics are actually a function of several parameters combined, each of these parameters within their own set of targets. As time goes by, some of these parameters, or the assumptions behind them, may change or even be dropped, and each of these changes can be marked as a metric deviation, increasing its drift. Once we get past some threshold, say 20%, we know that the original metric is stale, and we need to do something about it. It is no longer serving its intended purpose, since it’s not getting us a clear picture of our situation.

An example might illustrate what I’m trying to get at. A rollout I’ve seen more than once: multiple software platforms being built and deployed across multiple geographies, where progress is measured by a function of platform building progress (a set of modules per platform) and their deployment status in each of the geographies they are meant to reach. We can easily come up with a metric which combines both:

progress = module–country pairs done ÷ module–country pairs in scope

A pair counts as done once the module has been built and deployed in that country, so each module counts once for every country it is due in, and progress reads as a share between 0% and 100%. Alongside that number, we write down the assumptions it rests on: which platforms, which modules and which countries are in scope.

The thing is, these kinds of programs often take years to reach their goals, and by the time we are finally ready to call it a day, reality has shifted, and more often than not the initial assumptions have changed. Some of the platforms might have a different set of modules. Other platforms might have been dropped altogether. We might even have changed strategies to better fit the business goals. So the recorded progress is now trying to measure something that might not be achievable in the end or, even worse, is pointing in the wrong direction.

If we apply the proposed solution, the moment we start getting enough drift (let’s assume the 20% from before), we need to re-baseline what progress means. Whether the number goes up or down depends on where the change lands: dropping a platform nobody had started on would actually push progress up, but changes that affect work that had already been completed will pull it down.

Say we have four platforms of ten modules each, all due in five countries, so 200 module–country pairs, and after two years 100 of them are done: progress stands at 50%. Then one platform is dropped from the strategy (50 pairs) and two modules of another are redefined (10 pairs), so drift is 60 out of 200, or 30%, well past the threshold. The re-baselined metric now counts 150 pairs and, of the 100 delivered, 30 were on the dropped platform and 10 on the redefined modules, which leaves 60 that still count. Progress now stands at 40%, lower than before.

So instead of getting something better, it seems that we are putting ourselves in a more difficult situation, then? Not if we had agreed on the rules of engagement beforehand, and defined that the metric is only as useful as its freshness allows. We record the progress up until it becomes stale, start the cycle again with the re-baselined version, and nobody’s earlier work is re-scored.

The additional trick here is to have the two values (the old, now frozen metric, and the new one, re-baselined) side by side, which will actually be of greater value than a single, immutable, but stale measurement: the old definition closed at 50%, the new one starts at 40%, and 40 pairs of delivered work were overtaken by the strategy rather than lost by the teams. What could have looked like a step backwards becomes a precise account of how much the ground moved, which the old number, still reading 50% for platforms and modules that no longer exist as defined, could never have given.

Change aversion also gets addressed from the core: no one needs to be the one declaring the metric as dead, or stale. Drift will show it for us, and we just need to have agreed beforehand on the threshold and the mechanism to update - or drop - the metric. An expiry condition is therefore tied to the measured reality, instead of having an expiry date, or otherwise relying on someone taking the time to analyze how applicable the metric still is in the (possibly) new reality.

How to drive when the map keeps updating itself

It’s no surprise that an engineer would supply an engineering solution to a management problem, and that that solution will feel weird to those who are set in their management ways. It will feel even worse to those who have learned to “play the game” and optimize for the metric, not the goal.

I understand that it’s near impossible to save face when you are presenting a set of metrics in a board meeting in one month, and show worse figures on metrics with the same name in the next. Re-baselining means that the original goal might have been badly specified, might have changed along the way, or the metric itself wasn’t the correct one. More importantly, a re-baseline probably means that the DoD (definition of done) has moved, thus making it difficult to define clearly the moment when we get to “mission achieved”. Feelings aside, how are we supposed to steer our organizations, drive decisions, measure and keep momentum if we have moving goalposts and changing scoreboards?

The loophole that we need to explore, and explain, is that “done” belongs to fixed goals, but we’ve seen before that there is no such thing. Our objectives move, our goals get updated, and so must our metrics. So, actually, we need to update our initial premise of management through measurement. Metrics go back to being the tool they were created as. We get to put the cat back in the box. Re-baselining won’t stop anyone from gaming them, thus not completely solving Goodhart’s problem, but at least it prevents us from pursuing something that no longer makes sense.

On being S.M.A.R.T. and having OKRs

We need to address one last elephant that is still in the room. Most of us have hard, I mean S.M.A.R.T., objectives, or OKRs, depending on where we work, that motivate us to align our work with the goals of the broader organizations.

At the end of the day, if we are motivated to achieve those objectives, and work hard to do so, we won’t like it if they get updated along the way. The solution to the conundrum is a variation of the same: drift in the metrics behind an objective still decides whether it needs to be re-baselined, but for the purposes of evaluation, the change only takes effect at a set checkpoint, often enough that no one spends long being evaluated against something that no longer makes sense. Probably doing it once a quarter is the sweet spot, as quarterly business reviews are frequent, and we can use them as a reset and regroup. If the drift hasn’t crossed the threshold by then, nothing expires, and the objective carries on into the next quarter as it was.

The rest is the easy part. Each quarter is evaluated on its own, against the objectives that were in force during that quarter, and is worth up to 25% of the evaluation period, so the four quarters together make up 100% and a simple sum gives us our annual evaluation. We stop having stale metrics that frustrate everyone - the ones who don’t understand why they are still being measured by something that doesn’t make sense anymore, and the ones who are trying to set direction, based on some figure that seems to move very slowly, or not move at all.