Measurable Is Not Meaningful

There’s a pattern in teams that describe themselves as “data-driven”. Quite often, what they mean is “we have found some numbers that are easy to measure, and we have become very attached to them”.

That’s not data-driven. That’s metric-obsessed, and it’s a very different thing.

The Click Problem

Here’s a small example that captures the whole disease.

Imagine a team evaluating a change to a page with several rows of content on it. The metric they’ve chosen to judge it by is clicks per row. More clicks on a row, better row. Simple, measurable, dashboardable.

Except: every row is on the same page. So what the metric actually says is that every row must get as many clicks as possible - which means every row is competing with every other row for the same user’s attention.

A change that moves users more efficiently to the thing they wanted reduces clicks on the rows they skipped. And their metric calls that a failure!

The team has, in completely good faith, built a scoreboard that punishes them for helping users.

Browse surfaces are notorious for this, where giving weight to clicks sounds reasonable - it measures engagement, right? Until you realise that a user who finds what they wanted with fewer clicks is a success you’ve just scored as a zero.

Why Teams Do This

Because measurable things are comfortable.

A crash rate is a number. It goes up or down. You can graph it, alert on it, put it in a weekly report, and feel like you’re being rigorous. What you can’t do is ask it whether the product is any good, because it doesn’t know.

So when a team is unsure what it’s for - when the strategy is fuzzy, or the mission is contested - it reaches for the most measurable thing in sight and holds on tight. The metric becomes a substitute for the harder conversations about what actually matters.

Picture a team that ships a new feature, and the crash rate ticks up. Everything stops. It becomes an incident. Weeks of energy go into the number - meanwhile, engagement is up, people are finishing what they came to do, and users clearly like the thing.

Crash rate matters. It shouldn’t be ignored. But if nothing downstream has moved - if users are still engaging, still completing their jobs - then that tick is a curiosity, not an incident. It shouldn’t block the feature the downstream metrics say users want. Ship it, then follow up: investigate, fix, and keep the number from creeping up forever. Otherwise one day you’ll have a thousand features each contributing 0.01% to your crash rate, and no way to decide which ones to remove.

What it shouldn’t do is swallow all the attention that belongs to the question it’s supposed to be a proxy for: are users having a good time?

The Hierarchy of Easy

Metrics get chosen for the wrong reasons in a very predictable order:

1. It’s already in the dashboard. Someone built the chart two years ago, and now it’s load-bearing.
2. It moves fast. Daily signal feels like progress, even when it’s noise.
3. It’s ours. We can move it directly, so it makes us look or feel good.
4. It sounds rigorous. Statistical significance on a meaningless number is still meaningless.
5. It’s what the other team uses. Metric cargo-culting is rampant.

Notice what’s not on the list: it tells us whether we’re achieving the mission. That’s the only reason that should matter, and it’s the one that gets skipped because it’s the hardest to operationalise. Goodhart would be ashamed! But probably not surprised.

What It Costs

The obvious cost is poor decisions - the change that helped users gets rolled back because clicks dropped.

The less obvious cost is worse. Metric obsession is often a way of avoiding commitment. If you’ve picked a number you can always move, you’ve guaranteed you’ll never be visibly wrong. You’ve also guaranteed you’ll never learn anything, because a metric that only ever goes up has no information in it.

A metric should be a signal, not a lever - a good one has to be able to tell you no. Metric obsession is what you get when a team struggles with failure, ambiguity or uncertainty. And it acts as a positive reinforcement of that weakness, which usually ends up manifesting as teams individually succeeding while the organisation fails.

The Fix Is a Question, Not a Number

If your team is spending a lot of time staring at a metric, try asking:

• What question is this number actually answering?
• Are there “good”, “bad”, or neutral versions of the thing we’re measuring that hide in aggregate? e.g. “neutral crashes”, “angry clicks”
• Could we move this number by doing something that’s bad for users?
• Will we move this number by doing the right thing for users?
• What isn’t this number telling us that we’d want to know?

You’ll notice none of those produce another number. They produce a conversation about purpose.

Measurable is cheap - meaningful is hard.

Teams that confuse the two end up extremely well-informed about things that just don’t matter.