“We cannot improve what we cannot measure” is sage advice variously attributed to Lord Kelvin (yeah, the absolute temperature scale guy), W. Edwards Deming, Peter Drucker, and several other successful people who had a very scientific way of managing things. I don’t know who said it originally, but at face value, it makes sense. If we can’t measure something, how do we know we making it better or worse? Of course, I mean it in the context of work, not life in general.
A common problem when it comes to measuring productivity is that a lot of managerial thinking in organizations is based on activity, not outcome. This sort of measurement works well when the activity is directly associated with the output, e.g. on an assembly line — where one can see the finished goods pass down the line and the outcome of the production quota being met. However, this doesn’t work at all when it comes to knowledge work. There is always some confusion as to what is “Knowledge work”. For instance, if the sales executives have clearly defined, quantifiable targets, it is tempting to say their work doesn’t fit into the category of knowledge work. After all, sales targets have actual numbers and hence are not so vaguely defined as goals in, say software development. If a Sales Executive works 2 hours more (activity), she could possibly close a few more deals (outcome); whereas the same may not work the same way for a software engineer. So does that mean a Sales Executive is less of a knowledge worker than a Software Engineer? Of course not!
Instead of splitting hairs about what is or isn’t “Knowledge Work”, I stick to some basic questions:
- Would it take specialized training and weeks (if not years) of hands-on experience for someone else to do what you do at least as well as you do it? (Skill)
- Does the outcome of your daily work, successes and failures alike, make you better professionally? (Learning on the job)
- Does your role require you to upgrade your skills every few months, just to stay on top of things? (Constant Reskilling)
If the answers are all “yes”, you are a knowledge worker. You are paid to think as much as (or more than) to do and you can leverage emerging technology to do your job better. There is a cruder way of putting this: How quickly can you be replaced in an organization, without the work suffering any setbacks?
Measuring what’s easy
In the last few decades, many mundane activities have been automated including the exemplar assembly line manufacturing jobs. And this rate is accelerating, with some estimates saying about half of all work activities can potentially be automated (Source: McKinsey’s “Jobs lost, jobs gained” report, Dec 2017). More realistically, the report says, around 60% of the current occupations have more than 30% of their activities that are technically automatable. Note this is about activities, not entire jobs; although this would mean that most of us will have to reskill ourselves to keep up with these advancements. The takeaway is that almost all jobs would be based on knowledge work, where our unique mix of skills that we learned from our unique experiences would be the most crucial requirement, not just because we can operate machine X or know programming language Y or have a degree Z.
While it is, in general, fantastic news that the tedious bits of most jobs will go away and all of us will have more mentally stimulating work to do, it is also a bit of a challenge on how do we measure performance and productivity. This post-assembly-line world of the knowledge economy needs us to overhaul the antiquated measurement techniques. It is harder than it looks. For starters, these outdated measurement techniques are all around us. These are what we are taught and handed down from generations of workforces before us. Unlearning all that would be as important as inventing new techniques.
Unfortunately, confusing productivity with time is a common choice. It is surprising how many people still employ this, even in knowledge work. This stems from the old Taylorian models which worked well for industrial production, not for knowledge work. Managers tend to cling to number-of-hours-worked not because it is effective, but because it is convenient. As we advance further in the 21st century, we would need to rely on Metrics of Consequence, not just Metrics of Convenience. Such Metrics of Consequence are hard to define, and tend to be unique for each organization or even each team! It’s no wonder that people stick to irrelevant measures because they are better than nothing (more on this later…). Almost invariably, such irrelevant measures are easy to game, giving a false sense of progress.
“Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” — Goodhart’s Law
In simpler words: To show progress, any tracked metric that can be gamed, will be gamed.
To illustrate, here are some common Metrics of Convenience:
- No. of defects in a software product — these are defects that were found, it doesn’t really tell us about defects that we haven’t found. A terrible quality metric.
- No. of days since the last workplace accident — it doesn’t tell us about what have we done to make sure there will be no accident today.
- No. of hours worked (my perennial pet peeve)
- No. of new leads added — what does that tell us about the quality of the leads and the actual business gain?
- No. of crimes reported in a precinct — doesn’t really tell us about the crimes that were not reported, plus disincentivizes the police to report new crimes.
- I can go on and on…
If you look at these examples, which are sadly not uncommon, you will see that they don’t just paint the wrong picture, they are very easy to manipulate. Such metrics create all sorts of perverse incentives in the organization and yet they are still widely used just because they are easy to measure.
Metrics of Convenience tend to make people measure things that are more likely to yield happy results.
Choosing such Metrics of Convenience comes naturally to most of us. But there are many brutal side-effects of this. Whenever people choose to measure whatever is easier to measure, they are ignoring the other important parameters. This in turn creates a culture where the Metrics of Consequence are often dismissed as unimportant or even non-existent. It is a bit like the old joke where a drunk loses his keys in some bushes but searches for them under a lamppost because the light is better there. Once such a culture sets in, it is very hard to change the mindsets in a team. Nobody wants to do things the hard way. Another side-effect is that Metrics of Convenience tend to make people measure things that are more likely to yield happy results. In “How to Measure Anything”, author Doug Hubbard calls this phenomenon “The Measurement Inversion”:
“In a decision model with a large number of uncertain variables, the economic value of measuring a variable is usually inversely proportional to how much measurement attention it typically gets.” — Doug Hubbard
This may sound like a strange paradox, but if you look around you — it is everywhere. And if you reason it out, it makes sense. All the easy-to-measure metrics that have the most significant impact on business outcomes will be optimized quickly. Then it gets harder and harder to find metrics that have some consequence and are simple to measure.
It appears that we have an incredibly hard task ahead of us. In a knowledge economy, each one of us has a unique way of working; a unique mix of skills and capabilities. We cannot all be measured by the same yardstick. Using the age-old Metrics of Convenience would lead to the unjust treatment of people and would cause the team or organization to focus on the wrong things. We have to focus on deriving the right intelligence from the metrics we collect, not worship the metrics blindly. We have to use this intelligence in order to make the right decisions. The outcome of the decisions thus made should have a tighter feedback loop into the measure-analyze-decide cycle.
It is time we have a more “agile” way of dealing with metrics — they have to evolve. Instead of starting with Metrics of Convenience, we should start with defining the decisions we have to make and what information we need to make these decisions. And this knowledge should percolate down to the lowest levels (with the right amount of translation at each rung).
There have been several good attempts to deal with this — SMART Goals, OKRs, MBOs, KRAs, Balanced Scorecard, SAFe® — all encourage us to zoom out and think before we start obsessing with metrics. The problem is that once any such framework becomes mainstream, there is effectively a micro-industry built around what is the “right” way to do it. So there is a rash of certifications and governing bodies preaching the “one true way”. I feel that defeats the purpose of most of these frameworks as they grew organically to address a real pain point, not due to some corporate mandate. The reason is, of course, implementing these practices takes a lot of trial and error.
Insights, not just Analytics
Another major problem seems that different sub-organizations tend to use different tools to track the metric. The software team thinks they are doing a stellar job with Jira but are not aware of the project managers’ Gantt charts. The marketing team is using their fancy new marketing dashboards completely oblivious of the tools the product managers are using. There is nothing wrong with that — each function needs so much specialized monitoring that it best if we use different tools for each. But, if you are the CEO of the company, how do you get a sense of what’s going on? The default answer seems to be extremely dense metrics dashboards that have all kinds of graphs jostling for space. There seems to be an assumption that it is easy to derive insights from these graphs and tables just because they are huddled together in one neat-looking dashboard.
Although these dashboards are modeled on the pilot’s cockpit dashboards — the two are entirely different things. A pilot has years of training to understand what each dial and each indicator means, and these things are more-or-less well-defined and fixed. Imagine being a pilot who enters the cockpit just to notice there are 5 new dials you have never seen before! And yet, this is how things work in most modern growing companies. Various teams discover new things that affect their work and hence, the business, every day. These things hardly ever bubble up to the CEO. Andy Grove’s observation that “The CEO is the last one to know” rings truer now than ever.
What would it take to deliver insights, not just raw metrics from the various parts of the organization? How would a CEO know the impact of a failing test on the upcoming product launch next quarter? How would the VP of Engineering understand where to refocus her team’s time based on the forensic evidence of last month’s metrics? This is something that I have been thinking deeply about. There are no quick solutions, of course — but watch this space :)
