At this time of year, it’s common to set goals and New Year’s resolutions such as lose X kg of weight, or run Y km. If you have a goal like this, then I wish you well. It’s less common to have goals like have a better relationship with the people I love, even though they’re often the kinds of things people long for deep down. In this article I’ll poke at the difference between important on the one hand and neat and tidy on the other. I don’t promise any profound conclusions or easy answers – this is as much a reminder to myself as anything else.
How well is a country doing?
This video is a perfect example of the difference I’m talking about. It’s worth watching from the start, but I’m linking to the bit that’s most relevant.
The former Yahoo CFO quotes important statistics, and how they’ve improved – poverty and the unemployment rate for various groups. The Executive Director from Oxfam then counters with something even more important – dignity, and the consequences of it being denied from people.
The trouble is, how do you measure dignity? Can you say the total dignity in a country is 17.4 mega dignity units? The standard deviation of dignity is 3.5? What’s important and what’s easy to measure aren’t always the same thing.
Focusing on an output not an outcome
If you have young children and feel you’re staying too late at work, you might resolve to make it home for bath time at least twice a week. If you’ve never reliably been part of bath time this could be a big deal – if nothing else, it can be great fun for all concerned. However, what if you’re so busy on your phone with work email that you may as well not be there?
It’s often easy to measure the outputs of your decisions – the things that you immediately produce – and harder to measure the consequences or changes in the world that flow from the outputs. In this instance, the important thing – the outcome – is a deeper relationship with your children. The means to that end is the output – being in the bathroom while they’re having a bath.
However, the output isn’t always enough on its own to produce the outcome you want. If you merely fob off your children with “that’s lovely” as you scroll through the messages on your phone, why not be honest and play recordings of you saying “that’s lovely” while you go into another room and check your email where your device isn’t at risk of getting splashed with bathwater.
Pigeonholing people
In a past life, I dealt with describing what kind of job people did. There was a standard classification system of occupations which used 4 digits 0-9, so in theory could distinguish between 10,000 kinds of job.
Despite the best efforts of the people who designed it, there are still problems. Some jobs exist today that didn’t exist 10 years ago. Some categories in the classification are rather opaque, and not what you’d use on a business card or job description. So even if your job had a home in the classification system it might be hard to find.
Even if the classification system were perfect, what if you have more than one job? If there’s only one space in the database to store your job’s classification, which do you pick? Is it the one that you spend most time doing, the one that pays the best, the highest status one, the one you identify yourself as doing?
Estimating – Are we nearly there yet?
In a previous job, when we estimated how long some work would take, we had to produce 3 numbers – a pessimistic estimate, an optimistic one, and a most likely one. This meant we could express how much risk we thought there was in each piece of work. However, this isn’t a simple and direct answer to “how long will it take?”. It’s hard to sum, average and so on. So either you do some complicated maths to create a single number from each trio of numbers, or you pick the one you like the best and throw away the others.
Improving decisions
I understand some of the motivations behind doing what I’m describing as the wrong thing. There are some valid goals, such as a business spending time and money wisely, or a person trying to improve themselves in some way. However, it can be hard to measure e.g. wisely directly. There are two options – you either spend a lot of time and effort working out how to measure wisely, or you spend much less time and effort measuring something else. You hope that the something else is a close approximation to wisely.
The latter approach means that you are making decisions efficiently – you are getting a decision as output for only a little input of time and effort. However, there’s a risk that you are not making decisions that are effective. You might be just firing off bad decisions quickly.
You might want to be able to justify decisions better than just “it felt right to me” so numbers can give you a security blanket (“look: the spreadsheet says so”). You might also want consistency and fairness in decisions, to avoid bias. However, just because the numbers say so doesn’t mean it’s fair. The numbers in your spreadsheet or machine learning model could be encoding bias explicitly or by accident.
Some quotes
Here are a couple of relevant quotes. First, one from reality:
“I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.”
Lord Kelvin. Popular Lectures and Addresses “Electrical Units of Measurement” (1889).
And then one from fiction:
“Dark times lie ahead of us and there will be a time when we must choose between what is easy and what is right.” From: Harry Potter and the Goblet of Fire.
I really disagree with the first one. The stuff that’s really important – dignity, love, community and so on – can’t be meaningfully expressed in numbers, and yet can be known in a profound and rich way.
Conclusion
This is not meant as a counsel of despair. It’s just a reminder that life is messy and tricky. Reasoning about life will therefore also be messy and tricky, with caveats, error bars, asterisks. It’s easy to let difficulty and complex and guarded answers seduce us into focusing on easier-to-measure or easier-to-do-maths-with values, rather than what’s actually important to us.
Random Tech Thoughts 