What's in a number?

I was amused to read about the latest estimates of the number of intelligent alien civilisations, recently reported in the International Journal of Astrobiology. Apparently the discovery of more than 330 planets outside our solar system in recent years has helped “refine” the number of life forms that are likely to exist. The new research claims that there might be as little as 361 intelligent civilisations in our Galaxy and possibly as many as 37,964.

You have to admire such breathtaking precision. It brings to mind those heavily-flawed estimates we make of risk probabilities: the ones that suggest the likelihood of a risk might be, say, 40%, but without mentioning that the accuracy of the estimate is plus or minus 90%. Such estimates are obviously worthless as a means of prediction, though they’re often useful for building business cases for investment appraisal, or, ironically, to demonstrate prudent corporate governance to an auditor.

More interestingly, numbers can convey subtle degrees of spin, depending on their precision and context. A number with one or two decimal points comes across as well-measured. A round number sounds suspiciously like a guess. The exception is the 80/20 rule which is strangely compelling and plausible, even though most examples quoted are not based on any sound research. 

Donn Parker always used to quote a made-up, precise number when discussing security risks. Many people took him seriously, though he was actually making the point that such statistics are nonsense and should not be relied upon. He was absolutely right. Taking figures from external sources is potentially dangerous. Many assumptions do not apply outside their original context. That’s why “Assume context at your peril” is a key Jericho Forum principle.

The other problem is that figures tend to get distorted as they’re passed on from person to person. For example, 37% might quickly become “one in three” or “over 30%” or “around 40%” after just a few exchanges. In fact, research has long indicated that around 70% of the details of a story passed on by word of mouth are lost in the first five or six exchanges.

Douglas Adams hit the nail on the head when he suggested that the answer to life, the universe and everything was 42. Because in security, it’s the question that really counts, not the answer.