The flaw in cybercrime research

Conclusions drawn from survey results are only as good as the samples they represent

Conclusions drawn from survey results are only as good as the samples they represent

Last month's CBI Cybercrime Survey 2001 looked authoritative and useful. It was backed by PricewaterhouseCoopers, the Armor Group, the Fraud Advisory Panel, and Nottingham Trent Business School. But did it paint a realistic picture?

From the survey we learned that the most serious threat in terms of specific events over the past year was viruses (43.8%), followed by hacking (16.2%), adverse comments on the Internet (10.5%), intellectual property infringement (8.6%) and databases illegally accessed (7.6%).

Two-thirds of respondents reported that a serious cybercrime incident had affected them in the preceding 12 months.

However, quoting results as percentages to one decimal point implies an accuracy of one in 1,000.

To achieve this and make worthwhile extrapolations you must ask several thousand people. This is necessary to allow for input errors, ambiguous questions, less than candid answers and unevenness in the sample.

Most newspaper political opinion polls on voting intentions ask about 1,000 people and only claim accuracies of within two or three percentage points.

The entire CBI survey is based on just 148 responses.

The people polled were from the 2,000 organisations that are "direct" members of the CBI, though the national importance of the CBI comes from the organisations that are "indirect" members, via 180 affiliated trade associations.

The data on specific events comes from just 105 responses. At that level even the smallest change in experience - or in the perceptions of the various security problems that were extensively analysed - would give radically different results.

In fact, the CBI survey is a paragon of statistical rigour compared with the marketing-oriented surveys that are all too readily accepted.

You know the sort of thing - "82.5% of businesses are alarmed about insecure customer relationship management kit", based on 60 telephone calls in which the respondent would be an idiot to say he wasn't worried, and sponsored by some company with a related product or service to flog.

The CBI can say that its results are roughly in line with other similar surveys, but most of them, such as the US Computer Security Institute work promoted by the FBI, have the same problem of a small number of responses (538 in the most recent survey). This may mean that we know little of the experience and views of the 80% to 90% of people who do not reply.

Of the various UK surveys, only those of the Audit Commission and NCC/DTI are based on about 1,000 responses.

It is not that I disagree with most of the CBI's recommendations - reported extensively in Computer Weekly last week - but their value comes not from these statistics but from the experience of its advisors. But we do need reliable statistics, both for individual organisations and for government.

Every bit of security advice tells us to begin with a risk analysis, and that in turn must start with a knowledge of what the general

problems are. Here, as with other forms of crime, we need to test our perceptions of threat against actuality. Otherwise expenditure and effort will be wasted and misdirected.

At a government level, good statistics are essential to inform policy and allocation of resources.

How can one balance the needs of law enforcement against those of the industry that is supposed to create general wealth via e-commerce?

The related debates about encryption, costs of interception and data retention have taken place largely on the basis of law enforcement assertions about threat, not statistics.

So here is a recommendation which did not appear in the CBI report: set up a national consortium, properly funded, to give us worthwhile statistics.

Peter Sommer is senior fellow at the Computer Security Research Centre, London School of Economics

Read more on Antivirus, firewall and IDS products