Recently in Incident Response Category

The progressive worsening in BP's share price might in part reflect a continuing failure to address the finer points of strategic crisis management. Following on from the recent Toyota crisis, it leaves a worrying impression that many big international enterprises are not well equipped to manage large-scale incidents.

This is not a new problem of course. We've experienced many disasters before, and there are well established principles on how to go about crisis management. The snag is that they're not widely appreciated. Neither are they easy to execute. In fact very few senior executives, no matter how bright or well trained, seem to be able to translate expert advice into reality. 

Good crisis management is a rare skill. There are a few reasons for this. Partly it's because most executives are immersed in an organisation culture that is often itself a major contributing factor to the crisis, preventing them from seeing the wood for the trees. Partly it's because few executives are comfortable playing a dynamic, decision-making role that's completely different from their day job and prior experience. And partly it's because it's hard in practice to think clearly, objectively and strategically when you're under enormous pressure.

You can certainly spot some questionable decisions in BP's response: attempting to play down the size of the disaster; presenting a British image to an outraged US community; and offering up the CEO as a potential whipping boy. Lack of preparation or rehearsal for such events might also be a contributory factor, as there are press reports of factual errors in the published oil spill response plan.

As Dr Peter Sandman, a risk communications expert, once put it "The engine of risk response is outrage". An engineering response, not matter how elegant, will never suffice. Citizen rage needs to be directed to an appropriate target. President Obama clearly recognises this and is channelling it, along with his own rage, towards BP's British management.   

There are numerous learning points from this and other crises. My book "Managing the Human Factor in Information Security" contains a whole chapter on the subject of incidents and crisis management, setting out many of these points. It's a difficult art but one that needs to be studied and practised by a lot more senior executives.   

It saddens me to see good security initiatives holed by sloppy security practice. My in-tray has been full of emails urging me to comment on reports about the lack of security in the web site for the UK Cyber Security Challenge, sponsored by leading security institutes such as the UK Government's Office of Cyber Security, SANS institute, the Institute of Information Security Professionals and QinetiQ.

Operational security is easily overlooked when dealing with educational or research initiatives. That's the learning point. Reputation can be equally damaged by an incident on a minor web site as on a mission critical one. All public sites need to be safeguarded whenever brand value or reputation is important. Security professionals in particular need to aim for higher standards in widely promoted initiatives. 

The response now demanded is for the sponsors and organisers to demonstrate their crisis management skills and turn this threat into an opportunity. It's not easy, but it can be done.

Where does one turn to find objective, authoritative advice on security issues?

Certainly not the vendors if the recent reports of a security flaw in Internet Explorer are anything to go. There's a fair bit of spin or FUD in the announcements made in the last few days by Microsoft and its rivals. You have to carefully analyse the weasel words to get at the truth.

Nor can you rely on advice from governments, who seem to have created a hostage to fortune by recommending a temporary switch to other browsers. What does that mean? When will it be safe to go back? Are we talking days, weeks, months or years?

Security advice needs to consider the full range of circumstances. The size of the risk depends on many variables: products, versions, settings, behaviour, business impact, and of course the modus operandi, targets and capabilities of the attackers.

If Government wants citizens to use the Internet, then it needs to develop a more sophisticated approach to responding to vulnerabilities. Products cannot be judged to fine one day, and unsuitable the next. Security flaws in products are inevitable. We need defence in depth and better citizen education, not last minute panic warnings.

Every now and then we have to persuade our executive to think the unthinkable. But too much scaremongering can be counterproductive. You can read a few of my thoughts on the hazards of preparing for worst case scenarios on this Infosecurity Europe blog posting.

The recent two hour outage of Google's Gmail, affecting the majority of its 150 million users reflects the growing risks associated with the inevitable drift towards centralised system management.

At least Google was honest enough to issue an apology explaining that the incident was caused by an engineer's miscalculation and that they were investigating ways to ensure it did not happen again. (Mind you it's not the first of these incidents.)  That's a big improvement over O2 whose service was down for many customers during most of Saturday without any explanation.

Expect more of these crashes. Information technology is spectacularly vulnerable to tiny errors and we are building massive single point failure scenarios based on cloud computing, centralised management and technology monoculture. In response, we must all raise our game in business continuity and crisis response. 

Making a mistake once is good for your education. Making it twice means you're not learning fast enough.

On Tuesday, Twitter suffered its second denial-of-service attack in a week. Admittedly the site stood up better, being down for only 30 minutes this time. But it demonstrates the importance of immediately beefing up security following a damaging incident that might be repeated.

Customer perception drives business value. To go down once is unfortunate, to go down twice can seem careless, but to go down three times might suggest that the wheels are coming off.

A few postings ago I mentioned the growing importance of random acts of kindness by unsung heroes in rescuing or maintaining vital Internet services. Make no mistake; this is the future of security. When things get bad, we need to call on brilliant technicians to fix things. Fortunately some of the best have an altruistic streak.

Team Cymru are a good example. You might not have heard of them, and you might wonder why a top US security outfit would want to adopt a Welsh name. But I'm pleased to report that their hearts, as well as their expertise, are in the right place. In fact they are a strictly not-for-profit enterprise outfit, but with state-of-the-art skills.

The latest Team Cymru offering is a free alerting system to pinpoint open DNS resolvers in your immediate area. DNS is the DNA of the Internet, though it's based on a devolved management model which means that not all servers are as secure as you might like them to be. The Million Resolvers Project is a reporting system to alert participants when open resolvers are detected in their local address space.

If you're interested, you should contact them to get signed up. Like many information age initiatives, security is a two-way street. But what you get back is always more than what you put in.

For several years I've been preaching that the key to effective security management is to harness the power of social networks to help identify risks and prevent or respond to incidents. My book Managing the Human Factor in Information Security contains a chapter on the subject.

At this week's TED Global conference in Oxford, Internet lawyer Jonathan Zittrain of Harvard University gave a good example of how crack teams of volunteers responded to an incident in 2008 when Pakistan Telecom accidentally took YouTube offline. The service was rescued by "random acts of kindness" by unsung heroes.

Technology is spectacularly vulnerable to failures but also tremendously resilient. We just need to channel community effort in the right way. Unfortunately, they don't teach that on information security management courses.

By the way, if you want to read more about the TED conference, Howard Wright's blog has an excellent, comprehensive summary of the proceedings.

Several computer publications, including Computer Weekly, are reporting that T-Mobile USA is investigating an anonymous claim that a hacker broke into its databases and stole customer and company information. Some security experts have suggested that the claim appeared to be legitimate at first glance but is likely to be a hoax.

It's a sign of the times, a reminder that we are entering an information age ruled by spin, FUD and misinformation. A single whisper can undermine customer confidence in a global service provider. But at the same time a smart crisis response can take advantage of the free media coverage to boost public confidence in an organization's management of risks and incidents. It's time for enterprises to raise their game in crisis management.   

The current crisis of public confidence in UK Parliament, triggered by the publication of MPs' expenses records, demonstrates three interesting and very important lessons of crisis management. They are worth noting, as they play a part in all major crises. 

Firstly, there is the unexpected, unprecedented level of public and media rage. As Dr. Peter Sandman, a long-standing expert on risk communications once put it: "The engine of risk response is outrage". The need to manage extreme reactions should always be taken into account when planning any crisis response. 

Secondly, there is the invisible culture that surrounds the crisis organisation, blinding them from the true nature of the crisis. They cannot see anything wrong in their long-standing practices. Good crisis management requires an objective perspective. The crisis team needs to see events with the eyes of an outsider.   

Thirdly, there is the natural tendency to focus on the trigger of the crisis, rather than the underlying cause. The legality of MPs' expenses claims, and the rules governing them, are not the real issues. It's the perception of greed that outrages the public. And that demands a much more radical response.

Unfortunately when organisations are in crisis, they tend to operate on gut instinct guided by wishful thinking, instead of logic and expectation of the worst possible outcome. It's more comfortable to aim to muddle through, rather than confront difficult issues. But a crisis is a major turning point. Things will never the same again.