The challenge of the patch management process

Patch management is no longer a little administrative chore that you fit in around more important work; it has become one of the most pressing and difficult challenges facing security professionals, thanks to an increasing number of enterprise devices, applications and vulnerabilities.

How good are you at managing software patches? You probably have a regular routine to handle the patches that Microsoft issues on the second Tuesday of each month, but how long does it take you to apply them? Do you know for sure when the patches have been successfully applied to all affected machines, including laptops?

How do you handle those unscheduled patches we've seen so much of recently -- those 'out-of-band' patches for serious vulnerabilities that cannot wait for the next Patch Tuesday to come around? How does the rest of the business react when you ask for yet more downtime?

Then what about the patches for Linux, Adobe and increasingly, the mobile devices that users carry for remote network access? Do you have them in your schedule as well?

It is more likely that, like most companies, you manage some of the problems, but are struggling to keep up with the growing complexity of the task. Managing the patch management process is no longer a little administrative chore that is fit in around more important work; it has become one of the most pressing and difficult challenges facing security professionals.

It is important to get patching right, because any unpatched vulnerability is an open invitation to a cybercriminal to come in and steal information, according to Marcus Alldrick, head of information security for Lloyd's of London, the insurance underwriting organisation.

For more information

Ed Skoudis reviews how to develop a patch management policy for third-party applications.

"Organised crime is putting in significant amounts of money to develop malware, and Web applications are increasingly being targeted," he said. "It is very profitable for organised crime. They are not just after credit card details. We are now seeing targeted attacks against companies for specific information, or to gain the necessary access codes for that specific information."

In addition, the problem is no longer confined to MAicrosoft operating systems and browsers. Vulnerabilities are increasingly being reported in Linux, Adobe, Mozilla, Apple, Google Chrome and Microsoft Office.

And although there have so far been few attacks against smartphones, most experts agree that with their growing usage, they will become a favoured channel of attack by criminals. As security firm Sophos Inc. reported in July, vulnerabilities have already been discovered in commonly used smartphones, including the iPhone and Blackberry.

To make matters worse, this rise in complexity and volume of vulnerabilities is happening when companies are struggling financially and security budgets are tight.

"It can be difficult to get the business to accept the need for patching, because it has business consequences," Alldrick said. "Typically, companies that do patch will patch on the server side but don't give as much priority to the client side, even though that's where 95% of the vulnerabilities occur. But keeping clients up to date is hard. You have logistical issues to deal with, as well as people issues -- users may delay the patch because they want to get on with their work."

Some patches are very complex, are not easy to test and may require multiple reboots of a system, he said. If those systems are supporting business-critical applications, then the business units may argue for patches to be delayed, or for some other compensating controls to be put in place, such as a firewall or intrusion prevention system.

A risk-based approach can improve patch management processes. It is important to remind the business of the damage an infection can cause. For instance, the BERR Information Security Breaches Survey 2008 -- a revealing snapshot of U.K. security -- showed that 7% of infections took up to 50 person-days to recover, and 1% took more than 100 person-days.

Alldrick argues that organisations need to accept that patching is a "business as usual activity," part of a general maintenance regime that happens on a regular basis.

At Lloyd's, Alldrick has achieved that by integrating patch management into service management using the ITIL V.3 model, a set of recommended policies for handling information technology infrastructure and operations, including, in this case, the carrying out of changes and releases. "The patch goes through change management, and is all scheduled in to ensure there are no conflicts and the necessary downtime has been agreed on," he said about the patch management process.

Patch management success factors,

According to Marcus Alldrick:

* Get agreed patch management strategy and policy with senior management buy-in and support. Business impact reduction from a risk perspective should be a key driver.  

* Resolve conflicts between patching and business operations.  

* Schedule patch activity as 'business as usual,' but allow for emergencies.  

* Implement standard builds.  

* Reduce local administration privileges (to prevent users loading their own software) and consider application whitelisting.  

* Formulate an integrated process and automate where possible. Automated tools are essential, but so are the right people.  

* Allocate adequate resource, both management and line. Be flexible and agile.

Emergency patches are handled by the same system, and treated as incidents with high priority. But, as he said, the security department has to lay the groundwork for emergency work to take place.

"When you have emergency patches, then you need to rally the troops," Alldrick said. "If you are running a standard working week, you'll need resources that are prepared to work outside of normal hours. That is additional cost, so you need to have that budgeted and approved beforehand. You will also have to negotiate downtime on servers if they need patching, and that will impact the client side."

All that groundwork and negotiation is just the beginning. Actually applying the patches can be a challenge in itself, as Nancee Melby, head of product marketing for patch management company Shavlik Technologies LLC., explained.

"Most companies use a combination of doing things manually and using Active Directory and GPO (Microsoft's Group Policy objects) to manage systems," she said. "Active Directory is great as long as all your systems are domain-joined."

But in her experience of scanning networks, typically around 10% to15% of systems are not managed by Active Directory. "If they are not being managed, it means they are not being patched, and not being configured properly," she said.

Windows Server Update Services (WSUS) gives system administrators control over the distribution of patches, but is no use for non-Microsoft applications, and according to Melby, provides little control over when the patches are actually applied.

Even if you can apply patches, there is no guarantee they will stay effective. Alan Bentley, head of vulnerability management at Lumension Security Inc., said client machines can easily slip out of compliance if allowed. "There is the problem of patch drift," he said. "In the desktop area, you have constant change. Software can be installed and overwrite the DLL of a patch that was installed two weeks ago, and thereby makes it ineffective."

The patch management process, according to Bentley, should be treated in the broader context of vulnerability and configuration management, with technology keeping a constant watch over the machines on the network to ensure they stay compliant with policy and patches.

Products from Shavlik, Lumension and others, can help security and operations teams keep systems up to date and remove much of the legwork, especially for the non-Microsoft estate. But as Marcus Alldrick insists, patching is not easy and requires a broad range of skills, from the technical to the diplomatic in order to get the process accepted by the business.

And Alldrick warned of a new complication for those trying to keep systems safe. Up to now, it has made sense to concentrate first on any patches rated as "Critical" and to leave the others until a more convenient time. But now, he said, researchers at industry bodies and incident response teams such as FIRST and CERT, have noticed criminals switching their attentions to vulnerabilities of "Medium." "By combining medium vulnerabilities that come lower down the list of priorities, attackers are becoming more successful," he said.

The only solution, Alldrick said, is to limit the way corporate systems are used. That means whitelisting applications, standardising on desktop configurations wherever possible, and probably banning the use of social networking applications.

It is an unfashionable view and one that will hardly endear him with, perhaps, younger employees who expect to have the full range of communications options at their disposal. "We will have to accommodate the needs of the Y Generation eventually, but at the moment we are just not ready for it," he said.

Read more on Security policy and user awareness