To patch or not patch - that is the question for IT directors who must balance the need for security against everyday operational demands. Sally Flood looks at ways to fill software holes without spoiling the overall picture
One Saturday morning in 2003, the Slammer worm was unleashed on an unsuspecting world. In the first minute of its life, Slammer doubled the number of machines it infected every eight seconds. Within 10 minutes, 90% of all vulnerable machines had been infected - leaving businesses with a £500m clean up bill.
The bill was all the more painful because Slammer hit a full six months after Microsoft had released a patch to prevent it. If everyone had patched their systems in the first place, Slammer would never have happened.
So why are companies not patching? Simply put, many of them do not have the time or the resources. Researching the 4,200 vulnerabilities published by US-based security monitoring body CERT during 2003 (compared to the 1,200 vulnerabilities published three years earlier) would have required more than 700 man hours. IT services provider Computacenter estimates that assessing, testing and rolling out a single patch can take up to eight weeks and cost as much as £400,000.
Unfortunately, patching is a necessity. The time lapse between vulnerabilities being discovered and exploited is becoming increasingly short - Blaster was released just 18 days after the vulnerability it exploited was discovered. "Not installing patches is an extremely dangerous tactic," says Andy Kellett, a senior research analyst with Butler Group.
The good news is that an increasing number of companies are putting people, processes and tools in place to bring greater efficiency and control to the patching process. US analyst firm Yankee Group estimates that using effective patch management software and processes can reduce the cost of managing patches from £150 per PC to less that £100.
"Patching is not easy. It takes a lot of time, and we do not have the physical or financial resources to stick to the best practice," says Pat Black, IT manager at Homefirst, a company offering residential care services in Northern Ireland. Homefirst has a network of offices in seven counties, with more than 50 servers and 1,000 desktop PCs. "We only have four IT staff to support the whole system, and it is not possible for us to physically install patches on all those machines and make sure they are kept up-to-date," says Black.
Instead, Homefirst uses a patch management application from Sygate that scans machines and updates patches automatically on a regular basis. "If a machine does not have the right patches, the software quarantines that machine from the rest of the network until it is up to date," says Black. "It gives us a bit more breathing space, so we are not running around installing patches the whole time."
Nightclub group Ministry of Sound uses Microsoft Software Update Services (the predecessor to Windows Update Services) to automatically update patches at its London headquarters. "With SUS, we get a notification once a week with a shopping list of updates and we select the ones we need," says James Bacchuf, Ministry of Sound's chief technology officer.
"They are automatically installed on a Wednesday afternoon, when it will not affect network performance too much. The only thing the user has to do is restart their machine when it prompts them."
Ministry of Sound uses SUS to automatically update patches on all its desktop PCs, but still has to manually patch its servers, because they run on different operating systems. The company also has problems patching servers running some legacy applications not supported by Microsoft. "There is one machine running a radio application that is quite old, and we did not patch that because we could not get the software to run with the XP Service Pack 2 patch," says Bacchuf.
Despite the challenges, Bacchuf is a big supporter of automated patch management. "Previously we were going online every week to see what patches were released, reading news sites and bulletin boards, but never being 100% sure that all the systems were properly patched," he says. "Now IT staff can be developing applications or providing support rather than running round trying to install dozens of different patches each month."
Patch management software from specialists such as PatchLink and Shavlik automates the process of finding, downloading and applying patches. Different packages have different capabilities, but most patch management tools include checks for updates, automated deployment, roll back capabilities and centralised management of patches.
The advantage of automated patch management is that it reduces the risk of human error. However, the software does have its limitations, says Kellett. "This is a new market, and different applications have wildly different functionality," he says. For example, some patch management tools allow for automated deployment and group policies, while others will only alert you to a missing patch.
Perhaps most importantly, patch management software is only as good as the underlying security policy, says Bill Pepper, director of security with CSC's risk management practice. "Before you deploy any technology, you need to know as an organisation, how you want to manage patches," he says. "What patches will you apply, to what systems, and in what order? How will you assess new alerts?"
In the event that an outsourcing company manages your system updates, do not think you can relax and forget about patch management. "You can outsource the danger, but you cannot outsource the policy," says Pepper. IT directors must agree with outsourcers on how patches will be applied, when and with what priorities and how that fits with the business.
Patch management policies should be based on a comprehensive risk assessment. This means knowing what the potential vulnerabilities are, and how important it is to the business that they are fixed. "Your policy should take account of what is a critical system to the business, and whether the cost of patching a hole outweighs the potential cost to the business," says Pepper. For example, retailers often do not apply patches in December because it is the busiest time of year, and installing new software presents an unacceptable risk to the business.
Risk management should also specify which patches will be tested, for how long, and to what degree. In a worst-case scenario, firms may find they have to develop their own patches for custom or legacy applications, or to leave the system unpatched. "Microsoft in particular is becoming more bullish about not supporting legacy operating systems in its patches," says Pepper. "If you cannot afford to upgrade to newer operating systems, often you end up not applying patches at all."
Some suppliers, such as SAP and Microsoft, offer support to help companies resolve conflicts between patches and third-party systems, so it is important to speak to suppliers when drawing up a patch management policy.
It is also important to speak to suppliers about how they classify patches, and what is included in the increasing number of jumbo patches distributed by suppliers on a monthly or even quarterly basis. For example, Oracle recently revealed it will begin releasing software patches on a quarterly schedule, so customers can better plan for them. Microsoft began issuing monthly patches in October 2003, and Computer Associates and SAP have been on regular schedules even longer.
This is good news in the respect that IT departments do not need to deal with dozens of different alerts. However, sometimes it is difficult to know exactly what is included in a patch, and inadvertently fail to apply something that is critical. To this end John Meakin, group head of information security at Standard Chartered Bank, is using Qualysguard web-based scanning technology to help prioritise patch management.
"For banks such as ours with many large and complex interconnections to the outside world, it is vital to carry out effective patch management, but absolute security is an impossible goal," says Meakin "Our aim is to achieve the right level of security through implementing an appropriate risk-based strategy for each part of our global network, and we use Qualysguard as a tool to underpin this process and prioritise patches."
Pepper also says that selective patching has to be done with care. "We know from experience that patches often fix more than they say they are going to," he says. "That can be a problem if companies are only installing selected patches."
Equally, blindly applying patches without conducting a risk assessment can be counter-productive. Pepper cites the example of a UK bank that insisted on deploying a patch to a Unix box running a legacy application. "The patch conflicted with the legacy application, and payroll simply did not happen that month," he says. "The cost of fixing that problem far outweighed the risk of not patching immediately."
Once you have a clear patch management policy, consider what procedures can be implemented before patching. "There is a lot you can do to mitigate risk apart from patching," says Garry Sidaway, a security analyst with CyberTrust. For example, IT staff can use filters to strip off executable files from e-mails, and disable applications such as instant messaging, WinZip and media players to reduce the risk of introducing viruses to the network.
Intrusion detection systems, click here >>
Patching things up
A good patch management policy should follow these six steps:
- Understand your environment and what the vulnerabilities are. Not all systems are equally at risk or need to be fixed at the same speed. This can be done using vulnerability testing software
- Build an infrastructure to deliver patches. Off-the-shelf software can be used for non-critical systems, but it may be necessary to manually update critical servers or legacy systems. This can be done using patch management software or systems management software
- Reduce risks by using intrusion detection software, anti-virus and firewall software so that vulnerabilities will not be exploited before you can apply a patch
- Consider using external monitoring to try to get early warning of vulnerabilities and to filter real threats from irrelevant alerts
- Test everything then test again, to make sure the patch will work. This can be done internally or by a managed service provider
- Delivery of patches should include a back-up and test restore.
Case study: Always on Sunday for Schenker
The IT department at logistics and freight company Schenker was hit by the Blaster worm. Like millions of other businesses around the world, Schenker only realised there was a problem when network performance hit the floor.
"We were worrying about other things when we got hit, over a period of two days," says John James, the company's head of IT. "People were sitting twiddling their thumbs because their machines were grinding to a halt."
Previously, Schenker had two employees who worked full-time on patching, but James admits it was not seen as a priority. When Blaster hit, that all changed. Schenker had to send IT staff to each of its 1,700 global sites to manually patch and repair the infected PCs and servers.
The company considered using Microsoft Systems Management Server, but was unable to find staff with the right skills, and so began looking at specialised patch management software from Kaseya. The software runs on a centralised server and holds a repository of patch information, which is updated regularly. The software installs an agent on each device connected to the network, which scans the machine and provides a list of necessary patches.
It is possible to use the software to automatically deploy and install patches, but Schenker instead runs a scheduled patch update on a Sunday afternoon. "It means that if any machine is missing a patch, the agent will go and find it on a Sunday and fix itself," says James. "There is no impact on the business and we can see at a glance what has been done and where."