To patch or not patch - that is the question for IT directors
who must balance the need for security against everyday operational
demands. Sally Flood looks at ways to fill software holes without
spoiling the overall picture
One Saturday morning in 2003, the Slammer worm was unleashed on an
unsuspecting world. In the first minute of its life, Slammer
doubled the number of machines it infected every eight seconds.
Within 10 minutes, 90% of all vulnerable machines had been infected
- leaving businesses with a £500m clean up bill.
The bill was all the more painful because Slammer hit a full six
months after Microsoft had released a patch to prevent it. If
everyone had patched their systems in the first place, Slammer
would never have happened.
So why are companies not patching? Simply put, many of them do not
have the time or the resources. Researching the 4,200
vulnerabilities published by US-based security monitoring body CERT
during 2003 (compared to the 1,200 vulnerabilities published three
years earlier) would have required more than 700 man hours. IT
services provider Computacenter estimates that assessing, testing
and rolling out a single patch can take up to eight weeks and cost
as much as £400,000.
Unfortunately, patching is a necessity. The time lapse between
vulnerabilities being discovered and exploited is becoming
increasingly short - Blaster was released just 18 days after the
vulnerability it exploited was discovered. "Not installing patches
is an extremely dangerous tactic," says Andy Kellett, a senior
research analyst with Butler Group.
The good news is that an increasing number of companies are putting
people, processes and tools in place to bring greater efficiency
and control to the patching process. US analyst firm Yankee Group
estimates that using effective patch management software and
processes can reduce the cost of managing patches from £150 per PC
to less that £100.
"Patching is not easy. It takes a lot of time, and we do not have
the physical or financial resources to stick to the best practice,"
says Pat Black, IT manager at Homefirst, a company offering
residential care services in Northern Ireland. Homefirst has a
network of offices in seven counties, with more than 50 servers and
1,000 desktop PCs. "We only have four IT staff to support the whole
system, and it is not possible for us to physically install patches
on all those machines and make sure they are kept up-to-date," says
Black.
Instead, Homefirst uses a patch management application from Sygate
that scans machines and updates patches automatically on a regular
basis. "If a machine does not have the right patches, the software
quarantines that machine from the rest of the network until it is
up to date," says Black. "It gives us a bit more breathing space,
so we are not running around installing patches the whole
time."
Nightclub group Ministry of Sound uses Microsoft Software Update
Services (the predecessor to Windows Update Services) to
automatically update patches at its London headquarters. "With SUS,
we get a notification once a week with a shopping list of updates
and we select the ones we need," says James Bacchuf, Ministry of
Sound's chief technology officer.
"They are automatically installed on a Wednesday afternoon, when it
will not affect network performance too much. The only thing the
user has to do is restart their machine when it prompts
them."
Ministry of Sound uses SUS to automatically update patches on all
its desktop PCs, but still has to manually patch its servers,
because they run on different operating systems. The company also
has problems patching servers running some legacy applications not
supported by Microsoft. "There is one machine running a radio
application that is quite old, and we did not patch that because we
could not get the software to run with the XP Service Pack 2
patch," says Bacchuf.
Despite the challenges, Bacchuf is a big supporter of automated
patch management. "Previously we were going online every week to
see what patches were released, reading news sites and bulletin
boards, but never being 100% sure that all the systems were
properly patched," he says. "Now IT staff can be developing
applications or providing support rather than running round trying
to install dozens of different patches each month."
Patch management software from specialists such as PatchLink and
Shavlik automates the process of finding, downloading and applying
patches. Different packages have different capabilities, but most
patch management tools include checks for updates, automated
deployment, roll back capabilities and centralised management of
patches.
The advantage of automated patch management is that it reduces the
risk of human error. However, the software does have its
limitations, says Kellett. "This is a new market, and different
applications have wildly different functionality," he says. For
example, some patch management tools allow for automated deployment
and group policies, while others will only alert you to a missing
patch.
Perhaps most importantly, patch management software is only as good
as the underlying security policy, says Bill Pepper, director of
security with CSC's risk management practice. "Before you deploy
any technology, you need to know as an organisation, how you want
to manage patches," he says. "What patches will you apply, to what
systems, and in what order? How will you assess new alerts?"
In the event that an outsourcing company manages your system
updates, do not think you can relax and forget about patch
management. "You can outsource the danger, but you cannot outsource
the policy," says Pepper. IT directors must agree with outsourcers
on how patches will be applied, when and with what priorities and
how that fits with the business.
Patch management policies should be based on a comprehensive risk
assessment. This means knowing what the potential vulnerabilities
are, and how important it is to the business that they are fixed.
"Your policy should take account of what is a critical system to
the business, and whether the cost of patching a hole outweighs the
potential cost to the business," says Pepper. For example,
retailers often do not apply patches in December because it is the
busiest time of year, and installing new software presents an
unacceptable risk to the business.
Risk management should also specify which patches will be tested,
for how long, and to what degree. In a worst-case scenario, firms
may find they have to develop their own patches for custom or
legacy applications, or to leave the system unpatched. "Microsoft
in particular is becoming more bullish about not supporting legacy
operating systems in its patches," says Pepper. "If you cannot
afford to upgrade to newer operating systems, often you end up not
applying patches at all."
Some suppliers, such as SAP and Microsoft, offer support to help
companies resolve conflicts between patches and third-party
systems, so it is important to speak to suppliers when drawing up a
patch management policy.
It is also important to speak to suppliers about how they classify
patches, and what is included in the increasing number of jumbo
patches distributed by suppliers on a monthly or even quarterly
basis. For example, Oracle recently revealed it will begin
releasing software patches on a quarterly schedule, so customers
can better plan for them. Microsoft began issuing monthly patches
in October 2003, and Computer Associates and SAP have been on
regular schedules even longer.
This is good news in the respect that IT departments do not need to
deal with dozens of different alerts. However, sometimes it is
difficult to know exactly what is included in a patch, and
inadvertently fail to apply something that is critical. To this end
John Meakin, group head of information security at Standard
Chartered Bank, is using Qualysguard web-based scanning technology
to help prioritise patch management.
"For banks such as ours with many large and complex
interconnections to the outside world, it is vital to carry out
effective patch management, but absolute security is an impossible
goal," says Meakin "Our aim is to achieve the right level of
security through implementing an appropriate risk-based strategy
for each part of our global network, and we use Qualysguard as a
tool to underpin this process and prioritise patches."
Pepper also says that selective patching has to be done with care.
"We know from experience that patches often fix more than they say
they are going to," he says. "That can be a problem if companies
are only installing selected patches."
Equally, blindly applying patches without conducting a risk
assessment can be counter-productive. Pepper cites the example of a
UK bank that insisted on deploying a patch to a Unix box running a
legacy application. "The patch conflicted with the legacy
application, and payroll simply did not happen that month," he
says. "The cost of fixing that problem far outweighed the risk of
not patching immediately."
Once you have a clear patch management policy, consider what
procedures can be implemented before patching. "There is a lot you
can do to mitigate risk apart from patching," says Garry Sidaway, a
security analyst with CyberTrust. For example, IT staff can use
filters to strip off executable files from e-mails, and disable
applications such as instant messaging, WinZip and media players to
reduce the risk of introducing viruses to the network.
Intrusion detection systems,
click here >>
Patching things up
A good patch management policy should follow these six steps:
- Understand your environment and what the vulnerabilities are.
Not all systems are equally at risk or need to be fixed at the same
speed. This can be done using vulnerability testing software
- Build an infrastructure to deliver patches. Off-the-shelf
software can be used for non-critical systems, but it may be
necessary to manually update critical servers or legacy systems.
This can be done using patch management software or systems
management software
- Reduce risks by using intrusion detection software, anti-virus
and firewall software so that vulnerabilities will not be exploited
before you can apply a patch
- Consider using external monitoring to try to get early warning
of vulnerabilities and to filter real threats from irrelevant
alerts
- Test everything then test again, to make sure the patch will
work. This can be done internally or by a managed service
provider
- Delivery of patches should include a back-up and test
restore.
Source: Computacenter
Case study: Always on Sunday for
Schenker
The IT department at logistics and freight company Schenker was
hit by the Blaster worm. Like millions of other businesses around
the world, Schenker only realised there was a problem when network
performance hit the floor.
"We were worrying about other things when we got hit, over a
period of two days," says John James, the company's head of IT.
"People were sitting twiddling their thumbs because their machines
were grinding to a halt."
Previously, Schenker had two employees who worked full-time on
patching, but James admits it was not seen as a priority. When
Blaster hit, that all changed. Schenker had to send IT staff to
each of its 1,700 global sites to manually patch and repair the
infected PCs and servers.
The company considered using Microsoft Systems Management
Server, but was unable to find staff with the right skills, and so
began looking at specialised patch management software from Kaseya.
The software runs on a centralised server and holds a repository of
patch information, which is updated regularly. The software
installs an agent on each device connected to the network, which
scans the machine and provides a list of necessary patches.
It is possible to use the software to automatically deploy and
install patches, but Schenker instead runs a scheduled patch update
on a Sunday afternoon. "It means that if any machine is missing a
patch, the agent will go and find it on a Sunday and fix itself,"
says James. "There is no impact on the business and we can see at a
glance what has been done and where."