Late May and early June was not a fun time for Adobe, or its users. Suddenly, news began hitting the wires that a vulnerability had been found in its Flash player. Thousands of hacked websites were redirecting browsers to attack servers that exploited the software to execute remote code.
Symantec reported it as a zero-day attack. Others said that the latest version of the player fixed the problem. IT departments were confused, and the stakes were high. Flash player sits on a lot of corporate machines.
Luckily, the exploit turned out not to be a zero-day attack, and the latest version of the player was immune. But it highlights some interesting questions for IT departments. How do they decide whether to patch a desktop vulnerability? How do they know what the patch should be? How do they get it out to remote machines, and if a patch is not available, what should they do?
Adrian Hodder, director of operations at IT services firm Computacenter, says remote desktop management and patching plays a big part in the firm's approach to technical support services. "You recognise what software version people are at, and rather than us having to send engineers to do the patches, we automate the distribution of patches at the appropriate time."
Surely that's the way it should always work? The sneakernet approach to desktop management - sending people out to handle machines remotely - is counter-intuitive in a networked environment. But remotely patching desktop machines has its challenges.
Before you can follow an automated desktop management approach, companies must understand what inventory they have. For many organisations, that will be a separate project that will present difficulties of its own.
"All of the decisions around desktop management rely on having visibility into what version of operating system or Office application devices have, and what other software could be installed that could conflict with patches that you want to deploy," says Hodder.
For that reason, it is imperative to have a standard desktop build, he warns. Being able go guarantee the same image on each machine (a capability supported, hopefully, by a locked-down desktop that cannot be substantially modified by users) will help when it comes to remotely managing those desktops.
When it comes to patching, says Brian Bourne, founder of Toronto-based Microsoft-focused IT consultancy CMS, companies might want to focus on a subset of their environment by dealing with Microsoft software. Given that most corporate desktops are Windows-based, it is a good way to tackle the majority of a company's desktop software portfolio in one fell swoop.
Windows Server Update Services (WSUS) still exists as an independent product for patching Microsoft software, says Bourne, but it has also been rolled up into the supplier's Systems Centre Essentials (SCE) and Systems Centre Configuration Manager (SCCM) products.
"When you look at the desktops, there is patching of the Microsoft-related stuff - Windows and Office - and then there is patching of everything else on that machine," Bourne says. "And then there is the patching of non-Microsoft things plugged into the network. In that category, you have routers, and other equipment. So it is a long list."
SCE and SCCM can also patch non-Microsoft products, but administrators must import third-party catalogues from other suppliers. "The Systems Centre range will not tell you about that software automatically," says Bourne.
For more advanced patching options that span a variety of systems, companies might consider an alternative third-party product. Scriptlogic provides remote desktop patching as part of its Desktop Authority system, says Jon Rolls, senior director of product management at Scriptlogic.
"We patch third party applications as well. All of these applications - Realplayer, iTunes, and so on - have known exploits," Rolls says. "We combine all of the patches into one database, and give you centralised configuration and reporting."
Administrators must understand both the hardware that is in the enterprise and the software that runs on it. But the challenge is far from over at that point, says Bourne.
"Then, it becomes a matter of how you know what needs updating," he warns. The Flash exploit that did the rounds was public knowledge, but you can bet that a lot of IT managers did not read about it. Keeping Internet Explorer up to date with its patches isn't enough.
"You are just as vulnerable with fully patched Internet Explorer if you have the Adobe plug-in," says Bourne. "So you need to figure out that there is a new Flash, and that the previous one was vulnerable, and then you need to figure out a way to distribute it."
Keeping track of threats
Services such as Secunia, BuqTraq, and the CERTs are useful resources to keep track of what needs patching.
"We take patch information from a multitude of vendors, and we collate that in an automated way, and then work with customers under porfolio management to deploy the right patches to the right corporate standards," says Hodder.
Supplier-agnostic patch management systems from firms such as Lumension can provide patch libraries by pulling information in from multiple suppliers. Its service regularly updates a hosted library with patch information. This can then be accessed by local implementations of the Patchlink software, which in turn remotely patches devices in the organisation.
Vulnerability scanning systems (which Patchlink includes) can also probe desktops for potential problems, matching the equipment they find against known flaws. It is difficult for such systems to provide full protection, because of the risk of zero day attacks, but they at least up the ante by rooting out candidates for patching.
But once those candidates are found, and matched with the appropriate software upgrades, how can you be sure that the updates prepared for desktop applications are not going to break the machines? With so many different desktop configurations, software conflicts are always a possibility.
"Large enterprises often do not distribute patches without a formal testing and release schedule," says Bourne, adding that smaller organisations rarely have the resources to handle that. "The worst case scenario is where all the software on the machine updates itself."
Companies with limited resources should therefore weight up the potential impact of a problem occurring with a patch. How many machines will be infected if a fix goes bad? How easy is it to roll the patch back? Some suppliers are better at rolling back than others - a lot depends on the installer technology used, says Hodder.
If testing prevents a collection of machines from being patched immediately (or if a patch doesn't yet exist), IT departments can still take steps to protect their users. This is one area where Windows Vista's enhanced group policy features can come in useful. Workarounds can be pushed out to machines until the problem can be patched. When the initial panic over Adobe's security hole set in, on-the-ball IT departments would have remotely disabled Flash players using group policy settings (if they had such capabilities in the desktop operating system).
New firewall policies can also be set to solve some potential security problems. File permissions can be changed. Most workarounds that can help to solve a problem before a patch appears can be carried out with Group Policy.
The real problem may be finding the machines that are affected by a patch in the first place, and contacting them. "The biggest problems are those unmanaged machines the mobile user that connects once in a blue moon," Bourne says.
Network access control
James Quinn, an analyst at Info-Tech research, says network access control is starting to take hold among businesses. Pioneered by Cisco, Microsoft and security firms such as Symantec, the concept uses a policy server and authentication server to check mobile devices as they attempt to connect to the network. If the machines fail certain tests (such as having up to date operating system or application files, for example) they can be quarantined before being patched.
Those mobile machines do not have to connect to the network physically - they can be forced to update when they connect from outside the organisation. Application suppliers are getting better at forcing machines to connect to a server when using the software, says Bourne.
If desktop applications do not force remote machines to connect, there are other alternatives. Scriptlogic's product has a web service that customers can publish themselves. An agent installed on remote machines is programmed to "phone home" to the service and check for patches at regular intervals.
Some off-site machines can still be difficult to reach, even if they are stationary. Remote branches will often have several machines and no server, for example. There are solutions to minimise bandwidth usage by eliminating the need for each machine to download a patch independently from headquarters.
1E offers a system called Nomad Enterprise that works with Microsoft SCCM. A master PC at the remote site downloads the necessary patches and then acts as a cache from which all the other machines in the branch can update themselves.
Even on an office Lan, patching can be tricky. Many patches will take place overnight to avoid disruption, but machines will ideally be turned off then to save power. How can you patch a machine that's off?
With the development of the Core architecture, Intel introduced vPro, an out-of-band management system that, like its previous incarnation, Wake-On Lan, can be used to turn on machines. However, this one works across subnets, making it truly useful for larger organisations with overnight patching requirements. Now, management software can wake up machines and implement the patches without any manual intervention.
Facilities like these can help to sustain the competent remote desktop patching skills that are so vital, but patching capabilities are far from enough. The Flash player emergency turned out to be solvable with a single patch, (if executed properly), but the next time an alleged zero-day exploit happens along, it might be genuine - and it will be the broader skills around remote configuration, rather than simple software updates, that will save many PCs from trouble.