Solving the patch puzzle

New software should be perfect and ready to go, like any other new purchase. But it is often not until after it has run in the real world that weaknesses become evident. Be prepared to spend a substantial amount of IT time managing the repairs and updates, writes Nick Langley.

The word "patch" harks back to a bygone era; an age of make-do and mend from a time before everything from white goods to clothing had built-in obsolescence. Except in software, that is. The newer a software product is, the more likely it is to contain gaping holes in urgent need of plugging.

January's Slammer worm assault exploited one such hole, in Microsoft's SQL Server. The company had issued a patch for it in July 2002, but those who fell victim had not applied it. When anti-virus specialist Sophos polled 200 business PC users, 64% blamed their fellow system administrators for what happened.

But they look less culpable when you know that the vulnerability Slammer used was one of 74 security alerts Microsoft issued last year. Applying patches is not a trivial exercise. Only last month it issued another serious alert over a vulnerability in Windows 2000 that could allow hackers to take remote control of PCs, and the patch for that contained glitches that affected some users.

To quote Microsoft's security operations guide for Windows 2000: "If you decide that risk must be minimised at all costs, you could follow a strategy of shutting down all production systems every time a new vulnerability appears in your software. You may then choose to not start the systems again until extensive testing has been done. Once a patch is released you need to determine the risk of deploying it immediately against the costs of keeping services down or unprotected. If you do decide to test, you need to determine how much testing you can afford to do before the risks of not deploying outweigh the risks of deploying."

The trouble is, the systems you least want to put at risk are the ones your business most depends on, so you cannot afford to take them down for long. In other words, the more critical the system, the more risks you will have to take to keep it running.

Jonathan Mitchell, director of business process and chief information officer of Rolls-Royce, is chairman of the corporate IT user group, The Infrastructure Forum (Tif). He thinks suppliers have dumped responsibility for their shortcomings on their customers. "It makes us nervous about exploiting technology in the critical aspects of our business," he says. According to Mitchell one large Unix server provider issued more than 600 patches between July and December last year. "When you install a patch on a high integrity system, how do you know it is going to work properly? If you go through lists of patches from the main Unix suppliers you will see how many are issued to correct vulnerabilities created by earlier patches."

Patch management drives up the cost of doing business says Mitchell. Staff time is consumed by vigilance, risk assessment and testing. And when a virus or worm gets loose, the whole IT effort may be diverted to deal with it. When Slammer struck, one Tif member had more than 100 people working round the clock over an extended weekend of three-and-a-half days.

Tif has put together its own security group. "If one company gets hit, we immediately start exchanging information, so as the situation unfolds we can rely on collective experience to try to deal with it. That is what we have been driven to," says Mitchell.

Graham Cluley, senior technology consultant at Sophos, has detected the makings of a similar backlash. "Systems administrators are saying, 'Although we are partially to blame for this, Microsoft did not make it that easy for us, because we have either not got much confidence in the quality of the patches, or it has not put the necessary technology in place to easily roll them out'. There have been cases when the patches from Microsoft have not worked properly, and as with any new piece of software you are installing, it may create conflicts with existing software."

The problem may be that people at board level are not putting enough resources into IT security, Cluley says. And perhaps security is being ignored in favour of trying to make more cash, while keeping fingers crossed. "Sales and marketing may say, 'We do not want to take our web server down for x hours, because we stand to lose business'."

Another problem is the sheer number of warnings about vulnerabilities that IT managers have to deal with. "In Microsoft's defence, when it issued the [Slammer] patch, it said, 'This is a serious vulnerability'. But in the flurry of warnings, people may not have recognised its importance," Cluley says.

But different vulnerabilities will affect different companies, he says, depending on whether you use SQL Server, or are highly dependent on the Simple Network Management Protocol. "You cannot wait to be spoon-fed, nanny-style, and be told which ones are good for you and which can wait until later."

Cluley thinks Microsoft has got it partly right. "It has an easy-to-subscribe-to mailing list, and it is pretty open about the severity of some of the vulnerabilities in its software - probably more so than other suppliers. But less than half the people in our survey subscribed. It amazed us how many people heard about vulnerabilities from mainstream news, as if they were waiting for Trevor McDonald to tell them."

Tarek Meliti, technical director of server hosting group TDM, has to deal with patch management on behalf of all his customers. He says you can take steps to minimise risk and effort: 

• Disable services you can do without
• Do not make your database directly accessible from the internet
• Monitor suppliers' patch updates weekly
• Make sure that you absolutely need to apply a patch, and even then hold off if you can: do not be a guinea pig
• Never do anything you cannot roll back from and return to the point before the patch was applied.

Patches for Windows and Solaris are regularly bundled in service packs, and generally TDM waits for these, rather than applying raw individual patches at their first appearance. "If a customer has lived with an issue for weeks, they can wait another two or three days to make sure it is stable and tested. We had one customer which applied every single patch, and the machine went down and would not come up again."

Meliti says patches can be applied with little disruption to service. Having taken a back-up, you can transfer the system to a standby server while the production server is patched. If your servers are clustered, you can take them off in turn and patch them. "But in our experience very few systems cannot come down for five or 10 minutes at quiet times." Customers generally understand that the small interruptions in service are not TDM's fault. "But when the system is down, people can get angry."

Patching is not a great drain on TDM's resources. Two people are responsible, and it is far from a full-time job. TDM enjoys economies of scale: when the patch is applied once, all the customers benefit. "It would be a lot more effort for individual customers."

However, Aberdeen Group estimates that US businesses spend $2bn (£1.3bn) a year on patch management. Tif, which includes 135 members of the FTSE 250 group, is getting increasingly worried about these costs. "We are going to apply pressure for best practice, to force the industry towards better design," Mitchell says. "If software remains flaky and shaky, what foundations are we building our businesses on?"

But Cluley says users are accepting that they have some responsibility. "I do not think anyone expects perfect software. But people have a right to expect that software they buy will be free from well-known vulnerabilities such as buffer overflows."

Conflicting views on cyberterror

Internet security experts are at war with one another about cyberterrorism. On one hand you have organisations such as the US government and the UK consultancy Mi2g, which say it is not a question of if, but when parts of our infrastructure - such as power, banking, telecoms - are attacked from the internet. Mi2g asserts that the number of digital attacks around to world rose by 230% in 2002.

But anti-virus supplier Symantec says there were fewer attacks. And some independent consultants are calling into question whether cyberterrorism exists at all, and whether a man with a laptop in a cave can really break into some of our most securely established systems as alleged.

Pete Simpson, manager of Clearswift's Threatlab, is among the doubters. "There is a lot of nonsense being published on the subject. The threat of anybody getting into those systems from the internet is the square root of zero. They were designed before the internet, and have no connectivity externally. And they have got fairly robust security design. It would need inside knowledge, and even inside collaboration, to get at them."

He says the idea of cyberterror has been most strongly fostered in the US. "It has certainly improved budgets for the cyberterrorism tsar. When you look at what constitutes cyberterror attacks, you find it includes website defacement."

Threats do evolve, but nothing remarkably new has appeared recently, Simpson says. People would do better to concentrate on day-to-day security, and efforts should be put into combating the real risks, worms such as Slammer, Code Red and Nimda, which do not present a pattern that an anti-virus signature can be compared with.

Where to go for the latest advice on patches

• www.cert.org/ website at Carnegie Mellon University carries reports of newly identified vulnerabilities 
• Microsoft's vulnerability mailing list 
• Sun's support site, including Patch Portal with two-weekly patch reports, patch management tools, etc 
• www.Linux-Sec.net/ source of Linux patches 
• www.mimesweeper.com/support/threatlab/ alerts and tools for Clearswift customers 
• Anti-virus suppliers such as Symantec (www.symantec.com) and Sophos (www.sophos.co.uk) operate subscription-based virus updating services. Their sites also carry security alerts.

It is not just Microsoft - nobody's perfect

From the sheer number of column inches printed, you would assume Microsoft's software had far more vulnerabilities than other suppliers. Yet a glance at the lists of reports and patches Sun produces every two weeks shows that Solaris users, too, have a major job keeping up. And if you think open source has clean hands, take a look at www.linux-sec.net, or talk to Apache users hit by the Slapper worm last year. 

Oracle made itself a target by claiming that its servers were unbreakable - an irresistible challenge to some hackers. On Valentine's day, Oracle released the latest in a series of security patches for Oracle9i Database Server and Application Server. 

It is impossible to say which supplier has the worst record. More people use Microsoft products, so flaws are more likely to be discovered. According to Mi2g, Mac OS and some varieties of Unix are less vulnerable than Windows and Linux, but relatively few people use them.  

The problem may be the sheer pace of innovation, which does not leave time for exhaustive product testing. The current software industry business model drives suppliers to bring out new products at ever-decreasing intervals, and to drop tried-and-tested ones. According to Jonathan Mitchell of Tif, this is driving some user organisations back to IBM mainframes and AS/400s, where such churn-and-burn practices do not apply.

Top tips for server security

There are tools to help you to manage and monitor patches from Microsoft, Sun and independent sources. But before you can use them, you will need an up-to-date inventory of what versions of operating systems and applications are running on your servers, which bits are active, and who is responsible for maintaining them.   Tarek Meliti, technical director of server hosting group TDM, says the following practices work for him:  

• Make sure you only have the services you need running on your server, and disable all other services. Each service opens your server to different vulnerabilities 
• Only open firewall ports that need to be open. Ensure all other ports are closed to all but trusted IP addresses  
• If you have an e-business solution that comprises web and database servers, make sure that only the web servers are accessible from the internet, and that only the web server can access the database server. Applications that require the database server to be accessible from the internet would have fallen victim to the Slammer worm 
• Make a list of all applications that reside on your server, noting the version and any patches. Monitor suppliers' sites for patch updates on a weekly basis. Test patch updates and, unless they are critical, wait a couple of weeks before applying them to ensure the patches themselves are stable 
• The golden rule of updates applies to patches: if you cannot roll back, do not do it.
  

Sys admin comes under fire in Slammer survey   

Sophos asked 200 users about their experiences of Slammer: 64% blamed other systems administrators; only 24% said Microsoft was mostly to blame. Here are some of their responses: 

• "All software contains bugs. Microsoft's gets tested more than most. It is part of the systems administrator's job to keep up-to-date with patches. Not doing this puts them into P45-mode" 
• "The worm author is the first person to blame, and administrators who do not apply available patches are in second place" 
• "Some patches require undesirable downtime on systems and, as a result, are only applied when it becomes important to do so. It is a question of avoiding downtime without being left open to attack" 
• "We do not always apply patches immediately, due to problems these can cause. We take a wait-and-see approach, unless there is a pressing need to apply the patch" 
• "I do not see the point of waiting too long to apply patches. For some it seems to be only a matter of time before someone exploits it and causes a major problem" 
• "System administrators could shoulder some of the blame but if every potential problem is responded to, you spend more time worrying what could happen than carrying out core tasks - making the company money" 
• "If system administrators had the time and resources to fix every vulnerability or system problem then it might be their fault. In the real world they can only do the best they can, and Microsoft has to produce software people can afford, rather then 100% bug-free code that would cost a fortune" 
• "Who is to blame? The people who waste time writing viruses and releasing them into the wild" 
• "Why is access not being restricted at the firewalls? I wouldn't allow SQL queries across the wire from untrusted hosts in the first place" 
• "Having a good software update and deployment plan is crucial to getting security updates applied to all our servers in a timely manner."