Patch Tuesday is perhaps the most anticipated and feared day of the month for network administrators and security managers. They wait eagerly for the next batch of patches from Redmond, glad to have some protection against attacks on the vulnerabilities that have popped up since the previous month's release. But they dread it too, and with good reason, given the massive amount of work involved in rolling out a dozen or more patches to thousands of systems.
Known in some circles as Black Tuesday, the second Tuesday of each month in the last few years has become a kind of national day of mourning in the IT industry, as admins call all hands on deck and load up on pizza and Red Bull for the long night ahead. Microsoft moved to a monthly patch schedule after some pointed requests by large customers who were having a hard time dealing with the steady and unpredictable flow of patches. And many IT managers still say they like knowing exactly what's coming down the pike and when. Indeed, the monthly bulk patch release has served to increase awareness of available security fixes in both the enterprise and the consumer market.
But given all of that, I submit that it may be time to rethink the concept of Patch Tuesday.
The main thing that both Microsoft executives and IT folks cite in their support of the monthly patch cycle is its regular schedule. IT staffs know days ahead of the patch release how many patches are coming, what products are affected and how severe the vulnerabilities are that they correct. This allows for planned downtime for patching critical systems, schedule adjustments for personnel to help with the patching and time to digest the magnitude of the fixes and decide whether any of them can wait.
But even with all of that advance notice and the knowledge of exactly when the patches will be available, many enterprises still don't deploy them right away. Admins are loathe to deploy any patch without first testing it, and rightly so. Even with Microsoft's improved QA process, some patches still cause regression errors or problems with other applications. The problem here is that while IT departments are testing the patches, attackers are busy reverse engineering the fixes and building exploits to throw at the vulnerabilities. (That is, those attackers who haven't already bought an exploit somewhere else.) There is anecdotal evidence that attacks against publicly known vulnerabilities spike in the hours and days after patches are released, meaning that those organizations that don't deploy the fixes immediately are at an increased risk of attack once the patches are available.
A hacker only needs one unpatched system, one little crack in the fence in order to launch a major attack on a given network. The sheer volume of the patches Microsoft releases each month makes it quite difficult for even the most conscientious IT department to get every patch out to all of the affected systems in a reasonable amount of time. Patch management systems can help, but they have their limitations as well.
Another key flaw in the monthly cycle is the very nature of the schedule itself. Even Microsoft has resource limitations and its Security Response Center staff can only build and test so many fixes in a given month. That means that some vulnerabilities remain unpatched for months at a time, even when there is exploit code publicly available and confirmed attacks going on. The DNS RPC flaw that Microsoft patched this week is a perfect example. The first reports of the vulnerability surfaced just two days after the company released its patches for April, and despite widespread reports of attacks against vulnerable machines, including a worm, Microsoft did not publish an out-of-cycle patch. So attackers have had nearly an entire month to hammer on the vulnerability, which Microsoft itself rated as critical.
Microsoft officials have made it clear that they're committed to the monthly patch cycle and have said that when events warrant, they will release out-of-cycle patches. When the company began the monthly schedule several years ago, those incidents that might have called for an unscheduled patch were rare, but that is no longer the case. Now, it's rare for a month to pass without the disclosure of a zero day in some Microsoft product. And those are just the ones that we hear about. Who knows how many other vulnerabilities the hackers are using at any given time?
So where do we go from here? Back to the future. The value of the predictability of the monthly schedule simply doesn't outweigh the danger to customers posed by the flaws that go unpatched for three or four weeks between cycles. If a new vulnerability in Windows is disclosed today, Microsoft has shown in the past that it can bring enough resources to bear on the problem to produce a high-quality patch within a week. It's time to return to that approach. Release patches when they're needed, not when the calendar dictates.
This was first published in May 2007