Firms in the UK and US were left struggling to restore their IT systems after an error in McAfee's anti-virus signature updates led to vital system files on servers and desktops being deleted or put into quarantine.
The error in the update earlier this month caused McAfee's anti-virus system to wrongly identify a range of files, including Oracle, Excel and Adobe, as infected.
Businesses said this week that the incident caused serious damage. The risk manager at one large City-based company told Computer Weekly his firm had been "damned lucky" not to have lost thousands of business-critical files. He said 300 of his firm's 1,400 servers were affected by the error, which led to over 4,000 files being quarantined by mistake.
"It was time consuming to go to each server and move the files back to where they should have been. It took six hours. The files were system files, and if we had rebooted, the servers would have crashed," he said.
A systems administrator at a US university said it took a week to get systems back to normal after the anti-virus program deleted 3,000 files from 57 computers.
"Should this type of problem recur, I think I speak for many organisations when I say we would consider using another anti-virus provider," he told Computer Weekly.
Other firms posted their experiences on internet news groups.
"At my company, tens of thousands of files were deleted from dozens of servers and about 2,000 user machines. Affected applications included MS Office, and products from Shared, MapInfo, Macromedia, MySQL, CA and Cold Fusion," one IT professional reported.
A spokesman for McAfee said, "Since the incident occurred, McAfee Avert Labs has been working around the clock directly with customers - many of whom were proactively contacted - to help them assess the degree of impact and restore the files where possible."
McAfee said the problem was caused by an "unfortunate coincidence of several very subtle logic errors in a complex signature".
The company sent out a notification e-mail to customers within 70 minutes, and proactive e-mail notifications within two hours. New updates were posted within two hours and 20 minutes.
Plan in advance, says Sans Institute
Marc Sachs, director of the Sans Institute Internet Storm Centre, which monitors security issues, advised firms to avoid similar problems in the future by installing systems to pull back faulty anti-virus signatures.
"If a patch comes out and it is broken, you should have a method for undoing it. The same thing applies if you are sending out signatures: you should have a plan for pulling them back," he said.Firms should check new anti-virus signatures on test machines before rolling them out across the network, he said.