Question mark hangs over Linux/Windows reports

The accuracy of two recent reports comparing the relative costs and benefits of the Linux and Windows operating systems has been...

The accuracy of two recent reports comparing the relative costs and benefits of the Linux and Windows operating systems has been called into question.

The reports, Forrester Research's "Is Linux more Secure than Windows?" and a Yankee Group survey on the relative costs of running the two operating systems, were both issued in the past few days.

The security study - whose raw data was vetted by Linux distributors Debian, Mandrakesoft, Red Hat and SuSE Linux - found that on average, Microsoft patched flaws faster than Linux suppliers.

The Yankee Group survey reported that, except for small businesses with customised vertical applications, companies deploying Windows enjoyed a lower cost of ownership than those with Linux.

However, Linux distributors involved in the Forrester study have issued a joint statement calling the study's conclusions inaccurate. And the Yankee Group's methodology has been called in question, with critics arguing it could not have possibly delivered objective results.

Yankee's survey, it turns out, was based on the responses given by companies that had been selected from a mailing list devoted to Windows issues. The survey was funded and carried out by Sunbelt Software, a supplier of Windows utilities, which publicised the survey through a mailing list called W2Knews, which bills itself as "The world's first and largest e-zine designed for NT/2000 System Admins and Power Users".

Sunbelt itself clearly identified the survey as being aimed at Windows system administrators. In the 16 February edition of W2Knews, which launched the survey, the company said it and Yankee Group were "surveying Windows Sites" to see how they were "responding to the Linux phenomenon and the TCO question".

The survey was carried out via an online form, which contained no controls and so was open to manipulation. Yankee supplemented the raw figures with in-depth executive interviews taken from the list of survey respondents, who were all subscribers to W2Knews.

As such, the survey can only be said to be representative of system administrators already using Windows, rather than sysadmins in general.

In the executive report, its author Laura Didio wrote that "a significant Linux deployment or total switch from Windows to Linux, would be three to four times more expensive and take three times as long to deploy as an upgrade from one version of Windows to newer Windows releases".

However, Linux supporters said that such a claim knowingly gives only part of the picture to build the notion that Windows is cheaper than its open-source alternative. The survey failed to consider other important factors in switching operating systems, such as the freedom of choice that Linux makes available, since companies can easily change suppliers and support contractors.

These benefits are more readily recognised by chief information officers and IT directors, said Red Hat's European marketing director Paul Salazar, who added that the Windows-to-Linux focus was not representative, claiming that Red Hat (which controls about 70% of the Linux market) would rarely pitch Linux as a cheaper alternative to Windows servers.

Instead, he said, the major opportunity for Linux is the huge installed base of Unix servers. In this case, Linux costs less, runs on cheaper hardware and is more compatible than both Unix and Windows.

"With Windows it's never a night and day comparison," he added.

The Yankee survey is the latest to compare the total cost of ownership of Windows and Linux, but is the first (unlike those from Jupiter Research, Forrester and IDC) that have not been requested and funded by Microsoft.

Forrester's security study is a somewhat different matter. The research firm was eager to distance itself from the furore surrounding earlier publication of its Microsoft-funded research, which led Forrester to bar companies from publicising research they themselves had backed.

The company allowed Linux distributors to scrutinise its raw data, a database of all the security vulnerabilities for Linux and Windows over the course of a year, and made the data publicly available.

As a result of this collaboration, Linux suppliers accept that the raw data is correct, but in a public statement this week they said Forrester's analysis had led to "erroneous conclusions".

The report compares the "days of risk", calculated as the number of days between the disclosure of an operating system vulnerability and the release of a patch, for Windows and several Linux distributions. Microsoft took on average 25 days to release a patch; Red Hat and Debian 57, SuSE 74 and MandrakeSoft 82, Forrester said.

The Linux distributors claimed, however, that such figures are flawed, because they use a straight average and take no account of how significant the security holes are. As such, obscure, low-risk problems that do not need immediate fixing are treated the same as highly critical flaws.

"Our users will know that for critical flaws we can respond within hours," a statement issued by the suppliers said. "This prioritisation means that lower-severity issues will often be delayed to let the more important issues get resolved first. The average erroneously treats all vulnerabilities as equal, regardless of the risk they pose."

Forrester analyst Laura Koetzle, who authored the report, said she had considered giving critical vulnerabilities extra weight in the average, but decided against it.

"I considered responsiveness, or days to fix, relative severity and thoroughness separately, partly because I wanted the scoring to be exceedingly easy to understand and transparent for the readers," she said, adding that readers were free to analyse the raw data in this way.

The report distinguishes high-risk from lower-risk vulnerabilities, but the distinction was not included in the key average figures. The Linux suppliers also criticized the report's definition of high-risk vulnerabilities, arguing that it included numerous routine bugs.

"This is one of the worst cases of doublespeak out there," Red Hat's Salazar said. "It's exceedingly difficult to peel through those statistics."

The important thing, he said, was to make sure customers were able to have secure systems, and Red Hat was succeeding at that. "From our point of view, there's no crisis," he added.

If the open-source community wishes to see what in its eyes would be a more accurate reflection of the true costs of Windows and Linux, it could do worse than commission and fund its own independent review into the market.

Matthew Broersma writes for

Read more on Operating systems software