The accuracy of two recent reports comparing the
relative costs and benefits of the Linux and Windows operating
systems has been called into question.
The reports, Forrester Research's "Is Linux more Secure than
Windows?" and a Yankee Group survey on the relative costs of
running the two operating systems, were both issued in the past few
days.
The security study - whose raw data was vetted by Linux
distributors Debian, Mandrakesoft, Red Hat and SuSE Linux - found
that on average, Microsoft patched flaws faster than Linux
suppliers.
The Yankee Group survey reported that, except for small
businesses with customised vertical applications, companies
deploying Windows enjoyed a lower cost of ownership than those with
Linux.
However, Linux distributors involved in the Forrester study have
issued a joint statement calling the study's conclusions
inaccurate. And the Yankee Group's methodology has been called in
question, with critics arguing it could not have possibly delivered
objective results.
Yankee's survey, it turns out, was based on the responses given
by companies that had been selected from a mailing list devoted to
Windows issues. The survey was funded and carried out by Sunbelt
Software, a supplier of Windows utilities, which publicised the
survey through a mailing list called W2Knews, which bills itself as
"The world's first and largest e-zine designed for NT/2000 System
Admins and Power Users".
Sunbelt itself clearly identified the survey as being aimed at
Windows system administrators. In the 16 February edition of
W2Knews, which launched the survey, the company said it and Yankee
Group were "surveying Windows Sites" to see how they were
"responding to the Linux phenomenon and the TCO question".
The survey was carried out via an online form, which contained
no controls and so was open to manipulation. Yankee supplemented
the raw figures with in-depth executive interviews taken from the
list of survey respondents, who were all subscribers to
W2Knews.
As such, the survey can only be said to be representative of
system administrators already using Windows, rather than sysadmins
in general.
In the executive report, its author Laura Didio wrote that "a
significant Linux deployment or total switch from Windows to Linux,
would be three to four times more expensive and take three times as
long to deploy as an upgrade from one version of Windows to newer
Windows releases".
However, Linux supporters said that such a claim knowingly gives
only part of the picture to build the notion that Windows is
cheaper than its open-source alternative. The survey failed to
consider other important factors in switching operating systems,
such as the freedom of choice that Linux makes available, since
companies can easily change suppliers and support contractors.
These benefits are more readily recognised by chief information
officers and IT directors, said Red Hat's European marketing
director Paul Salazar, who added that the Windows-to-Linux focus
was not representative, claiming that Red Hat (which controls about
70% of the Linux market) would rarely pitch Linux as a cheaper
alternative to Windows servers.
Instead, he said, the major opportunity for Linux is the huge
installed base of Unix servers. In this case, Linux costs less,
runs on cheaper hardware and is more compatible than both Unix and
Windows.
"With Windows it's never a night and day comparison," he
added.
The Yankee survey is the latest to compare the total cost of
ownership of Windows and Linux, but is the first (unlike those from
Jupiter Research, Forrester and IDC) that have not been requested
and funded by Microsoft.
Forrester's security study is a somewhat different matter. The
research firm was eager to distance itself from the furore
surrounding earlier publication of its Microsoft-funded research,
which led Forrester to bar companies from publicising research they
themselves had backed.
The company allowed Linux distributors to scrutinise its raw
data, a database of all the security vulnerabilities for Linux and
Windows over the course of a year, and made the data publicly
available.
As a result of this collaboration, Linux suppliers accept that
the raw data is correct, but in a public statement this week they
said Forrester's analysis had led to "erroneous conclusions".
The report compares the "days of risk", calculated as the number
of days between the disclosure of an operating system vulnerability
and the release of a patch, for Windows and several Linux
distributions. Microsoft took on average 25 days to release a
patch; Red Hat and Debian 57, SuSE 74 and MandrakeSoft 82,
Forrester said.
The Linux distributors claimed, however, that such figures are
flawed, because they use a straight average and take no account of
how significant the security holes are. As such, obscure, low-risk
problems that do not need immediate fixing are treated the same as
highly critical flaws.
"Our users will know that for critical flaws we can respond
within hours," a statement issued by the suppliers said. "This
prioritisation means that lower-severity issues will often be
delayed to let the more important issues get resolved first. The
average erroneously treats all vulnerabilities as equal, regardless
of the risk they pose."
Forrester analyst Laura Koetzle, who authored the report, said
she had considered giving critical vulnerabilities extra weight in
the average, but decided against it.
"I considered responsiveness, or days to fix, relative severity
and thoroughness separately, partly because I wanted the scoring to
be exceedingly easy to understand and transparent for the readers,"
she said, adding that readers were free to analyse the raw data in
this way.
The report distinguishes high-risk from lower-risk
vulnerabilities, but the distinction was not included in the key
average figures. The Linux suppliers also criticized the report's
definition of high-risk vulnerabilities, arguing that it included
numerous routine bugs.
"This is one of the worst cases of doublespeak out there," Red
Hat's Salazar said. "It's exceedingly difficult to peel through
those statistics."
The important thing, he said, was to make sure customers were
able to have secure systems, and Red Hat was succeeding at that.
"From our point of view, there's no crisis," he added.
If the open-source community wishes to see what in its eyes
would be a more accurate reflection of the true costs of Windows
and Linux, it could do worse than commission and fund its own
independent review into the market.
Matthew Broersma writes for
Techworld.com