Forrester Research has changed its policy toward supplier-sponsored research following criticism of last month's publication of a controversial Microsoft-funded study which compared the cost of developing applications on Linux and Java with a Microsoft-based approach.
The policy change was announced in a letter written by Forrester chief executive officer George Colony, which has been posted to the Forrester website.
"We will no longer accept paid for, publicised product comparisons," Colony said. "The best example of that would be the Microsoft report."
The Microsoft-funded study claimed that developing certain types of web applications was 40% more expensive when Java and Linux were used rather than Microsoft software.
The survey was widely seen as a blanket statement about the cost of ownership of the platforms, rather than a more qualified statement about their relative costs for running certain types of applications, according to Forrester analyst John Rymer, the author of the report.
"There was a huge outcry about the Microsoft study," said Rymer, who blamed the way the media covered the report for much of the criticism. "'Microsoft cheaper than Linux'. That was the basic headline. There were a dozen variations on that. Obviously, if you read the report, the conclusions in the report are much more qualified than that," he added.
Critics also took exception to the fact that Forrester's results were based on only 12 companies, as well as the study's methodology.
With the policy change, Forrester will still do product comparisons for suppliers, Colony said, but it will not allow them to publicise the results, as was the case with the Microsoft-funded study, as well as a recent customer satisfaction study funded by Peoplesoft, also released in September.
Both studies were promoted by the suppliers' respective public relations departments. In Microsoft's case, the study was offered as the latest data point in the company's "fact-based" campaign against Linux, an initiative promoted by Microsoft general manager of platform strategy Martin Taylor.
"Martin Taylor went out and visited with a bunch of reporters, and he was referring to the study and using it to advance his case that Linux doesn't have a lot of advantages," said Rymer. "George [Colony] was uncomfortable with this."
Forrester stands behind the results of the study, but Colony admitted the company made a mistake in allowing it be publicised. "It just looks bad. It's a conflict of interest," he said.
Paul Onnen, chief technology officer at WebMD, said that he had no problem with the findings of the Microsoft study, which he found to be "relatively fair". But, he added, he generally gave little credence to vendor-sponsored reports.
"I am, in general, very leery of sponsored research because there's lies, damned lies and statistics, and you can make statistics look however you want them to," he said.
"What we need to do in the IT community is stop pointing fingers between the Java community and the Microsoft community," said Tim Brown, a systems architect with AirNet Systems. Brown believed a comparison of total cost of ownership is so difficult to quantify accurately that research firms would be better off steering away from these sort of studies.
"It's really hard to prove how one operating system is better from a TCO perspective," he added.
Robert McMillan writes for IDG News Service