Microsoft searches for the data holy grail

Microsoft has taken its innovative, data-centric WinFS file system out of Longhorn just a year after including it. Has the...

Microsoft's next-generation operating system Longhorn is losing one of its most promising features: the WinFS file storage system.

The company first showed WinFS as part of an uncharacteristically early operating system preview a year ago at the Microsoft Professional Developer's Conference. It was to be a hugely important technology for the company, offering client-side users a unified data storage and search mechanism.

For example, instead of having to search within multiple applications such as e-mail, calendar, sales booking system, file and photograph folders for data about a particular contact, you could search in one place and retrieve all that data, linked to key events and dates.

You could easily find not only who you met at an event last summer, but also what photographs you took and what files you sent to whom afterwards. The system was based around an XML schema which held metadata about files, making it extensible to include custom data types.

At least, that was the plan until 27 August. Then, Microsoft pulled WinFS out of Longhorn and announced it would release the first beta of the storage system at the same time as Longhorn proper. The full release is likely to ship at the same time as Longhorn Server in 2007, said Microsoft staff.

"The ambition of rewriting the storage system has always been the holy grail," said Neil Macehiter, research director at analyst firm Ovum. "Reaching it is going to tax Microsoft. It would tax anyone, doing it at the scale that Bill Gates envisages."

One area affected by the removal of WinFS is likely to be the Microsoft Business Framework, which is a software layer designed to sit on top of .net, providing base-level functions for business applications.

"There was a strong dependency between the Business Framework and WinFS," Macehiter said. "If people have relied on WinFS, does this mean they are now back to the drawing board? Potentially it does."

On the other hand, it could be a bonus for the Business Framework in the long run. Mike Davis, senior research analyst at Butler Group, said one of the problems with WinFS was that it did not offer enough benefit when it was a client-only product. He said, "It is a gimmick. It really only gives corporate value when you can link it back to the server end."

Bill Gates and group vice-president for platforms Jim Allchin have said that part of the reason for the delay was because they wanted a server element.

"People said that if you do just the client, some of the scenarios will not work as well. We would really like the new synchronisation capabilities to go to the server as well, so we want client/server capabilities," said Allchin. He added that they also want access to tabular data.

Previously, synchronisation in WinFS simply meant synchronising data between clients on the Lan and other devices such as personal digital assistants. Moving it into the server arena gives it more power.

Similarly, providing tabular data access at the server would enable relational queries to be routed to server software through the WinFS APIs, making it easier for applications to talk to server applications through WinFS - doubtless something that the Business Framework developers would die for and certainly something that groupware suppliers such as IBM (Lotus) and Novell would be happy to use.

"I am sure IBM and other collaborative groupware suppliers are in conference negotiations with Microsoft over this," said Bill North, research director at IDC.

He said it is difficult enough maintaining compatibility with your own back-end applications without managing the complexity of a not-yet-completed file storage system with broad system implications.

As Microsoft tries to address these issues by making WinFS more server-friendly, it faces the challenge of making it easy to scale up. WinFS is not SQL Server, but Microsoft has based the system on some of the same underlying technologies as the database (as indicated by the reshuffling last year of Microsoft's top database engineer Peter Spiro to manage the WinFS team).

It gives the company a fighting chance, said Macehiter who, like Mark Quirk .net group technical manager at Microsoft, said customers are unlikely to need a standalone copy of SQL Server to use WinFS to access server data from the desktop.

"Some of the design is common, and they certainly share some of the same knowledge and expertise. I do not see that because you happen to be running it on the server you would need SQL Server on the server when you do not need it on the client."

Quirke underscored Microsoft's strategy to base all of its storage requirements on a single core engine using SQL Server, but emphasised that underlying elements of the SQL Server code can be used without trying to sell customers the full, packaged product.

Eventually, we will see products including Active Directory (currently based on Jet) and WinFS all using the same underlying engine from the SQL Server group. The transition of Exchange from the Jet database it currently uses to an SQL Server-based engine has also been on the cards for years.

As Microsoft thrashes out these file storage issues, Macehiter is telling his clients to hold off. "My advice is not to bank on WinFS, period," he said, predicting a three and a half to four-year wait before the technology is production-ready.

"Microsoft said there will be a beta, but I will not be banking on that now and basing my design decisions on a version of WinFS any time soon," he said.

Microsoft will be more hopeful about WinFS because it represents an ongoing attempt to move out of the application ghetto in which it and its customers have languished for the past 30 years.

Making data document-centric has been a goal for almost half that time, and Microsoft's failure to do it the first time around - with the Object File System in the mid-1990s - is a testament to the magnitude of the project.

Read more on Database software