Users in the frontline on software testing

Sivaramakichenane Somasegar, vice-president, Windows engineering services group at Microsoft, explains the input customers have...

Sivaramakichenane Somasegar, vice-president, Windows engineering services group at Microsoft, explains the input customers have in product development, and why the company may never produce software engineered entirely as plug-in components

Its software is used by more people worldwide than any other. Its latest operating system, Windows 2000, is allegedly more complex than the Nasa project that put Neil Armstrong on the moon. Yet the software produced by Microsoft has had a poor reputation in terms of quality.

One man has control for the overall project and release management of the Windows 2000 operating system family. His name is Sivaramakichenane Somasegar and he is the vice-president of the Windows engineering group at Microsoft. Somasegar manages the development groups for application compatibility, software development kits (SDKs), device development kits (DDKs) and system integration testing.

Microsoft has a reputation for rapid product rollout, which typically takes 12 months. "We have three to four months of development and eight months of testing," says Somasegar. Products begin life when a customer requests something new from a Microsoft product manager. Once a project has been given the all clear, says Somasegar: "Ideally I would want the testers and developers to work simultaneously from day one."

Testing is a complex process involving several stages. Somasegar says Microsoft uses functionality testing to check the project does what it is supposed to. Stress testing examines how the software works when pushed to the limit. Microsoft uses scenario testing to check each component within an IT system. On servers, Somasegar says, Microsoft runs long-haul tests over 28 days, to check how the server performs.

With all this internal testing, Microsoft still relies on actual users outside the company. A small selection of users join what Microsoft calls a Joint Development Programme (JDP). Somasegar says: "On Windows 2000 we had 50 to 60 customers who acted as extensions to our own development team." These companies, he adds, "came in at the design stage of the project."

The benefit in joining such a programme, says Somasegar, is that the users have more influence on product design. A further benefit is: "We will help them develop their own IT systems with the new Microsoft software." It is a deep relationship, he notes, with Microsoft providing product support and consulting services. Beyond the JDP, Microsoft typically runs a technical beta programme involving as many as 40,000 participants which tests if the new Microsoft software will work with various user software/hardware configurations. But with such large numbers involved, Somasegar says only the top 30 to 40 users will provide Microsoft with direct feedback.

With software as complex as the products Microsoft creates, changing one part of an application can have a knock-on effect elsewhere in the application. The dilemma at Microsoft - not uncommon in IT departments - is establishing what the effect will be of making one change. Somasegar says the way to overcome this is by building compartmentalised software where separate components perform a particular function, while the internal workings are hidden from view. But Somasegar concedes that Microsoft is, "not the best at building software this way".

An example where software is fully engineered as components is Windows NT where "each [software] component is isolated," Somasegar explains. While this approach offers clear merits in terms of software quality, it is unlikely Microsoft will ever make its software fully in this way. "If this was the main goal at Microsoft our rate of innovation would be slower," says Somasegar. "It is always a fine line."

And so, for a company such as Microsoft, that prides itself on innovation and the ability to develop more intelligent software, using entirely component-based software to produce higher quality software is perhaps just a pipedream.

Should customers have a greater role in software development?

Read more on Business applications