It seems like a bit of a mystery how open source software ever sees the light of day. The idea of a large number of people working for glory and the greater good rather than recompense seems too idealistic to be true.
In the early days this is exactly what happened, but now commercial companies have arisen, or switched over to open source, to act as the marketing arm of these communities. There are now a large number of paid workers at Red Hat, Novell, Ingres and other open source businesses and these corporations are beginning to act in ways that mirror their closed source counterparts.
Joe Brockmeier, Novell's openSUSE community manager, says, "We employ people to work on things that some of the volunteers are not necessarily going to jump to. For example, not many volunteers want to do long-term maintenance or go back and fix an old bug. Most want to develop an existing feature or something that does not exist yet. So we have to hire people to do some of the heavy lifting that volunteer contributors typically do not do."
Brockmeier describes the other strategies used to push ahead in the open source market. "One strategy is to identify the company that started a project and go ahead and buy it so you have a great deal of influence in the direction of that technology.
The other way is to identify a project and get involved to influence the project, as Sun Microsystems did with PostgreSQL when it hired Josh Burkus who was then a core team member for the product."
After two years, Berkus has decided to move on and will be replaced by Peter Eisentraut, another member of the PostgreSQL core team. This typifies the fluid nature of the open source community.
People move from project to project as their interests dictate and this can act as a hedge against the burn-out that full-time coders seem to suffer from in closed source companies and offers much more freedom. "Typically, there is a lot of give and take. Very few people in the open source community are willing to be hired on or dictated to," Brockmeier says.
In terms of acquisitions, the open source movement is maturing and this has been marked by the growing string of acquisitions of smaller companies with bright ideas. Notably. Novell bought Platespin to simplify virtual server implementation, and Red Hat acquired Qumranet for its KVM (Kernel Virtual Machine) product for virtual machine deployment and management.
Whatever route is taken to adopt and develop a project, the key requirement is a loyal following, a community. This is probably the greatest challenge in open source.
Graham Taylor, chief executive of OpenForum Europe, a business consultancy promoting the use of open source, says, "In the early days there were a number of companies which saw the open source, free software tag and suddenly announced they were going open source. All they did was to make their product available under an open source licence. The fundamental thing, that people now understand, is that if you do not have a community behind you and with you, it will fall flat."
Ingres, a former proprietary database company that open sourced its codebase in 1998 after its acquisition by CA, was a pioneer of the transition from closed to open. Although the company knew that a community was essential, it had no idea how to attract support. Thinking like a proprietary company, its board decided that a competition would do the trick.
Emma McGrattan, senior vice-president for engineering at Ingres, recalls, "We were wondering how we could get a community around Ingres from day one so we launched what we called the Ingres $1m Challenge. The competition was to build a migration toolset to take applications from Oracle, SQL Server, MySQL and other database companies. We thought $1m would attract hundreds of entries but we had fewer than 20 - it was disappointing. One of the lessons we learnt about communities at this early stage was that money alone does not buy it. You have to put a lot of work into building relationships with the community."
Now an independent company again, Ingres has successfully attracted its community within its 10,000-strong customer base and developed its own way of working with it. "We have created a roadshow where we take our engineers to meet some motivated people from our user groups.
They meet together in a room for a three or four days and the engineers work side by side with the community members to develop a feature that will fix a bug or add a feature that is interesting to that particular member," McGrattan says.
As an example of the results, she explains that Ingres' audit capability lacked the capabilities that a customer required to meet EU regulations. Although the database allows the auditing of all activity down to the row level, it could not record every time a certain row was selected for viewing.
But an EU regulation specified that a viewing of a judge's home address needed to be tracked and recorded. The Ingres engineers and the customer were able to put that in within the course of the three-day roadshow.
Some projects attract a community more easily than others. The high profile of the Linux kernel has attracted almost double the number of developers that worked on Microsoft Windows Server 2000 - but servicing a fifth of the number of lines of code.
The pivotal position of the kernel attracts engineers from all the major IT companies to ensure that they have a say in what lies upstream. The upstream concept cascades down through Linux. Changes in the kernel have to be anticipated by the downstream distributions and, beyond that, the applications.
Max Spevak is the community architecture manager for the Fedora Project, Red Hat's free distribution of Linux, and needs to know what is happening upstream in the kernel development as well as what Red Hat is adding to the distribution. In addition, applications included with Fedora need to be kept aligned with these planned changes.
The recruitment of developers for the project has been revamped by Fedora program manager John Poelstra. "What Poelstra has done makes the process the same for a community developer who may be studying in university or for Alan Cox [Fedora's star kernel hacker].
If you want a feature in Fedora, you go onto the Fedora Wiki where there is a section for each release. You write a brief description of what your project involves: use cases, implementation plan, requirements, specifications - standard stuff for any technical project," Spevak explains.
Once registered, a deadline timetable is given for that release and, if the feature is ready by the halfway point in the release schedule, it is reviewed by a group called the Fedora Engineering Steering Committee, formed from community contributors who have been given leadership roles in determining the direction of Fedora.
If the group feels the feature is in good shape, it stays in and goes through a bug fixing and polishing mode ready for release. Features deemed to be unready for advancement are rescheduled for the next release in six months' time.
Some projects need to be assigned, but the volunteer system ensures dedication. Attracting the developers requires a bit more work. "The more persuasive way is by presenting a vision that is compelling to make engineers keen to identify themselves with it that makes it work organically. Over the years, that has led to a lot of important work getting done and two-thirds of Fedora is now maintained by non-Red Hat engineers," Spevak says.
From the customer viewpoint, a lot has been made of the cost benefits of free software. Simon Phipps, chief open source officer and evangelist at Sun, denies that this is the main attraction. "It is not about getting free stuff. If you hear a CIO say he is going to reduce his costs by not having to pay for software licences any more, then he has the wrong idea. Open source is about having control over what you pay for, what you hire for, and what you do not pay for. That is a sustainable way of keeping costs under control as you are deploying software."
Open source is a financial model based on after-sales services and training. Closed source products rely on licensing as the main cash cow and every item used has to be paid for. If a database adds another server, it has to be covered by a licence. With open source a new server is provisioned and launched at no extra cost.
Open source offers a different approach to developing software products. Transparency, openness and community, which are inherent in open source projects, are becoming an ever more essential early step in the transformation of a technology idea into a product fit for business use.
The core of Linux, the kernel, is developed by over 3,700 individuals worldwide, one of the largest development communities ever assembled. The team releases a stable update quarterly with the aim of including significant new features, increased device support and better performance in each iteration.
The workforce comprises many individual developers, but it also engages engineers from over 200 companies that normally compete fiercely with one another in the market but work, for the most part, harmoniously on the kernel project. The top five companies contributing to the work are Red Hat, Novell, IBM, Intel and the gatekeeper of the Linux world, the Linux Foundation.
The Foundation is a non-profit consortium that emerged from the Open Source Development Labs and the Free Standards Group in 2007. Its charter is to foster the growth of Linux, originally developed by Linus Torvalds in 1991.
The first full release of Linux 1.0.0 was in 1994 and it had 176,250 lines of code. The code base has since swelled to just under ten million lines. Every day the kernel is changed, with an average of 3,600 lines being added, 1,500 lines removed and 1,500 lines changed.