Over the years, there have been a number of approaches to provide end users with the information they need to make decisions. They've been variously labeled decision support systems, management information systems and executive information systems.
Because of their cost, they tended to be restricted to higher echelons, and often demanded both technical skills and knowledge of sophisticated statistical and analytical techniques.
With the arrival of Web technology, companies have a mechanism for deploying business intelligence (BI) throughout the organisation, cheaply and with a low support requirement. "Intranet and extranet adoption will bring enterprise decision support to a broader class of users, both within and outside the enterprise," said Henry Morris, the vice-president of IDC's Data Warehousing and Information research programme.
Server-based BI applications can be accessed from any device with a Web browser. But some major problems remain in integrating the various sources of data users might need. Kevin Magee is sales and marketing director of Information Builders, which has been providing query and reporting tools for multiple data sources since Larry Ellison was poor. "Most BI tools only access data stored in relational databases," he says.
"This means more than half their data sources are inaccessible, with 70% of all data still residing in legacy sources on mainframes." Information Builders' WebFocus uses the company's integration technology to access more than 80 data sources, including legacy, relational, and ERP data.
Steve Bavis, the business information solutions manager at the ERP solutions company Intentia, said far from being an up-and-coming technology, BI is already a "given" with 95% of new customers. Bavis cites one of Intentia's customers, Cummins Diesel, which, thanks to a BI system, discovered a new market for its products among existing customers.
"As companies continue to consolidate they will find themselves with dispersed information sources, often on multiple platforms, structured, unstructured, object orientated, relational. It's a real mess. However, suppliers are responding to the move toward more collaborative relationships between companies by making their software more open, in order to communicate with other systems and make it possible to merge data."
'E-Researchers' Survey.com give Oracle top spot in the overall data warehousing/BI software market, with over 50% of 600 study participants naming it the leading vendor. Clustered behind Oracle are Microsoft, IBM, and Cognos. "The market has been attracting many new entrants, but the leading suppliers, including Oracle, have been able to keep their positions strong," said Nancy Stewart, Survey.com's programme director for enterprise computing research.
She said the flood of new offerings is expanding choice and bringing the technology within reach of those who previously couldn't afford it. Stewart said BI specialists Brio and Business Objects are making good headway in the enterprise segment of the warehousing/BI market.
"Microsoft holds sway in many small and medium-sized businesses, and Oracle and IBM are strong in almost every segment, particularly at the enterprise level, so second-tier suppliers have their work cut out for them.'
Survey.com predicts that businesses will increase their annual spending on data warehousing/BI solutions by 74%, and in Europe by 85%, in the next three years. "The warehousing and BI market will weather the current economic downturn better than many other segments of the technology market," Stewart asserted, although she said spending on initial implementations is down by about 40% from last year: "This reflects in part the current volatile market, but also the change in the market. More, smaller sized solutions are being implemented, with more packaged software and cost effective components."
In a Gartner Group survey, more than 80% of IS professionals said providing tools for users to access data directly is a strategic priority. What they don't want to do is turn enthusiastic but inexperienced users loose on the organisation's production systems, with the risk of the fabled "query from hell".
One alternative is the data warehouse, a completely separate system which extracts data from the production systems and holds it in a form optimised for user queries. Data from many different sources has to be extracted, validated and "cleaned", and transformed into a suitable format for the data warehouse.
This can involve creating "multi-dimensional" databases which contain the same information as the production databases, but which can actually be much larger, because the information is presented in different ways to facilitate analysis. The data has to be regularly updated, or refreshed, at intervals that can vary from weekly or monthly, to near-real time.
Many organisations have set up smaller "data marts", which address the needs of specific business areas. These are much quicker to build. However, by definition, they are incapable of giving the big picture. There's also a risk that they will be set up in ways that give different interpretations of the same data, or that they take snapshots of corporate data at different times, so it's impossible to get a consolidated view across departments.
According to one of the BI market leaders, Cognos, the fact that current business cycles are six months or less makes data warehouses unviable, particularly since they must include new e-business data sources, as well as the legacy and ERP systems which most data warehouse implementation and management systems were designed for.
The best way to implement an enterprise business integration solution, Cognos asserted, is to isolate a single critical area and begin there, using data and business "dimensions" (such as time and geography) that can be re-used in future datamarts.
This will ensure consistency, and make it possible to link the datamarts to create a reliable big picture.
The success of data warehouses and marts depends on "metadata", which consists of information about where data is held, which reports contain it, when it was last refreshed, and when the next extraction from the source databases is due.
According to Survey.com, IBM is the leading supplier of data warehouse technology, with twice the market share of its nearest competitor. Customers find the one-stop shop approach, including consultancy and implementation from IBM Global Services, reassuring in a market fraught with stories of disastrous multi-supplier systems integration exercises.
IBM's solutions are based on the DB2 Universal Database, and include the Data Replication Family, and DataJoiner for multi-supplier database access. DB2 Warehouse Manager handles business metadata, and the Data Warehouse Centre is used to manage technical metadata.
Thanks in part to the introduction of smaller scale, lower cost technology by Microsoft, the datawarehouse/datamart business is doing well. "All segments of the data warehouse tool software market are contributing to the overall market's healthy growth," said IDC senior analyst Dan Vesset.
"However, information access is leading the charge. The gradual shift of revenue from warehouse generation and management to information access tools is a factor of the maturation of data warehousing. As buyers move from the initial stages of use to full deployment, spending priorities will shift from capturing and storing the data toward delivering the information to a wider range of users."
One of the techniques used for getting information from data warehouses, and sometimes operational databases, is data mining. This differs from straightforward query and analysis, since it can be used to find significant patterns that the organisation may have been unaware of. Like BI in general, data mining is being brought to the masses. First into the field with data mining built into the database was Microsoft, with SQL Server 2000, which offers algorithms for customer segmentation and predictive analysis, followed by IBM, and more recently, Oracle, whose 9i database comes complete with tools to analyse CRM data.
IBM's DB2 Intelligent Miner Scoring software enables database administrators to embed scoring algorithms directly into DB2. Scoring is used to rank items according to predetermined criteria, the most profitable or credit-worthy customers, for example.
Intelligent Miner Scoring is also compatible with Oracle databases. Partners such as Unica, Xchange, Angoss, the Centre for Data Insight, and LogMetrix are using it to develop analytical applications. The market for packaged analytical applications is booming, according to IDC, and the fastest-growing analytic applications sector is CRM.
Silvon, best known for data warehousing products for the iSeries, offers the Stratum analytical application suite, which covers customer analytics, marketing, sales, financial, procurement, e-business and manufacturing. "The drive for pre-packaged applications comes primarily from a need to increase the speed of implementation, reduce costs, improve the return on investment and put effective tools direct into the hands of users," said Silvon's European managing director Mark Mahara.
Mahara said such applications extract extra value from existing ERP investments. "They can also reduce the need for companies to change their ERP systems at all. Because they can draw data from multiple ERP systems, they are ideal at providing a bridge between companies within the same group. The fact that the ERP market has not really recovered from Y2K is partly down to the growth in analytical applications that are considerably cheaper and quicker to implement, and offer considerably more value for money."
Analytics also take pressure off IT, which no longer has to create reports for users, Mahara said. "Users can control their analytical environment themselves, without requiring any additional in-depth training. In broad terms, timescales for implementing analytical application are between one and three months."
The big ERP suppliers are moving into the analytic applications market themselves. SAP, PeopleSoft and Oracle all have their own. JD Edwards, working with BI tool supplier MicroStrategy, says its Business Intelligence suite can be used with other suppliers' core applications.
And with e-commerce booming, analytical applications are turning to the Web. "Because of the need to understand buying behaviour and build customer relationships, the basic server log based traffic analysis applications that once dominated the market are being replaced by more sophisticated web site analysis applications," said IDC's Vesset.
The focus is moving from delivering BI via intranets to Internet "portals", essentially virtual spaces where enterprises meet their customers, suppliers and employees. Modelled on public services like Yahoo, they can provide "personalised" access to information, based on an individual's role and status, or location. They support collaborative work, enabling people involved in projects, perhaps across companies, to share information.
"IT has historically been unable to provide an effective organising principle for the desktop, that co-ordinates multiple data sources, processes and people," said Ovum principal consultant David Wells. Web based publishing and corporate intranets are adding to information overload by capturing and distributing ever-larger amounts of information from multiple internal and external sources.
"The overall result is a fragmented view of information, and fractured business processes within the organisation," Wells said. "Portals represent a radical transformation of corporate information management, where the emphasis has shifted from creating and storing information in databases and repositories, to describing what information is available and relevant, and making it easily accessible to end users in a single interface."
IBM's Enterprise Information Portal (EIP), based on WebSphere, is described as "a foundation for access, not the interface". In other words, it's middleware, which requires user organisations to build or buy a custom portal on top. BusinessObjects has an InfoView for IBM EIP integration kit, to link IBM's middleware to its own portal technology.
"Business Intelligence and Knowledge Management can't be separated," said Barrie Pike, head of business intelligence at business technology solutions provider Conchango, who sits on the Microsoft Data Warehouse/Business Intelligence Partner Advisory Council.
"Portals are set to replace intranets, as they let you combine BI and KM, dealing not only in structured data, but also unstructured information; the stuff in people's heads."
Tools market growth
"The overall information access tools market will earn solid growth of approximately 21% compounded annually through 2004," according to IDC senior analyst Dan Vesset. "However, comparing the growth rates of each of the market's sub-segments will reveal significant discrepancies. End-user query and reporting, data mining, and online analytical processing will be fast growers; while executive information systems, spreadsheets, and statistical and technical data analysis will all be slow-growing markets."
Data warehousing project
The telco Energis is carrying out a huge data warehousing implementation which, only 15% of the way into the implementation, had already reduced customer churn by 1%. Energis' financial reporting time has been reduced from six weeks to one day. DataMirror's Constellar Hub is used to integrate data into the warehouse, which sits on a DB2 database and is driven by an IBM SP2 massively parallel server.
In a survey of 300 companies conducted by Delphi Group, top applications for Enterprise Information Portals were: Knowledge bases and learning tools 13.2%; Business process support 12%; Customer facing sales, marketing and services 11.7%; Collaboration and project support 10.2%; Access to data from disparate corporate systems 9.1%; Internal company information 9.1%; Policies and procedures 7.1%; Best practices and lessons learned 6.6%; Human resources and benefits 6.4%; Directories and bulletin boards 5.8%; Identification of experts 4.6%; News and Internet 4.1%.
BA's e-BI strategy
British Airways uses Business Objects e-business intelligence (e-BI) tools to deliver strategic information worldwide, in support of Customer Relationship Management, yield management, revenue management, flight punctuality and baggage handling.
"We analyse routes, market share data and bookings through travel agents and computerised reservation systems to calibrate routes and schedules," says Peter Blundell, British Airways' knowledge strategy manager. Understanding market dynamics enables marketing campaigns to be launched into areas likely to be fruitful. "You want to segment your market so you don't flood the market with deals that nobody's going to take up," Blundell says.
"Getting the right offer to the right group is very important. Customer intelligence enables us to understand what our booking profiles and customer profiles look like, so that we can make the right offer to the right person. Blundell estimates that BA's BI has brought $100m in cost savings and new revenues.