It has been a busy year in the business intelligence market. Hyperion has been acquired by Oracle, SAP is buying Business Objects, and IBM announced plans to buy Cognos. The strategies of these major software companies reflect a growing focus among IT managers.
The need to rationalise and standardise multitudinous business intelligence tools is rising up the IT agenda as organisations struggle to obtain a single version of the truth from their information and contain costs.
In many firms, data has become fragmented and dispersed because of the ad hoc way in which they have historically purchased business intelligence tools. For the past decade, individual departments have chosen products from either independent suppliers or database firms to analyse data from specific local applications.
IT teams, meanwhile, have often opted for another range of analytical products to work against large horizontal systems, such as finance or logistics. The waters have been further muddied by large application providers such as SAP starting to provide their own, often embedded, business intelligence systems.
Dale Vile, research director at analyst company Freeform Dynamics, says, "The issue is that, traditionally, organisations have not budgeted for business intelligence as a discrete entity, and so expenditure tended to be tacked onto other projects. The majority of current problems are arising from the subsequent fragmentation. These issues are greatest in large companies, but they are still there in smaller ones too."
This means that many users are simply unable to access all of the relevant information they require, or they are forced to cobble together bits of data from different sources and applications to come up with a sound answer to a query. They also find that information is frequently inaccurate, not least because of duplication and because there are often no standard definitions of basic terms, such as "revenues".
Andreas Bitterer, vice-president of research at analyst firm Gartner, says, "What I would call revenues is different to what you would call revenues, and we end up with two different figures to answer the same question because inconsistency is built into the system. The issue then, of course, is which one do you trust if you are trying to make decisions?"
This situation has huge ramifications in terms of cost. Not only do staff waste time searching for relevant information, which adversely affects their productivity, the IT department wastes money by not exploiting possible economies of scale in relation to software licensing deals. It also ends up with a management, support and training headache.
According to a survey of 1,114 companies in the UK, US and mainland Europe by Freeform Dynamics in July, this scenario is unlikely to change significantly any time soon.
The report, entitled "The Business Intelligence Inflexion Point: Information is a Right, not a Privilege", indicated that 33% of companies with more than 5,000 staff have a formal business intelligence strategy, and the smaller the organisation, the less likely they are to have a business intelligence plan.
Although 50% of companies questioned expected their investment in such technology to increase over the next few years, only 13% anticipated ring-fencing money in the budget to this end. The rest were incorporating it into other budget lines, or justifying expenditure on a case-by-case basis, rather than adopting a strategic approach.
"The course that they are on, largely speaking, is at best not going to solve the problem and, at worst, is going to aggravate it. The more you spend on fragmented systems, the more serious the situation gets, but it can be quite difficult to get business people to take business intelligence seriously. They scream for information but often are not willing to put their money where their mouth is," Vile says.
This is because such projects are notoriously difficult to make a business case for as they do not necessarily result in direct savings. The real benefit comes from providing managers with better information and enabling them to make more effective decisions, but it is tricky to put a figure on that, and cost justification can be based largely on faith.
Case study: North Wales Police
One organisation that has benefited from introducing a well-planned, properly funded business intelligence strategy is South Wales Police. It found that, because it had multiple operational policing systems, information was siloed in different places, with each underlying database having its own set of business intelligence tools. This situation made it difficult to extract and analyse data in a consistent way.
The police force considered it crucial to address the issue as not only is intelligence information becoming more sophisticated, the organisation will need to comply with the Home Office's forthcoming Impact Nominal Index initiative. Due to go live in 2010, the initiative is intended to help police forces share information more effectively.
As a result, in early 2005, South Wales Police started a project with the help of IT services provider Atos Origin to consolidate its various applications down to two systems. "At the same time, we looked at the impact on business intelligence, which we knew would be significant," says Chief Superintendent Richard Lewis, head of performance management at South Wales Police.
A year later, this led the force to scrap its fragmented toolset and start implementing a datawarehouse and reporting tools from SAS. "Now we have all of our data in one place, in one format, and with one data model, which enables us to extract new and historical information more effectively," says Lewis. "It has given us far better intelligence and awareness of our interactions with customers, whether they are suspects, victims of crime, witnesses, or family members that may have contacted us."
The trickiest part of the project was cleaning up existing data and improving data quality processes, which was "a massive job", particularly given the sensitivity of the information involved. "We introduced business rules to bring information together, but some of it had to be judged manually. If you merge data on two people with the same name, you could end up arresting the wrong person, so we could not take any risks," says Lewis.
Another challenge was to win the support of officers who would be using the system, not all of whom were technically proficient. This was undertaken with the help of a business change manager who facilitated seminars for senior and middle managers. They, in turn, provided input on the optimum ways to undertake process change. Police "communications champions" were also trained to cascade information down to colleagues and help them with any difficulties.
After an optimisation phase, the next step for South Wales Police will be to introduce data and text mining tools to enable the organisation to spot trends more effectively. A further aim is to introduce business data, such as human resources information, to the datawarehouse.
"Our strategic aim is to put all of our organisational data into the system, but this is a big project, so we will take a breather and ensure that what we have done so far, we have got right," says Lewis.
Bitterer says initiatives of this type are usually very complex and typically take a few years. Prior to organisations starting a business intelligence project, they need to undertake an audit of their assets so that they know what is in place, before tackling the thorny issue of standardising terminology and metrics and introducing a clear governance model. Migrating to a new platform is also often a long-term, gradual process to minimise business disruption.
Case study: Virgin Media
One organisation that is at the start of its journey here is Virgin Media. Because its former NTL and Telewest acquisitions have themselves made numerous cable franchise purchases over the years, the combined business has disparate datawarehouses, data marts and related tools that were often put in place to address specific project requirements.
The challenge now is that, although these silos of information are able to share common reference data because of the many points of integration between them, it is not always easy to obtain a single view of customers.
As a result, Virgin Media is about to issue a request for proposal for assistance from organisations such as consultants, systems integrators and software suppliers to help it tackle what it envisages will be a three-year business intelligence project. This will include creating a datawarehouse based on Netezza's appliance and rationalising its query and reporting tools.
Paul Froggatt, information management manager at Virgin Media, says, "Our vision and strategy is to bring our information assets into a single consolidated environment that will deliver intelligence to the enterprise as a whole. The goal is to produce an holistic view of individuals as customers and identify all the relationships they have with us and our product sets."
This includes understanding what products customers use, how they use them, and being able to see any inbound or outbound contact that has been made with them in order to improve customer service, reduce churn and enhance cross-selling and up-selling opportunities.
One of the first steps will be to establish a governance team. This will be created in the next few months and be tasked with agreeing on a single version of data definitions and metrics.
The governance team will also prioritise what should be done when and govern the quality of what is finally delivered. When the project is complete, the team will continue to own and manage the data definitions, control who can use the data, and be in charge of any future enhancements to the system.
Another important task is to engage with key stakeholders across the business to identify use cases and establish which information they need access to. This activity will also help the IT department gain more insight into the scope and size of the programme as it gains a clearer understanding of what needs to be delivered.
"A business like ours changes very fast, so you cannot afford to go away, build something in isolation and then throw it over the fence into a live support environment. You have to be realistic about what you can achieve in what timescales, and that means ensuring you fully understand the business requirements and prioritise accordingly," Froggatt says.