How to eliminate duplicated data

Master data management packages promise to ease the problem of having inconsistent and duplicated data stored across multiple enterprise systems

For many companies, having inconsistent data is not just inconvenient, it is a major impediment to business agility and competitiveness.

Yet many businesses suffer from this affliction, in which information about the same customer or product, for example, appears in multiple systems and in multiple formats across the company and simply does not tally from system to system. This undermines reporting initiatives and can seriously impede managers' efforts to make sound strategic decisions.

This situation certainly cannot be blamed on a lack of effort or technology investment, however. On the contrary, organisations spent millions of pounds throughout the 1990s on enterprise resource planning (ERP) suites that promised to provide a central, consistent set of enterprise data.

The problem with this was that the vast majority of businesses needed to implement additional software products after the ERP system was introduced, and each new system would have its own database, data format and, frequently, its own version of data that appeared elsewhere.

"To put the problem in context, a change of address in a company's customer relationship management system, for example, may not find its way into other databases that also use that information, such as the logistics system - with obvious repercussions when it comes to the delivery of ordered goods," says Nick Millman, a senior manager at consulting firm Accenture.

Enter master data management (MDM). This is a practice and technology that aims to help businesses improve data consistency and accuracy across all systems and divisions, and also identifies and manages the complex web of relationships that often exists between disparate data elements.

BT, for example, uses MDM to ensure that customer names and addresses are consistent right across the company using its name and address database.

This acts as a central repository, which is used to populate all other applications, says Dave Evans, senior data management consultant at BT.

"In that way, we eliminate easy mistakes that cost us money - such as an engineer turning up at the wrong address. Mistakes like that really demonstrate the importance of data quality," he says.

Xerox Europe, meanwhile, is using MDM to standardise data held by each of its 16 in-country operations before loading the data into a single instance of SAP.

"On average, we have found that in the best countries about 10% of the data is duplicated. In the worst countries it is about 30%," says Andy Bloomfield, SAP deployment manager at Xerox Europe.

"We need to consolidate that all into a single, reliable source of information - but that is quite a big operation, so we need all the automation that technology can provide," he says.

But MDM goes way beyond a one-time clean-up operation for enterprise data. In essence, it is about centrally controlling definitions and formats for a wide range of data entities in order to achieve harmony across multiple systems on an ongoing basis.

As a result, true MDM products offer a technical integration architecture for centrally controlling and distributing definitions of customers, brands, suppliers, financial indicators and so on in near real-time.

The MDM database checks and revises data constantly, says Rob Toguri, director of business information management at systems integration specialist Capgemini.

"By defining changes to data in one place and then rippling that through to relevant databases, MDM ensures that key data relating to customers, suppliers, products, employees and so on is current, consistent and accurate across all systems in the enterprise," Toguri says.

For many companies, that is an appealing prospect, and MDM is a high priority for many clients, says John Radcliffe, an analyst at Gartner. "But there is a lot of confusion," he says - not least because of the range and diversity of suppliers eager to grab a slice of the market.

That confusion does not seem to be stifling sales, however. According to Rob Karel, an analyst at Forrester Research, MDM licence and services revenue reached £550m in 2006, with 68% spent on services and the remainder spent on software licences.

Karel expects the MDM market to reach £3.4bn by 2010, with year-over-year growth averaging between 55% and 59%, and the services-to-software ratio slightly decreasing but holding steady at about 2:1.

Already, big industry names have staked out a substantial claim in this market. For example, Radcliffe estimates that IBM, SAP and Oracle already account for more than 50% of licence revenue in this sector. Microsoft is hard on the heels of these firms, he says, following its July 2007 acquisition of MDM specialist Stratature.

Broadly speaking, the MDM software market can be divided into three camps: customer data integration, product information management and other MDM-focused niche products.

Customer data integration is the most mature segment of the MDM product, according to Karel, consisting of transactional customer hubs that collect customer data from disparate systems. These hubs then standardise and consolidate data to produce a single customer view that can be synchronised, delivered and consumed throughout an enterprise.

"The majority of customer data integration applications provide single-supplier support to enable a majority of the capabilities required with the MDM ecosystem, but they only have out-of-the-box business rules and data models to support customer master data. Support for other data domains, like product, asset or contract, is not usually an option," Karel says.

IT suppliers offering customer data integration products include IBM, Initiate Systems, Oracle, Purisma, SAP, Siperian and VisionWare.

Financial services company Irish Life & Permanent is currently implementing IBM's Websphere Customer Center product in order to establish a single record of its customer base, formed from the 1999 merger of insurance company Irish Life and bank Irish Permanent.

Product information management products, meanwhile, provide many of the workflows, business rules, data models and integration capabilities required to support either an organisation's internal product master set or its external supply chain data.

"Product information management systems specialise in cataloguing, standardising and synchronising product data from disparate sources, but they are also limited to product data, and rarely support other critical data domains," Karel says.

Companies offering product information management products include GXS, i2 Technologies, Oracle, SAP and Tibco.

US restaurant chain Brinker uses Tibco's MDM product to standardise the data it holds about menu items so that it can calculate the cost of food supplies across a network of 1,600 food outlets.

The third category of MDM products includes niche products that "offer a solution to one of more components of the MDM core, but do not have the same breadth as a typical customer data integration or product information management supplier," Karel says.

Since a core component of creating master data is the ability to first perform data-quality profiling and then apply standardisation, matching, merging and enrichment logic, Forrester includes data quality tools from suppliers such as Business Objects, DataFlux, Informatica and Trillium Software.

Likewise, other suppliers such as Kalido and Hyperion Solutions - best known for producing datawarehousing and business intelligence - also offer data workflow, model management and hierarchy management tools that complement customer data integration, product information management and data quality products.

Whatever product is chosen, however, attempts to impose common business definitions require substantial collaboration between IT and the business people who use and rely on the data.

That is a tall order, says Radcliffe. "It requires a great deal of understanding, effort and discipline to achieve. The good news is that for companies that succeed at MDM, there are substantial rewards to be gained."

Study: Irish Life & Permanent

Financial services company Irish Life & Permanent is currently implementing IBM's Websphere Customer Center product in order to establish a single record of its customer base.

"While the two businesses - life and bank - continue to operate separately and have their own brands and offices, we nevertheless want to cross-sell to our combined customer base, and there is a lot of crossover between businesses and products," says Noel Garry, executive manager of IT strategy and planning at Irish Life & Permanent.

"If we have a 'Noel Garry' on our life database and a 'Noel Garry' on our bank database, it is vital we know that we are dealing with the same person.

"At first, we thought we might simply load data from the bank file into the life file, but after a bit of exploration, we got talking to Gartner about MDM and we realised that it would provide us with a more long-term answer to the challenge of keeping our data consistent," Garry says.

A massive data clean-up is now under way, using automated technology that brings together and consolidates matching records. When agreed thresholds for matches are not reached, the system marks the records up as "suspect" and flags them for human intervention.

Using that approach, the group has already identified 117,000 exact duplicates within the Irish Life database alone.

Now that those duplicates have been eliminated and the rest of the data has been cleaned up, the group can start to load that database on to the Websphere Customer Center, Garry says.

The data held in the bank database, meanwhile, will also be cleaned and loaded into the MDM product towards the end of 2007.

"Once that is complete, every time a new customer is entered into any of our systems, that system will reference Websphere Customer Center and use the hub to standardise the data captured.

"And if a customer calls into the bank and tells us their mobile phone number has changed, the new number is automatically distributed by the hub to all systems that hold information on that customer," Garry says.

MDM, he says, is just a starting point for Irish Life & Permanent. "It will be a springboard for all types of new initiatives, which is why we see this very much as an enablement project for all types of improvements to the services we can offer our customers," he says.

Read more on Privacy and data protection