There are two main approaches to master data management: operational MDM and analytical MDM. The former focuses...
on ensuring that data that should be the same across different operational systems actually is the same. Analytical MDM, on the other hand, is generally associated with data warehousing and has been adopted by organisations looking to improve the speed and quality of their business intelligence (BI) reporting processes.
The close relationship between MDM and data warehousing isn’t surprising, since the ‘dimensions’ of a data warehouse (eg: customers or hierarchies of products) are essentially master data. But these two important areas tend to be treated as being entirely separate from one another.
Earlier this year, The Information Difference conducted an online survey on the link between MDM and data warehousing. Our aim was to better understand the scale, scope and success rates of combined MDM and data warehousing initiatives.
A total of 208 respondents from a wide spectrum of industries completed the survey: most were located in North America (57%) and Europe (27%). Drawing upon the results, which were released in April, we offer six tips on managing master data for enterprises that are working on MDM and data warehousing projects or plan to embark on them.
Implement data warehousing and MDM systems in parallel. Almost half (46%) of the surveyed organisations had at least one data warehouse and an MDM implementation that was live, in development or being planned. That compares with 30% that had data warehouses but no MDM initiatives and 7% that were working on MDM only. This shows that MDM has become more established in combination with data warehousing, and we recommend that enterprises would do well to implement MDM systems alongside their data warehouse(s). Encouragingly, 69% of the respondents active in both data warehousing and MDM said they had data governance programmes in place. In our experience, implementing an MDM system and MDM software without a parallel data governance initiative is a recipe for disaster.
Source BI-related master data (ie: dimension data) from your MDM system. Nearly two-thirds (61%) of the participating organisations with data warehouses and MDM initiatives are feeding master data to their BI applications from points within their MDM systems instead of maintaining it within the applications themselves. There is clearly also a strongly held view that BI master data ideally should be stored in and sourced directly from an MDM hub: 91% of the respondents in this group agreed, and 36% were already doing so. Even more (93%) said it would be best to first feed the master data from an MDM hub into a data warehouse, and then on to the BI applications. It’s our view that these approaches are the most effective route to ensuring that you have consistent business intelligence data.
Implementing an MDM system and MDM software without a parallel data governance initiative is a recipe for disaster.
Avoid using ETL tools to manage data quality. Ensuring that high-quality data is loaded into a data warehouse is a prerequisite for reliable BI reporting. However, we found that 43% of the respondents with data warehouses and MDM programmes still rely on extract, transform and load (ETL) software to manage the data quality process. Many ETL tools offer limited visibility of data transfer rules and provide little data quality functionality. Organisations should ensure that if they opt for this route, either data quality functionality is built in or an additional data quality tool is ‘bolted on’. Even more disturbingly, 24% said they assumed that data quality issues were addressed within source systems before information was transferred to a data warehouse. In our experience, that is seldom the case!
Look beyond traditional master data domains. Although the common master data domains of 'Customer' and 'Product' were unsurprisingly cited most frequently, the mean number of domains reported by respondents with data warehouses and MDM projects was four and the median was three. This shows that organisations often do have a need for managing master data that involves more than just information relating to their customers and products – for example, location, supplier and financial master data. Be aware, though, that some MDM vendors still fall short on supporting multiple domains.
Plan for and budget adequate resources. The median number of full-time equivalent (FTE) positions reported as being required for ongoing maintenance of the MDM implementations was five, with a corresponding figure of 10 FTEs for the data warehouses. Those figures provide a useful guide for planning and budgeting purposes to enterprises that are embarking on MDM and data warehousing implementations.
Maintain business rules in the MDM system. Business rules are critical for any organisation, and not maintaining them properly is a frequent source of failure in BI programs. According to the survey results, most organisations are still maintaining at least some of their business rules in ETL scripts, where the rules tend to be inaccessible. Maintaining them in a data warehouse was the second most frequently selected option, while using an MDM hub was well down the list. We believe that enterprises should look more to maintaining business rules within an MDM system to ensure that they are visible and can easily be updated.
Dr. David Waddington is co-founder and senior vice-president of The Information Difference Ltd., a London-based analyst firm that focuses on MDM. Waddington advises corporations as well as IT vendors on data management strategy and architecture. He worked previously as a vice-president and research director at consulting firm Ventana Research and as chief IT systems architect for Unilever’s two food products groups in Europe.
(Editor’s note: TechTarget was a media sponsor of the survey referenced in this article and sent emails on behalf of The Information Difference to invite potential respondents to participate in the survey.)