Bridging the information gap

When a customer calls insurance company Martinez and Partners, staff can see immediately whether the client has filed a claim, what correspondence has been sent and received, and whether any outstanding payments are due.

It sounds simple, but it has taken two years of hard work for the firm to get to this point, says Jonathan Evans, managing director of Martinez and Partners. "A lot of our work is done on paper, so we have huge amounts of information contained in letters, claim forms, reports, and so on. But then we also have information contained in databases on prices, customer details, accounts and so on. Pulling the two together used to be extremely time-consuming," he says.

Bridging the gap between structured and unstructured data is a goal for many businesses, says Douglas Coombs, information-on-demand advocate with IBM Software Group. "Typically, information is held in silos and when the time comes to make a decision, you do not know where all the information is. You do not have a complete picture. Bridging the gap means you are making better decisions and you can serve the customer better."

The growing focus on compliance, audit trails and data security is fuelling a boom in information management products and services, notes Mike Davis, a senior analyst with Ovum. "All the big banks are doing it because they suddenly want to know where all the collateral debts are. Shareholders want to know why their brokers did not see this coming - where are the e-mails that prove they are innocent? It is a massive area of focus for 2009, and I think it is something we will hear a lot about next year."

Integrating structured information (such as names, addresses and figures in databases or spreadsheets) with unstructured information (such as text in CRM applications and letters stored in records management applications) is no easy task, and organisations have traditionally had to build integration manually from one data source to another.

But the benefits of greater integration and holistic information management mean more and more IT departments are embarking on information management projects. "People are beginning to realise they are not accessing huge parts of the corporate memory because it is held in unstructured or semi-structured data," he says. "It is all very well knowing that the customer cancelled a contract, but if you don't have the background to that decision, how can you possibly know the best way to react?" asks Davis.

Consistent view

Having a consistent view of all types of data can also be important for compliance. "If you delete information in one system, do you know all instances of that data are deleted? Or have you not been able to find and delete the e-mails or letters relating to that account? It is massively important as we focus more on risk and risk management. You absolutely need information management and a single information management policy to be sure you are acting consistently," says Coombs.

Information management is an umbrella term for bringing together structured, unstructured and semi-structured data. It is a logical next step from using knowledge management or business intelligence tools, says Davis. But organisations cannot realise the full benefit of such tools because they are accessing only part of the corporate memory.

Traditionally, information management has been difficult for enterprises to achieve because suppliers tended to specialise in either structured or unstructured data, says Coombs. "Very few people have a holistic approach to utilise, connect and use all the information you have," he says.

There are several approaches to information management, depending on your current IT environment. Large enterprises may be able to use, for example, MySAP to put a web front-end on information pulled from ERP and CRM systems. Some companies have bought add-ons to content management applications from suppliers such as Xenos, which can pull information from an ERP system into the enterprise content management system for a more complete view of the data.

A second approach is to use an enterprise search tool that can look for information across structured and unstructured data sources. "Companies such as Autonomy, Decca and Microsoft are now pushing towards bringing structured data into search, and eventually you will have something like Google is able to do," says Davis. For example, searching a company's name on Google will often turn up stock charts and presentations alongside web pages. "It is a very fast approach to information management, and it is almost recreating the portal approach within search," says Davis. "However, the limitation of search is you still need a human being to look at the results and see what is important to a specific activity or process."

The third approach to information management is the "suck in, spit out" tool - software that will pull in information from any data source and translate it into a common data format, such as XML. IBM Websphere and products from vendors such as Informatica act as a middle layer between the data repository and applications, using a combination of business intelligence and data mining.

"The advantage of using this kind of tool is that you can turn information from multiple sources into any type of format you need," says Mark Dunleavy, financial services manager with Informatica. "XML is the most common, but it can be anything, including your own proprietary standard. What we are basically doing is taking all your structured and unstructured data and turning it all into a common structured format that can be consumed in the organisation."

IBM is now working towards "information on demand", says Coombs - providing tools that will enable IT departments to label metadata irrespective of its source and then apply policies to data that will manage information through its lifecycle, based on business processes rather than data type or location. "The idea is that you will be able to define metadata management across both types of data, then apply a single information policy to manage the lifecycle," he adds.

IBM says it will be able to deliver a complete information management system incorporating data management and optimisation (using DB2, for example), enterprise content management, Infosphere information integration, data control and business intelligence and performance management, driven by Cognos.

How about alternatives? At the moment, Microsoft is leading the enterprise pack in information management, says Ovum's Davis. "Microsoft is significantly further ahead than Oracle in bringing together structured, unstructured and semi-structured information, and is working towards making various sorts of data more inter-connected. However, it is still some way off having a real, working solution comparable to Autonomy or IBM."

Instead, says Davis, much of the integration of data held in systems such as Microsoft SQL Server and Oracle will involve specialist tools from third-party suppliers such as Clarabridge. "Over time, I expect more of these suppliers to be bought or to become partners of the enterprise suppliers, in the same way that SAP bought Business Objects, which had already bought Insight."

Build on basics

Deciding which approach is right for you depends largely on how quickly you need a solution and the level of customisation required. As a rule of thumb, enterprise search tools can be up and running very quickly but will not offer customisation - which might be important in a specialist industry. IBM will take longer to deploy, but would offer much greater depth of integration and personalisation, says Davis. "If you have got lots of IBM repositories, then it makes sense that IBM will be able to help you get that information out. In a more heterogenous environment, the choice might be more complicated."

Whichever approach you take to information management, it is vital to have the basics in place. Before any information management project, the experts advise thorough data cleansing and auditing of your data collection procedures to ensure you are getting the right, current information.

"You cannot abdicate responsibility for understanding the basics of your data and the information you are dealing with, or you will fall for the perception of 'facts' that are presented to you," says Clive Longbottom, service director at analyst firm Quocirca. For example, a child who only ever uses a calculator will trust the calculator to such an extent that if they enter 2 + 2 and the calculator tells them the answer is 5, children will believe the machine, says Longbottom. "You need to know you can trust the information, and that someone has a sense of when things are right or wrong."

Although good information management will give people access to better information, that does not mean the decision-making process should be completely automated or handed over to people with little responsibility. "Anything that is complex must enable the user to drill down to the underlying levels of data to check what has been reported against," says Longbottom. He also advises IT managers to ensure information is linked with clear audit trails so users can see what action is taken and for what reason, and that when underlying data changes, decisions are revisited where necessary.

Finally, IT managers must also be aware of the potential danger of project creep, says Davis. When bridging data, do not make the mistake of imagining that "go live" will happen only when all the bridges have been built - because you will always be able to find another bridge that needs building. "If you only have a bridge between the CRM and content management systems, then you can still go live," he says. "Look for incremental changes and provide value from day one. I don't think this is something that can wait."

The insurance agents at Martinez and Partners are already using the new information management system, which was developed on top of a highly customised CRM system. "We are now able to see structured data from the financial management program together with unstructured data from the CRM system through a web front-end," says Evans. "It saves a lot of work and means key processes can be largely automated, but most of all it means when a customer calls and wants a risk management report, we can easily find the information and pull it together, rather than manually going through old reports and creating it from scratch."