Enterprise search tools move from luxury item to business essential as data builds up

The name Google has become synonymous with the concept of searching the internet. Yet its success in raising the profile of search tools - along with others such as Microsoft MSN, Ask Jeeves and Yahoo - has, to some extent, undermined the seriousness with which search technology is treated within the enterprise.

The name Google has become synonymous with the concept of searching the internet. Yet its success in raising the profile of search tools - along with others such as Microsoft MSN, Ask Jeeves and Yahoo - has, to some extent, undermined the seriousness with which search technology is treated within the enterprise.

In the past few years we have seen the emergence of tools designed specifically to search a company's intranet, rather than the internet. It is worth noting the difference between an internet search engine and an enterprise search tool.

Mike Lynch, chief executive of Autonomy, one of the biggest providers of enterprise search tools, said, "When you talk about search, people think of internet search engines, but they are irrelevant in the enterprise, because there is a need for access control and security. Enterprise search tools have to be able to understand who can see which document - and 80% of the data in an enterprise is unstructured. It is also stored in many formats in many repositories. So the technologies should be different."

But the most important issue to consider when choosing enterprise search technology, according to Lynch, is that it correctly meets the search requirements of the organisation.

What is becoming apparent is that the misconception of enterprise search as a "nice to have" is being seriously challenged as the volume of data stored continues to grow at a staggering rate, and the need to identify and retrieve information across the enterprise - from the desktop to the intranet to corporate data repositories - becomes increasingly important in boosting productivity and efficiency, and ultimately competitive advantage.

Effective enterprise search technology is about far more than a white bar and a list of results. It should play a key role in compliance, business intelligence, market intelligence, risk mitigation, and a host of other applications.

For example, increased regulatory pressures on organisations require them to be able to search through and make sense of the information they store.

At the same time, information stored across various data containers, including e-mail servers, desktops, enterprise application databases, content management systems, file systems, intranet sites and external websites, could hold vital information or competitive intelligence that can only be located or even identified using sophisticated search tools.

Enterprise search technology should also help to minimise duplication of effort. It is estimated that knowledge workers spend more than twice as much time recreating content than they spend creating new content. If they are able to focus on creating new content, the benefits and productivity improvements could be substantial.

Mike Davis, senior research analyst at Butler Group, said, "IT suppliers often say that between 20% and 80% of a knowledge worker's time is spent looking for information.

"So by identifying important information faster and more accurately, an effective enterprise search tool should allow employees to become more efficient and productive, increase job satisfaction and identify anomalies in the quality of data stored.

"In theory, enterprise search technology should therefore provide competitive advantage."

A Butler Group report on document and records management goes even further. It said, "The greatest challenge for organisations, and the one that Butler Group poses as the highest risk of failure to achieve, is the ability to locate information."

According to Davis, desktop search is the starting point for any enterprise search strategy, as it is estimated that 80% of corporate information exists on employee PCs.

"The reality for all of us is that a great deal of time is spent searching for content that is created and stored locally, and unless we have implemented a decent taxonomy on the C drive, and adhered to it, each of our PCs is just an unstructured data dustbin," he said.

Indeed, Microsoft recently reiterated its focus on enterprise search when it announced a desktop search addition to MSN Search, which allows enterprise users to scan e-mail, desktops or shared network drives on machines running Windows 2000 or XP and Office.

In doing so it joined the ranks of Google, Verity, Fast and Yahoo, which already offer desktop search software to enterprises.

Angela Ashenden, senior analyst at research firm Ovum, said, "Desktop search is relatively new, with few people using it in the enterprise yet, but that will change.

"Enterprise search, meanwhile, provides the ability to search information across an entire organisation, primarily internal content such as the intranet, but also external content, such as subscriptions on the internet."

According to Ovum, sales of  enterprise search and discovery technology grew by a 37% in 2004 to £290m, and it is expected to expand with a compound annual growth rate of 16.4% over the next five years to reach £640m in 2009.

Most enterprise search suppliers offer some form of desktop search tool as a matter of course, considering it merely the starting point of a true enterprise search tool.

In an attempt to distance themselves from the "Google internet search" perception, enterprise search providers are focusing their products along the lines of information access, or knowledge discovery and retrieval.

The "search" label, in fact, underplays some of the more interesting and sophisticated capabilities offered by enterprise search tools. The search engine, as used on the internet, uses mathematical algorithms to rank results and is the most basic form of technology.

But enterprise search and discovery products offer increasingly sophisticated functions, such as access control and security, auto-categorisation, taxonomy navigation, concept-based retrieval, search analytics, and visualisation and personalisation. In future, natural language search and semantics will replace the keyword search.

Suppliers' claims that they can measure the return on investment from search tools are difficult to verify. However, Whit Andrews, research vice-president at Gartner, has some suggestions.

"Enterprise search tools decrease the time spent finding and recreating information and offer a substantial payback in terms of improvement in the way the enterprise does business. They should provide insight, collaboration and efficiency," he said.

"The most important thing a CIO needs to understand is that we are now dealing with the 'Google generation' - everybody working for you now used Yahoo throughout school and Google in college, and cannot imagine a world that is not searchable. Where it was a curiosity 10 years ago, it is now critical."

So how does one select a suitable search tool? Ashenden believes users should consider both high- and low-end products.

"Some parts of the organisation have complex requirements, and need a high-end product, while others need a Google-type solution," she said.

She suggested that a Google-like product is a good starting point, allowing an enterprise to understand what it really needs from a search tool.

"It is then in a much better position to know what it wants, in terms of complex search requirements, from suppliers such as Verity, Autonomy and Fast," said Ashenden.

Enterprise search suppliers and their products

The five leading enterprise search suppliers - Verity, Autonomy, Fast Search & Transfer (Fast), Google and Microsoft - accounted for almost half of all sales in 2004.

Since then Autonomy has acquired its main rival Verity for £290m. Its Intelligent Data Operating Layer (Idol) is a high-end, neural-network-based platform that connects more than 300 formats and data and information repositories across the enterprise.

Google's products include Search Appliance (for the enterprise), Google Mini (for SMEs), Desktop Search for Enterprise (desktop) and, of course, its internet search engine.

As well as MSN Search and MSN Desktop Search for the enterprise, Microsoft offers search through Sharepoint Portal Server.

Fast's Enterprise Search Platform (ESP) is designed to help firms build bespoke applications with customised user interfaces for specific functions or user groups.

Omnifind is IBM's search tool. It offers enterprise search capabilities across corporate intranets, public websites, relational databases and content management systems, but it is not tuned to specific industries.

Endeca focuses on guided navigation, which exposes different facets of the same information.

Convera's Excalibur provides a secure index of the web, and the company continues to sell Retrievalware, its information access suite of technologies.

Inxight recently acquired the enterprise search assets from Intelliseek, which will complement its own natural language, visualisation and analysis product.

Open Text Livelink Discovery Server offers a scalable platform for conventional enterprise search.

Other enterprise search suppliers include Dieselpoint, EasyAsk, Entopia, Hummingbird, InQuira, Isys Search Software, Kaidara Software, Knova Software, Mercado, Mondosoft, Recommind, Thunderstone, Vivisimo and Zylab.

Case study: BAE Systems reduces retrieval time

Defence and aeronautics manufacturer BAE Systems deployed Autonomy's Intelligent Data Operating Layer (Idol) platform, after discovering that more than 80% of networked employees were wasting an average of 30 minutes a day retrieving information and 60% were spending an hour or more duplicating the work of others.

The first two departments to implement Idol were corporate communications and the virtual university (BAE Systems' learning, research and best practice division). They use the platform to aggregate structured and unstructured information from the company intranet and 10,000 news feeds per day.

This content is then automatically categorised without the need for manual intervention, allowing employees to navigate very easily through the site to access pertinent content. It also automatically alerts employees to documents in the system that relate to their areas of expertise and interest.

As a result, BAE Systems has been able to personalise the delivery of business-critical information to staff across the company, and has reduced the time spent retrieving information by more than 90%.

The company said it achieved return on investment seven months after the initial implementation.

Richard West, organisation and e-learning manager at BAE Systems, said, "We discovered engineers working, in different parts of the country, on precisely the same problem - a wing construction issue - but in very different areas, a military aircraft and an Airbus.

"One group took the step to establish best practice, which was transferred to a plant in another geographical location with multimillion-pound savings. Tools like Idol ensure this sharing of best practice is more likely, particularly in a global organisation."

Read more on Database software