Dealing with the data explosion

We can’t solve problems by using the same kind of thinking we used when we created them.

We can’t solve problems by using the same kind of thinking we used when we created them.

The complexity of the modern business world is more than matched by the complexity of today’s IT infrastructure. Growth, mergers, acquisitions and long-term IT investment have created a technology landscape characterised by silos of information held on different systems in different departments, subsidiaries and geographies.

More data is captured and stored by businesses now than ever before. A typical business today stores 10 times more data than in 2000 and Gartner estimates that storage requirements will have increased by a factor of 30 by 2012.

In an attempt to curb departmental and regional anarchy, businesses have introduced supplier consolidation policies. But have these policies gone too far? Are they shutting out next-generation technology solutions that are designed for the job in favour of sticking with inefficient, costly, cumbersome infrastructures that won’t easily scale?

Indeed, when Albert Einstein said, “We can’t solve problems by using the same kind of thinking we used when we created them,” he may well have said the same about trying to manage the data explosion with the same technology that causes it.

The challenge for businesses is no longer just capturing data; it is putting that data to effective use. Data analysis is key to understanding the performance of a business, and can be used for numerous tasks including performing profitability analysis, generating reports for legislative compliance, analysing sales and marketing campaigns and predicting buying trends.

A business’s success absolutely depends on insight into the key factors that affect it, in as close to real-time as possible. Analysing performance based on an up-to-the-minute, consolidated view of operations and empowering staff to detect and respond to trends, opportunities and threats are very important to maintaining an edge on competitors and improving business processes.

Relational databases are ideally suited to the task of capturing and storing huge quantities of data and are designed to cope with a massive throughput of transactions. Despite this, data warehouses built from traditional databases create almost insurmountable problems when trying to analyse the massive volumes of data generated by an entire business.

This is because a data warehouse built on relational technology is usually several times the size of the databases it draws its information from, owing to the need not just to import the source data, but to store all the indices and aggregated views required for analysis.

Such a database is designed for transaction throughput, not for the rigours of analytics. There is a vast drop in performance when trying to use relational technology to generate business intelligence, meaning that reports can take hours or even days to produce.

These performance issues can be addressed, to some extent, by a good database administration team who can tune a database to respond to specific queries. However, as the data grows unabated, tuning reaches an upper limit of effectiveness and bottlenecks occur.

A typical response is simply to add more hardware to the system, more storage, more servers and more bandwidth to the data warehouse as a whole. But this does not solve the issue. In the end, all this does is help capture more data, increasing maintenance and storage costs with more discs, servers, IT staff and bigger facilities to house it all.

Using and adapting existing tools does not adequately address today’s need for high performance and mission critical business intelligence, where the onus is on real-time analytics and operational intelligence. Merely adapting existing systems creates further complexities and costs as these systems can be inflexible and fragile with a complex reporting environment.

Many organisations are being held back by supplier consolidation policies that advocate the use of one-size-fits-all relational databases. Not only does this lead to the accepted practice of having to wait hours or days for intelligence reports, it is much more expensive to buy, build and maintain.

It also takes longer to implement and affects the agility of a business – it simply cannot scale to meet the decision making needs of a business operating in an increasingly complex, data-rich world. In terms of supplier consolidation policies, standards across the industry are good and are in place for a reason. However, it is important that these policies are reviewed frequently to ensure they are working effectively and are not inhibiting business.

It is easy to understand why companies are reluctant to introduce new technology and suppliers. But open-mindedness and radical change is just what is required to optimise business performance and maintain business agility. An alternative concept to using relational databases is using vector technologies with column-based storage solutions.

Sybase IQ is a column-based database built on the principles of vector technology. The result is a scalable, nimble solution that provides excellent query performance, data loading and data compression with proven scalability for structured and unstructured data stores. It is a tried and tested solution with over 1,700 customer installations worldwide. Indeed, some have tested and proved their BI strategy and scalability to 30 years ahead with reassuring results.

In the highly competitive financial sector, where data volumes are exploding exponentially, optimising business performance through more agile decision-making is vital for business success. The relational database approach is not well suited for the purpose of analytics for many organisations due to growing volumes of data and user numbers.

Many organisations have researched the options, embracing purpose designed column-based analytical database technology like Sybase IQ to overcome today’s business intelligence challenges and scale to meet future requirements.

In this way, businesses need to start rethinking the way they go about storing their data in order to enhance analytics, improve business processes and give themselves the best possible competitive advantage.



Read more on Database software