We can’t solve problems by using the same kind of
thinking we used when we created them.
The complexity of the modern business world is more than matched
by the complexity of today’s IT infrastructure. Growth, mergers,
acquisitions and long-term IT investment have created a technology
landscape characterised by silos of information held on different
systems in different departments, subsidiaries and geographies.
More data is captured and stored by businesses now than ever
before. A typical business today stores 10 times more data than in
2000 and Gartner estimates that storage requirements will have
increased by a factor of 30 by 2012.
In an attempt to curb departmental and regional anarchy,
businesses have introduced supplier consolidation policies. But
have these policies gone too far? Are they shutting out
next-generation technology solutions that are designed for the job
in favour of sticking with inefficient, costly, cumbersome
infrastructures that won’t easily scale?
Indeed, when Albert Einstein said, “We can’t solve problems by
using the same kind of thinking we used when we created them,” he
may well have said the same about trying to manage the data
explosion with the same technology that causes it.
The challenge for businesses is no longer just capturing data;
it is putting that data to effective use. Data analysis is key to
understanding the performance of a business, and can be used for
numerous tasks including performing profitability analysis,
generating reports for legislative compliance, analysing sales and
marketing campaigns and predicting buying trends.
A business’s success absolutely depends on insight into the key
factors that affect it, in as close to real-time as possible.
Analysing performance based on an up-to-the-minute, consolidated
view of operations and empowering staff to detect and respond to
trends, opportunities and threats are very important to maintaining
an edge on competitors and improving business processes.
Relational databases are ideally suited to the task of capturing
and storing huge quantities of data and are designed to cope with a
massive throughput of transactions. Despite this, data warehouses
built from traditional databases create almost insurmountable
problems when trying to analyse the massive volumes of data
generated by an entire business.
This is because a data warehouse built on relational technology
is usually several times the size of the databases it draws its
information from, owing to the need not just to import the source
data, but to store all the indices and aggregated views required
for analysis.
Such a database is designed for transaction throughput, not for
the rigours of analytics. There is a vast drop in performance when
trying to use relational technology to generate business
intelligence, meaning that reports can take hours or even days to
produce.
These performance issues can be addressed, to some extent, by a
good database administration team who can tune a database to
respond to specific queries. However, as the data grows unabated,
tuning reaches an upper limit of effectiveness and bottlenecks
occur.
A typical response is simply to add more hardware to the system,
more storage, more servers and more bandwidth to the data warehouse
as a whole. But this does not solve the issue. In the end, all this
does is help capture more data, increasing maintenance and storage
costs with more discs, servers, IT staff and bigger facilities to
house it all.
Using and adapting existing tools does not adequately address
today’s need for high performance and mission critical business
intelligence, where the onus is on real-time analytics and
operational intelligence. Merely adapting existing systems creates
further complexities and costs as these systems can be inflexible
and fragile with a complex reporting environment.
Many organisations are being held back by supplier consolidation
policies that advocate the use of one-size-fits-all relational
databases. Not only does this lead to the accepted practice of
having to wait hours or days for intelligence reports, it is much
more expensive to buy, build and maintain.
It also takes longer to implement and affects the agility of a
business – it simply cannot scale to meet the decision making needs
of a business operating in an increasingly complex, data-rich
world. In terms of supplier consolidation policies, standards
across the industry are good and are in place for a reason.
However, it is important that these policies are reviewed
frequently to ensure they are working effectively and are not
inhibiting business.
It is easy to understand why companies are reluctant to
introduce new technology and suppliers. But open-mindedness and
radical change is just what is required to optimise business
performance and maintain business agility. An alternative concept
to using relational databases is using vector technologies with
column-based storage solutions.
Sybase IQ is a column-based database built on the principles of
vector technology. The result is a scalable, nimble solution that
provides excellent query performance, data loading and data
compression with proven scalability for structured and unstructured
data stores. It is a tried and tested solution with over 1,700
customer installations worldwide. Indeed, some have tested and
proved their BI strategy and scalability to 30 years ahead with
reassuring results.
In the highly competitive financial sector, where data volumes
are exploding exponentially, optimising business performance
through more agile decision-making is vital for business success.
The relational database approach is not well suited for the purpose
of analytics for many organisations due to growing volumes of data
and user numbers.
Many organisations have researched the options, embracing
purpose designed column-based analytical database technology like
Sybase IQ to overcome today’s business intelligence challenges and
scale to meet future requirements.
In this way, businesses need to start rethinking the way they go
about storing their data in order to enhance analytics, improve
business processes and give themselves the best possible
competitive advantage.