Forrester: Big data – start small, but scale quickly

At extreme scale, traditional data management and business intelligence become impractical and companies do not get what they demand — more insight to drive greater business performance.

Big data is not only technology, but also people with the appropriate analysis skills

At extreme scale, traditional data management and business intelligence (BI) become impractical, and companies do not get what they demand – more insight to drive greater business performance. The amount of data available is growing faster than our ability to deal with it, and more is coming. 

As firms seek value from digital and analogue sensors, social media, financial systems, e-mails, surveys, and customer call centres, to name a few, new tidal waves swell on the horizon. All of this data promises to open up new frontiers of business opportunity – if we can figure out how to turn it into insight.

Forrester defines "big data" as “techniques and technologies that make handling data at extreme scale affordable”. Despite its simplicity, this definition is not trivial. Forrester asserts that big data is not only technology but also people with the appropriate analysis skills, and makes dealing with extreme scale affordable. By making it cost-effective, more firms can adopt big data to increase their competitive advantage, which leads to more innovation, more supplier solutions, and more momentum, among other benefits.

Big data will get easier

Our research confirmed that big data, in its current immature state, is challenging. Successful firms will meet this challenge through establishing a new level of collaboration between business and IT; developing new processes to deliver solutions; and mastering a rapidly evolving technology landscape.

In Forrester's most recent Global Big Data Online Survey, 70% of respondents said that big data is or will be a collaborative effort between business and IT.

Our interviews revealed big data teams of highly skilled scientists working closely with business executives to discover new uses for more data. To succeed in this brave new world of big data, CIOs must partner with their business peers to identify opportunities and solutions, turn over part or all of big data solution delivery to business leaders, and be ready to support rapid growth.

New processes

The unique characteristic of big data will drive new processes. We found that nearly one-third of firms surveyed are currently using different project management and software delivery lifecycle (SDLC) processes, while nearly as many are not sure what they will do once efforts move beyond prototyping. For example, new big data initiatives will require new processes to manage open source risks as well as new agile processes for solution delivery.

No one has the answers, at least not yet

The name "big data" originated as a tag for a class of technology with roots in high-performance computing, as pioneered by Google in the early 2000s. Today, the big data market is expanding quickly and includes new technologies, such as distributed file and database management tools led by the Apache Hadoop project; big data analytic platforms, also led by Apache; and integration technology for exposing data to other systems and services.

How to manage big data

While big data technology may be quite advanced, everything else surrounding it – best practices, methodologies, organisational structures, etc – is nascent. While no one has the answers, at least not yet, Forrester suggests that firms should:

  • Understand why traditional business intelligence/data warehousing cannot solve a problem. Big data is not necessarily the only answer. If scaling your data servers up and out will solve a volume problem, it may be your best option.
  • Develop a minimal set of big data governance directives upfront. Big data governance is a chicken-and-egg problem – you can't govern or secure what you haven't explored. However, exploring vast datasets without governance and security introduces risk and firms must address this.
  • Remember the three S's – small, simple, and scalable. When launching big data initiatives, avoid getting too complicated too fast, and be prepared to scale once a solution catches on. Keeping big data as small as reasonable, reducing scope to the simplest, most valuable objectives, and being ready to scale are secrets to success.

Big data poses new risks 

Big data can drive big changes in business models as firms discover new ways to capture more value from more data. However, achieving success will also mean that a single version of the truth may not exist for all of your data. Many firms depend on having central databases at the heart of their applications to provide a single, consistent version of the truth – the truth is that this architecture will not serve them as scales reach the extreme. Also, big data will redefine information governance. The scale and nature of big data operations may render some traditional governance techniques inappropriate, exposing firms to risk, and governance must evolve to deal with a big data world.

Brian Hopkins and Boris Evelson are principal analysts at Forrester Research, where they contribute to Forrester's offerings for enterprise architecture and business process professionals.


Read more on Business intelligence software