- Big data processing technologies mature
- Assisting the business in opportunity recognition
- Maintaining a holistic approach to information architecture
- Evaluating how applications will integrate knowledge via BI, PA and SOA
- Updating your infrastructure technology road map and watchlist
It's no secret that the flood of information from the internet, sensors, and image capture holds untapped opportunities for companies looking to improve the bottom line. But when confronted by the reality that much of this data exists in large volumes or streams at extremely high data rates - 100 to 1,000 times larger than what IT systems typically handle today - the prospect of extracting useful business value becomes less fathomable. As demands for better value from more information stretch - and will eventually exceed - the capabilities of traditional computing platforms, big data processing promises to overcome current limitations by using massively parallel, distributed architecture to capture, store and iteratively sift through raw data on a large scale.
This diverse set of technologies will also provide new tools and techniques that shift the way companies use information to compete. Every detail of complex machinery can be captured and evaluated as it runs; for example, a Boeing jet engine produces 20 terabytes of data per hour, which engineers can monitor and examine in near real time, or mine later for engineering improvements. In the arena of consumer sentiment, big data processing applications are emerging to mine transitory market intelligence from the 24 billion tweets that populate Twitter in any given month. Similarly, information about every product in every store can be tracked globally using RFID tags, producing a flood of updates. Big data processing technologies can mine information about product and customer movements, allowing retailers to understand what is being purchased where with greater accuracy, and gain insight as to why with greater speed.
As these technologies continue to mature, organisations can position themselves to capitalise on future opportunities by taking a business-value-oriented approach to incorporating big data techniques into their enterprise architecture. This requires your business to shift its thinking about business data from a cost to an asset. It also requires that IT avoid the temptation to view big data as a point solution in its own silo. In recent research, Forrester recommends enterprise architects lead their organisation's thinking by: 1) assisting the business in opportunity recognition; 2) maintaining a holistic approach to information architecture; 3) valuating how applications will integrate knowledge via BI, PA and SOA; and 4) updating your infrastructure technology road map and watchlist.
Use a business capability map to foster dialogue about opportunities to extract value from sets of data thought too large and cumbersome to handle. Develop powerful stories using business scenarios, then drill through to affected processes and metrics. Business scenario analysis creates a narrative of the people, processes and technology changes required to get from where the business is to where it wants to be.
Update your information architecture strategy considering the special characteristics of candidate big data sets. Define metadata to understand key attributes required for analysis and develop a sourcing strategy to answer important questions about where the data will come from (organic growth or acquisition); how fast it must be captured; how good it is; and where it will be maintained. Ensure data governance processes are adequate for the volume, velocity and content of proposed big data sets. Security, liability, and intellectual property are important issues as well - understand how information in large data sets is different, especially after analytics produce consolidated knowledge.
Use your understanding of potentially large data sets to develop BI and PA integration patterns to create a healthy service-oriented architecture (SOA) capability. Consider how knowledge from large data sets can be used to drive processes via event-driven SOA and complex suppliers to provide big data processing capabilities where possible when the data is appropriate to outsource.
Plan to enable horizontal scale by updating your technology road map with virtualisation infrastructure technologies and quickly put them in place when required. Evaluate data management technologies and be ready to test these in proof of concepts in support of business case development. Place big data application processing technologies on your watchlist and monitor their maturation in the context of valid business scenarios.
Brian Hopkins is a principal analyst at Forrester Research, where he covers emerging technology and its impacts on business and IT for Enterprise Architecture Professionals. He blogs at: http://blogs.forrester.com/brian_hopkins.
This was first published in August 2011