Every government agency needs a "big data" policy to cope with the rising tsunami of data, to protect personal data, and to extract information on which it can act, says a report from the US President's Council of Advisors on Science and Technology (PCAST).
The report, Designing a Digital Future, said the collection, management and analysis of data was a "fast-growing concern" of network and IT (NIT) researchers as sensors and new data sources pushed up data quantities exponentially.
The authors of the 119-page report included Teradata CTO Stephen Brobst and Richard Winter, president of Winter Corporation, a consultancy that specialises in very large data solutions.
The report explored five themes: the growth of big data, the security of large systems, the importance of privacy in data collecting and use, interoperability between systems at all scales, and supply chain vulnerabilities.
The report described "big data" as data sets too big for common IT tools commonly to handle in a useful time. Big data sources included web logs, sensor networks, social media, detailed data captured from telecommunications networks, astronomical observations, biological systems, military surveillance, medical records, photographic archives and video archives, it said.
Advances in massively parallel processing (MPP) frameworks such as shared nothing relational databases, MapReduce programming frameworks, and cloud infrastructure, were essential for processing and extracting value from big data, it said.
Brobst said, "We are entering the Zettabyte Age. It is not hyperbole to say 'Every federal agency needs to have a big data strategy.' This is an imperative, and leading commercial organisations are already refining such strategies."
Winter said nearly all government goals needed better, faster decisions backed by key facts and insights. "These must often be discovered by analysing mountains of data," he said. "The stakes are now too high for the outdated approaches that got us to this point."
Winter called for "a well-defined, forward-looking analytic data strategy" that allowed officials to quickly analyse vast quantities of data so that they could respond quickly and effectively to events, and to do so without compromising people's privacy, as government gave greater public access to its information.