Forget big data, welcome crunchy data

Gartner’s Doug Laney has identified what he calls “dark data” as a new kind of asset class worthy of being put on the corporate books.

Andy Green, content specialist at Varonis also likes this new term and says that dark data is a subset of big data i.e. enormous but without formal boundaries as defined by database schemas.

“In other words, it’s the human generated content in documents, presentations, spreadsheets, notes, and other readable formats that make up the bits and bytes of a corporate file system,” said Green.

Crunchy data cometh

For data-centric developers, DevOps staff, DataBase Administrators (DBAs) and programmers focused on so-called “content-centric” applications (is there any other kind of application?) the arrival of dark data is perhaps only overshadowed by the next big thing on the data scene…

… get read for crunchy data.

While corporate dark data comes about as a natural by-product of employees creating content to communicate ideas (as Green reminds us, every document is after all a just a thought that’s been converted to bits) the inexorable rise of crunchy data is just one step away.

Crunchy data defined

So by way of definition, the origin of crunchy data stems from a classical economic definition as laid down by the late Nico Colchester, who was Deputy Editor of The Economist and before that Foreign Editor of the Financial Times.


Colchester described “crunchy economics” as the antithesis of soggy economics i.e. that state of total transactional being where “small changes have big effects” and so therefore…

… crunchy data is that most valuable of data where small changes to its values have the biggest effects.

Crunchy data is, if you will, distilled big data and cleaned dark data such that it has real value in the modern ERP system.

To list Colchester’s original thoughts as below:

“Crunchiness brings wealth. Wealth leads to sogginess. Sogginess brings poverty. Poverty creates crunchiness. From this immutable cycle we know that to hang on to wealth, you must keep things crunchy. Crunchy systems are those in which small changes have big effects, leaving those affected by them in no doubt whether they are up or down, rich or broke, winning or losing, dead or alive. The going was crunchy for Captain Scott as he plodded southwards. He was either on top of the snow-crust and smiling, or floundering thigh-deep.”

If you’re dealing with applications and data streams that have been defined within the big data universe, then pull away from dark data all you can and make sure you move towards crunchy data from now on.

Image credit: Packaging News