Microsoft evangelist: big data needs tiny data skills

This is a contributed piece for the Computer Weekly Developer Network by Matt Ballantine, principal evangelist at Microsoft UK.

Ballantine recalls a curious observation taken from a conversation with a colleague who’d just arrived back from UCL with one of their Engduino ( devices…

Big data, tiny data

The Internet of Things might mean that whilst we need “big data” to collate and process masses of information…

… we might need some very different “tiny data” skills to be able to collect the stuff in the first place.

200911 Matt Ballantine Colour.jpg

We’ve started to see the emergence of very cheap, very low powered computing devices that are networked wirelessly. Think Arduino, but smaller, cheaper, and more connected. These devices will be able to sense and sample the world around them – temperature, humidity, light – whatever, really, can be built into electronic sensors.

When these devices become really small and really cheap, they’ll be able to open up entirely new ways for us to monitor the world around us; imagine, say, tiny networked and environmentally aware devices scattered across agricultural land to give continual data about the growing conditions.

In that kind of application, the amount of data that might be generated en masse will have the potential to be colossal.

Device “frugality”

But the key to success will lie in making the devices themselves as frugal as possible so as to be able to operate at very low power. Batteries have a finite life; solar power is only possible in places where there is light; constant streams of network traffic (as anyone who has a smartphone will well know) drain power sources faster than you can say “remember when a phone battery would last all we…”

Thinking about tiny data might mean a re-alignment of the way in which we think about collecting and transmitting information – certainly something a long way from the “capture everything” approach that seems to dominate the big data world.

There is a story that Nicholas Negroponte told in his 1995 book Being Digital that springs to mind; it went something like this:

A husband and wife are at a dinner party with friends. Conversation turns to the subject of a mutual friend of all those present, who isn’t in the room. The husband and wife know that the subject of the discussion is currently having an affair — knowledge of which the others at the table are not aware. The husband subtly winks at his wife. That single bit of data- the wink – conveys a huge amount of information between the two of them because of the shared knowledge and context.


Contextual data compression

This kind of contextual data compression might hold a key to ways in which the tiny amounts of power and processing available through Internet of Things sensors could deliver meaningful services in the future.

Whether the world of software engineering is ready for this kind of extreme efficiency after decades of Moore’s Law giving seeming boundless power (and some may argue bloated software as a result) is a whole other question.