Digitalisation feeds big data

This essay is a guest blog by Yves de Montcheuil, Vice President of Marketing at Talend.


Knowingly or not, many enterprises have embarked on a path to digitally transform their organisation. Also known as “digitalisation”, this transformation takes many forms, which can range from the introduction of data processing at every step of existing business processes, all the way to the pivoting of the enterprise’s business model towards the monetisation of its data assets.

Examples of digitalised organisations abound. Century-old catalogue retailers adopt modern big data technologies to get a 360-degree view of their customers and optimise their marketing. Banks from the City use advanced statistical models to predict the risk of customers defaulting on their loans. Manufacturers deploy sensors along their production lines to fine tune capacity and provide predictive maintenance, avoiding costly downtime.

Business model pivoting

Outsourcers of all kinds are especially prone to business model pivoting: because their models provide them with data sets from many clients, they can resell insight on these combined data sets – in an anonymised and aggregated mode, of course. From payroll providers selling statistics on salary ranges per function and location, to advertising agencies comparing a client’s ad returns with benchmarks, to ski resorts informing leisure skiers how they fared against local champions, or even aircraft manufacturers optimising routings (and fuel savings) based on information gathered from aircraft flying the same route earlier – the examples are limited only by the creativity of business development executives (and the propensity of clients to pay for such services).

Born digital

Some companies do not need to “digitise” – they were born digital. Looking beyond the obvious examples – Google, Amazon, Facebook, Twitter – many companies’ business models are based on the harvesting of data and its trade, in one form or another. Next-generation private car hire or hitchhiking/ridesharing providers are not transportation companies but intermediation platforms, bringing together drivers and riders based on location, and ensuring a smooth transaction between the parties.  Fitness apps collect data from exercising enthusiasts, providing this data, reports and alerts in an easily consumable format to the member, and further reselling it to interested third parties.

The common thread across all these different organisations? Their digital businesses are consuming and producing vast amounts of data. Social data, location data, transaction data, log data, sensor data constitute both the raw material and the outcome of their business processes.

For companies that were not born digital, some of this data existed before digitalisation began: banks stored credit instalments in paper ledgers for centuries and in computers for decades, aircrafts have fed measurements to flight data recorders since the 1960s, sensors in factories are nothing new but were historically used primarily to raise alert conditions. As digitalisation is embraced, this legacy data (or “small data”) becomes a key part of the big data that is used to re-engineer business processes, or build new ones. It will get augmented in two dimensions: more of the same data, and new data.

More of the same: digitalisation requires, and produces, more of the same data. In order to accurately predict consumer behaviour, detailed transaction data must be collected – not only aggregate orders. Predictive maintenance requires sensor data to be collected at all times, not just to raise an alert when a value exceeds a threshold. Route optimisation demands collection of location data and speed parameters at frequent intervals.

New: digitalisation also requires, and produces, new types of sources of data. Meteorological information is a key input to combine with aircraft sensors to optimise fuel consumption. Traffic data helps compute transit time. Web logs and social data augment transaction history to optimise marketing. 

The technical key to successful digitalisation is the ability of the organisation to collect and process all the required data, and to inject this data into its business processes in real-time – or more accurately, in right-time.  Data technology platforms have evolved dramatically in recent years – from the advent of Hadoop and then its transition to a data operating system with a broad range of processing options, to the use of NoSQL databases with relaxed consistency to favour speed at an affordable cost, and, of course, to the role open source is playing in the democratisation of these technologies.

Of course, while technology is necessary, it is not sufficient. Business executives need to fully embrace the possibilities that are now presented to them by the new wave of data platforms.


About the author

Yves de Montcheuil

Yves de Montcheuil is the Vice President of Marketing at Talend, which does open source integration. Yves holds a master’s degree in electrical engineering and computer science and has 20 years of experience in software product management, product marketing and corporate marketing. He is also a presenter, author, blogger, social media enthusiast, and can be followed on Twitter: @ydemontcheuil.