Hadoop Summit 2016: Data in motion governs Hortonworks roadmap

Hadoop distributor Hortonworks has announced a clutch of alliance and product developments, at Hadoop Summit 2016 in Dublin, which emphasise its plan to support “data in motion”. The summit also marks the tenth anniversary of Hadoop.

Hortonworks president Herb Cunitz told a group of analysts on the eve of the summit, at which the open source company has hegemony, that it is now set on managing all the world’s data – in motion as well as at rest. It had previously declared an ambition to manage half of the world’s data, he said.

The data in motion effort is based on the 2015 acquisition of US National Security Agency (NSA) spin-out, Onyara – a so-called “internet of anything” (IoAT) company. This was originally open sourced under the NSA’s technology transfer programme as an Apache project called Nifi to process and distribute data, with Onyara being the company that commercialised it.

It is now termed Hortonworks DataFlow (HDF) and is said to “collect any and all IoAT data from dynamically changing, disparate and physically distributed sensors, machines, geolocation devices, clickstreams, files and social feeds via a highly secure lightweight agent”. It transfers data into the company’s Hadoop data platform.

Royal Mail, Centrica, insurance company Markel and Dutch telecommunications company KPN said, in a customer panel at the conference, they were using HDF, as well as Hortonworks Data Platform (HDP), the supplier’s more established product.

Thomas Lee-Warren, director, technology data group (business intelligence, commercial insight and analytics) at Royal Mail, told Computer Weekly that HDF is being used by the 15-strong data insight group there: “We’ve got a place now where we can incubate large amounts of data, but there is also the matter of pace. With HDF, we can do a lot of things much quicker, and be more creative with data.”

The company has announced an expansion of a partnership with data integration company Syncsort to extend Hortonworks’ data ingestion capacity to mainframe computing environments. Hortonworks will resell Syncsort’s DMX-h Hadoop extract, transform and load (ETL) tool.

Hortonworks said, in a statement, that the combination of HDP with DMX-h enables data from traditional legacy systems like mainframes to be onboarded into Hadoop in a format that admits for “fast and easy analytics”. It will begin reselling Syncsort DMX-h in the second quarter of 2016.