maciek905 - Fotolia

Hadoop Summit 2016: Data in motion governs Hortonworks roadmap

Hadoop distributor Hortonworks uses Hadoop Summit 2016 in Dublin to announce alliances and product developments to support data in motion focus

Hadoop distributor Hortonworks has announced a clutch of alliance and product developments, at Hadoop Summit 2016 in Dublin, which emphasise its plan to support “data in motion”. The summit also marks the tenth anniversary of Hadoop.

Hortonworks president Herb Cunitz told a group of analysts on the eve of the summit, at which the open source company has hegemony, that it is now set on managing all the world’s data – in motion as well as at rest. It had previously declared an ambition to manage half of the world’s data, he said.

The data in motion effort is based on the 2015 acquisition of US National Security Agency (NSA) spin-out, Onyara – a so-called “internet of anything” (IoAT) company. This was originally open sourced under the NSA’s technology transfer programme as an Apache project called Nifi to process and distribute data, with Onyara being the company that commercialised it.

It is now termed Hortonworks DataFlow (HDF) and is said to “collect any and all IoAT data from dynamically changing, disparate and physically distributed sensors, machines, geolocation devices, clickstreams, files and social feeds via a highly secure lightweight agent”. It transfers data into the company’s Hadoop data platform.

Royal Mail, Centrica, insurance company Markel and Dutch telecommunications company KPN said, in a customer panel at the conference, they were using HDF, as well as Hortonworks Data Platform (HDP), the supplier’s more established product.

Thomas Lee-Warren, director, technology data group (business intelligence, commercial insight and analytics) at Royal Mail, told Computer Weekly that HDF is being used by the 15-strong data insight group there: “We’ve got a place now where we can incubate large amounts of data, but there is also the matter of pace. With HDF, we can do a lot of things much quicker, and be more creative with data.”

Read more about data in motion in big data technology sets

The company has announced an expansion of a partnership with data integration company Syncsort to extend Hortonworks’ data ingestion capacity to mainframe computing environments. Hortonworks will resell Syncsort’s DMX-h Hadoop extract, transform and load (ETL) tool.

Hortonworks said, in a statement, that the combination of HDP with DMX-h enables data from traditional legacy systems like mainframes to be onboarded into Hadoop in a format that admits for “fast and easy analytics”. It will begin reselling Syncsort DMX-h in the second quarter of 2016.

Apache Metron

The company has also announced a cyber security Apache product Metron, which aims to make real-time data management the basis of security. The Metron community includes Rackspace, ManTech, B23, as well as Hortonworks.

It is said that when an organisation is attacked, Metron users can process and compare data from comprehensive feeds across the platform in real time. 

Jamie Engesser, vice-president and general manager of emerging products at Hortonworks, said: “Cyber security professionals need real-time big data analytics to provide a comprehensive and contextually intelligent view of all security data to enable rapid detection and response to advanced security threats.

“Traditional security tools with a rules-based approach do not scale to match the speed and frequency of modern cyber security threats.”

Hortonworks also announced that its fellow Open Data Platform company, Pivotal, will standardise on HDP, and that Hortonworks will provide professional customer support and professional implementation services for Pivotal’s SQL-on-Hadoop tool Hawq.

Read more on Big data analytics