The automation of knowledge work featured in a McKinsey report last year as one of ten IT-enabled business trends for the decade ahead: ‘advances in data analytics, low-cost computer power, machine learning, and interfaces that “understand” humans’ were cited as technological factors that will industrialise the knowledge work of 200 million workers globally.
On the surface seems at odds with the rise of the data scientist. It has become commonplace in recent years to say that businesses and other organisations are crying out for a new breed of young workers who can handle sophisticated data analysis, but who also have fluent communication skills, business acumen and political nous: data scientists.
The problem is, not surprisingly, finding them. I’ve heard a few solutions offered. Stephen Brobst, Teradata’s CTO, suggested that physicists and other natural scientists – that is to say, not only mathematicians – are a good source.
Another approach is to automate the problem, in different ways and up to different points. Michael Ross, chief scientist at eCommera and founder of online lingerie retailer Figleaves, contends that online retailing does require industrialisation of analytics.
He told me: “E-commerce is more Toyota than Tesco. It’s more about the industrialisation of decisions based on data. It’s not about having an army of data analysts. It’s about automating. Physical retail is very observable. Online you’ve got lots of interconnected processes that look much more like a production line”.
And he drew a further parallel with the Industrial Revolution, which de-skilled craftsmen: “This stage is all about replacing knowledge workers with algorithms”.
As it happens, Ross is a McKinsey (and Cambridge maths) alumnus himself, but was basing his observations upon his experience at Figleaves, and elsewhere.
The supplier community – and Ross belongs to that at his company – is keen to address this problem space. For instance, SAP is developing its predictive analytics offer in the direction of more automation, in part through the Kxen Infinite Insight software it acquired last year. Virgin Media is using the software to generate sales leads by analysing customer behaviour patterns.
The limitations of Hadoop
Actian, the company that encompasses the Ingres relational database, has now positioned itself as an analytics platform provider. The pieces of that platform have come from a raft of recent acquisitions: VectorWise, Versant, Pervasive Software, and ParAccel. I attended a roundtable the supplier held last week, at which CEO Steve Shine and CTO Mike Hoskins talked about the company’s vision. Both deprecated what they see as a regression in data management inadvertently caused by the rise of the Hadoop stack and related NoSQL database technologies. Hadoop requires such a “rarefied skills set” that first phase big data projects have yielded little value, said Shine.
Hoskins said his colleague had, if anything, been too kind. “MapReduce is dragging us back to the 1980s or even 1950s”, he said. “It’s like Cobol programming without metadata”.
He said the data technology landscape is changing so massively that “entire generations of software will be swept away”. Mounting data volumes in China, and elsewhere in Asia reinforces much of what has been said in the west about the “age of data”, he continued and he characterized the ‘internet of things’ phrase as “instrumenting the universe. We are turning every dumb object into a smart object which is beaming back data”.
As for a putative industrialisation of analytics, he said: “the Holy Grail is ‘closed loop analytics’. Where one is not just doing iterative data science to improve a recommender system or fraud detection by 10%, but rather to drive meaningful insight into a predictive model or rule which then goes into a day to day operational systems. So it’s about closed loop analytics that enable constant improvement”.
The automation of data analytics does seem to make business sense. Will bands of data scientists emerge to contest its worth?