News Stay informed about the latest enterprise technology news and product updates.

Elastic twangs in snappy machine learning

No self-respecting data management firm operates today without a healthy dose of machine learning at the heart of its technology stack. Data search, logging, security and analytics shop Elastic clearly resonates with this new de facto reality as it now adds machine learning into its core arsenal or capabilities.

Elastic is of course the company behind the open source Elasticsearch and the Elastic Stack products.

Into the Elastic 5.4 release then… (as a result of the recent acquisition of data anomaly detection business Prelert) Elastic’s machine learning features will work on any time series data set to automatically apply machine brain intelligence.

What functions evidence machine learning?

That’s an easy question to answer i.e. functions such as:

  • identifying anomalies,
  • streamlining root cause analysis,
  • reducing false positives within real-time apps.

The concept behind this technologies is that it should be used when trying to spot infrastructure problems, cyber attacks or business issues in real-time.

“Our vision is to take complexity out and make it simple for our users to deploy machine learning within the Elastic Stack for use cases like logging, security and metrics,” said Shay Banon, Elastic Founder and CEO. “I’m excited that our new unsupervised machine learning capabilities will give our users an out-of-the-box experience, at scale to find anomalies in their time series data — and in a way that is a natural extension of search and analytics.”

Elastic Stack is being used to by developers for collecting, enriching and analysing log files, security data, metrics and text documents etc.

Why machine learning is tough

The firm says that machine learning is tough to bring online. Why is this?

Because the biggest challenge lies in developing real-time operational systems for existing workstreams and use cases.

“Scarce and expensive data science skills are needed to figure-out the correct statistical models for different, diverse data sets and hand-crafted rules are brittle and often generate many false-positives,” says Elastic.

Elastic’s new machine learning capabilities use a familiar Kibana UI . The software installs into Elasticsearch and Kibana with a single command as part of X-Pack.

Join the conversation

1 comment

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

An interesting point that comes out of this article is that most of the time, collection and management analytics data is a job carried out by IT, whilst the final analytics reports - i.e. the real stuff effecting business strategy - is required by a completely seperate group of non-tech people, wanting Business Intelligence rather than simply raw statistics.

This seperation of roles often leads the tech guys into deciding: "let's just collect/store all the raw data since we don't know how the business guys will be using it".

Although at face value this seems a safe approach it is very dangerous since it changes big data into exponentially bigger data which in turn increases infrastructure costs to maintain that data. It is also the reason people in the business section end up wasting time battling with their pivot tables and Database servers before they can come up with a semi-useful BI report that can answer their real-life product management questions.

In the worst case scenario, the amount of technical resources and time required to report on big data by far exceeds the short-term ROI, which makes the project seem unfeasible and the plug is pulled before anything great can come out of it.

One method by which a company can shorten the ROI (and reduce the risk of killing the project before it has time to show results) is by getting the IT guys to understand more specifically how the business section will use the final data. This will allow the IT to cleanup the data at source and store it in a report friendly manner as close as possible to the final reports needed by the business intelligence team. When dealing with big data, having it partially processed and cleaned up at source makes it much more efficient to store and report on.

The advantages are two-fold:
- Infrastructure requirements and costs to store and process reporting data are drastically reduced
- Product Managers can answer their businesss queries faster and thus will be happy to experiment with the available data since reporting is no longer a hassle to generate.

This leads to a faster ROI and better chance of survival since visible results become more evident in the shorter term.

Keith Fenech - Trackerbird Software Analytics -