How to make Hadoop logfile analysis fun & interesting

21e12devwev.jpgThe heady world of server log analysis kicks up another gear this week with new products proffered forth from XpoLog Ltd, the company that says it “invented” augmented search for IT log analysis.

What’s the augmentation and why is it interesting?

We’re glad you asked — this is a process by which DevOps professionals can discover hidden:

• errors,

• anomalies,

• problems and,

• patterns…

… that may be hiding inside IT log data.

What is IT log data or a logfile?

This is a file that records “events” occurring throughout an operating system or software application (in the case of XpoLog this software is focused on open source Hadoop) and it can also incorporate message information between users in a network — a list of other log types can be found here if you need bedtime reading.

Augmented auto-detected intelligence


XpoLog is aiming to bring more to the party than plain old log searches and now offers what it calls search with “augmented auto-detected intelligence” based on user search context.

We digress, back to the news… XpoLog Augmented Search 5.0 brings XpoLog’s troubleshooting capabilities to the Hadoop platform.

The company states the obvious but still pertinent factor at play here i.e. testing applications on Hadoop (a large-scale, distributed data processing platform) isn’t a trivial task.

Pattern and anomaly detection

“XpoLog adds intelligence to log file search context with semantic analysis, and pattern and anomaly detection (to uncover insights and trends into application problems, systems, and user behaviour),” said XpoLog VP Solutions Omry Koschitzky.

“This helps users analyze problems within the Hadoop infrastructure and applications that run on the platform. It offers visibility into the distributed architecture, automatically triaging issues and errors for severity, and presenting results in a dashboard interface,” he added.

The company has made the XpoLog platform free for processing up to 1 gigabyte of log data per day.

More fun and interesting than you thought?