Going beyond search: Elastic’s observability and security play

Elastic, the company behind the popular Elasticsearch open-source project, has evolved significantly from its origins as a tool for integrating search into applications. Today, the company offers capabilities spanning security, observability, and increasingly, generative artificial intelligence (GenAI) – all built on a foundation of flexible, open-source technology.

In a recent interview with Computer Weekly on the sidelines of the ElasticON developer conference in Singapore, Ken Exner, Elastic’s chief product officer, explained how the company’s product strategy has adapted to accommodate this expansion while remaining true to its open-source roots.

“It started off as Elasticsearch, an open-source project for pulling search into an application,” Exner said, noting that it is “the most popular Java open-source project of all time”.

Since then, Elasticsearch has evolved in two ways. “One is, search has evolved from what used to be just text-based search into semantic search and vector search and sort of generative AI use cases,” he said. “The other way it has evolved is using it for observability and security.”

Elastic’s expansion into observability and security arose from observing how its users were using Elasticsearch to comb through large volumes of logs for operational and security insights. “People started using us to search through logs,” Exner said. “So, rather than use the grep command, they could use a search engine.”

Recognising this use case, Elastic developed out-of-the-box capabilities for log analytics and threat hunting, laying the groundwork for broader observability and security offerings. Today, the company offers a suite of tools spanning application performance monitoring (APM), metrics, traces, profiling, endpoint protection, cloud security, as well as identity and behavioural analytics.

This expansion, Exner noted, typically follows a “land and expand” pattern. “We typically land with logs,” he said, “and then customers realise they can use us for metrics too.” He cited the example of US bank Wells Fargo, which is using Elastic’s observability and APM capabilities to not only monitor events but also combining that with business data to quantify the financial impact of operational issues on customers.

However, the breadth of functionality of Elastic’s platform can pose a challenge in catering to vastly different user personas, which the company is addressing by providing tailored experiences through its new serverless offering that was built on a new stateless architecture.

“We’ve created user experiences for each type of customer, because if you’re an SRE [site reliability engineer] or DevOps person, you don’t know what endpoint protection or search relevance is,” Exner said, adding that by crafting distinct interfaces and workflows tailored to specific roles, Elastic lets users focus on their areas of expertise while still enabling them to benefit from the platform’s underlying flexibility.

In fact, this flexibility, Exner said, has been a key differentiator for the company, which competes with Splunk and IBM in the security information and event management (SIEM) market, as well as Dynatrace and others in the observability space.

“Most companies tend to create something that’s very easy to use but not flexible, or something that’s very flexible but not easy to use,” he said. “The way we’ve approached things is you start from something that’s very flexible, and then you layer on levels of abstraction and make it easier to use, but customers can always drop down to the lower level.”

The company’s open source roots also contribute to its flexibility. While acknowledging that some features are available only to paying customers, Exner said for core functionality like log analytics and vector search, “there’s not much difference in what the community and our paying customers get.”

He also addressed the challenge of monetising open-source software, particularly in the Asia-Pacific region. “What I often find is, if you have something that makes sense economically, people will choose to pay,” Exner said, pointing to Elastic’s popular searchable snapshots feature that lets customers enjoy tiered storage options and cost savings by leveraging Amazon S3 storage.

Looking ahead, Exner sees GenAI as a natural evolution of search, with Elastic well-positioned to capitalise on this trend. “We’ve been doing vector search, and we’ve extended that into the retrieval augmented generation workflow,” he said, adding that Elastic also provides observability and security tools for large language models to help organisations monitor and secure generative AI applications.

Despite its expansion, Exner said Elastic remains committed to its open-source philosophy. While some features are licensed under a proprietary licence, he said the core of the product remains open source. “Everything we do is on GitHub where people can submit pull requests to us,” he said. “And it’s not just dependent on us – the community can contribute as well.”