Blackpoint Cyber Taps ChaosSearch to Improve ThreatOps and Drive Growth! Check out the video-->
Blackpoint Cyber Taps ChaosSearch to Improve ThreatOps and Drive Growth! Check out the video-->
Start Free Trial

ChaosSearch Blog


Can You Use the ELK Stack as a SIEM? A Fresh Take

Can You Use the ELK Stack as a SIEM? A Fresh Take

A SIEM system (Security Information and Event Management) is often used by security operations centers (SOCs) for real-time detection of suspicious activity and security events.

While some teams choose to adopt a purpose-built SIEM, others rely on the same DevOps tools they are already using for tasks like troubleshooting and operational log data analysis.

That’s why we’re here to explore whether the ELK Stack (Elasticsearch, Logstash, and Kibana) can be sufficiently used to replace SIEM tools.

While the ELK Stack is not a SIEM itself, it can be used to build one. However, alternative approaches like a modular security data lake are less costly, and can augment existing real-time monitoring and observability solutions you already have in place.

Let’s learn more.


Using the Elk Stack as a Siem


Understanding SIEM tools

SIEM systems have become an important part of a cybersecurity tech stack, because of their ability to effectively monitor user activity and track any anomalies in data. In addition, SIEMs can help SOC analysts gain visibility into who is accessing your systems, when, and by what means. They work by gathering event files and log data across various systems, and normalizing the data into a standard format so it can be effectively analyzed.

While SIEMs are great for real-time threat detection and prevention, advanced persistent threats (APTs) are increasingly common. APTs linger in your system and engage in activity that appears to be completely normal at the surface level. Over time, an APT may escalate its privileges and exfiltrate sensitive data. When an SOC is focused on detecting real-time threats, an APT may remain unnoticed. However, by analyzing long-term log files, security teams can more effectively engage in proactive threat hunting to track down these bad actors.

Now that we know more about a SIEM, let’s look at the basic capabilities of the ELK Stack.


ELK Stack Overview

The ELK Stack, also known as the Elastic Stack, comprises three open-source software tools: Elasticsearch, Logstash, and Kibana. When integrated, these tools create a solution for centralizing log data aggregation, management, and querying, applicable to both on-premises and cloud-based IT environments.

Want to learn more about Elasticsearch, Logstash, and Kibana? Check out the Ultimate Guide to ELK Logging Analysis.


Check out the blog! The Ultimate Guide to ELK Log Analysis. What to expect when deploying, configuration and operating an ELK Stack.


Can you use the ELK Stack as a SIEM?

The ELK stack itself is not a SIEM, but can be used to build one. The ELK stack shares many common features with a SIEM on its surface.

For example, you can collect and query log data from a variety of sources. However, it’s important to note that you must configure it appropriately for real-time security threat detection and carefully monitor its outputs, which may be a significant challenge for under-resourced teams.

Another important aspect of a SIEM is the log processing pipeline in Logstash. SIEMs typically normalize data coming from different sources — meaning that data is broken down into meaningful field names, mapped to the right field types, and enriched with more information (for example, geographical information). Without data normalization, it’s incredibly difficult to analyze your log data in Kibana. Configuring Logstash to process various log file types is complex, and requires dedicated expertise.

With that said, many teams choose to use open-source tools as they architect their own SIEM solution. The pros? Open-source tools like the ELK Stack are cost-effective to get started with, and can be customized in a number of ways. There’s also a vast community supporting the open source solution. But, the downsides of ELK such as the management complexity, resource intensity, hidden cost centers (e.g. the high costs of log ingestion and retention), and expertise required may deter many from using ELK as a SIEM.

Retention can be an issue, in particular. Managing Elasticsearch clusters is costly and complex. If you don’t retain enough historical log data, it can be nearly impossible to determine whether a security event is anomalous or normal.

As one Quora user put it, “While you certainly can run security analysis on ELK, SIEMs typically provide additional out-of-the-box workflow features for Security Analysts that are vital when handling threats.”

For example, if you’re using ELK, you’ll need dedicated expertise to be able to normalize various log data formats, query data, create dashboards and reports, and correlate data with rules so that you receive alerts when an incident occurs. SIEMs also include dedicated incident management workflows, which are not provided in the ELK Stack.


Ready to stand up a security data lake? Learn how ChaosSearch enables scalable log analytics for security operations and threat hunting. Read the Solution Brief!


Alternatives to ELK Stack for SIEM

There are a wealth of SIEM tools available today from companies including Splunk, QRadar, Elastic (the proprietary software company behind the open-source ELK Stack), and more. These tools are effective for short-term security workloads, such as real-time threat detection and alerting.

Some SOCs find shortcomings with SIEMs, due to the sheer volume of cloud telemetry data.


The most common SIEM challenges:

  • Cost and Maintenance: The upfront costs, along with the costs of setting up, fine-tuning and maintaining a SIEM can add up. This includes the cost of data retention and processing. Security analysts often need to manually tag and continually update these systems in order to keep signals accurate, which can take a lot of time for companies facing a security talent shortage.
  • Data retention: Many SIEMs only retain data for 30 days due to cost issues, which limits the ability to investigate advanced persistent threats.
  • Alert fatigue: SIEMs throw a lot of false positives, which makes it hard for security analysts to distinguish between signal and noise within their real-time alerts.


Consider a Modular Security Data Lake (with or without XDR)

Another option is to embrace a modular security data lake option. Many teams choose to augment Splunk and other SIEMs with a solution that’s built for historical data analysis. Data lake solutions like ChaosSearch, for example, leverage low-cost cloud object storage like Amazon S3 to avoid the high costs of ingestion and data retention.

A modular security data lake combined with an XDR (Extended Detection and Response) solution is another option to consider. Security attacks today are increasingly sophisticated and rarely exploit a single endpoint. An XDR can move beyond the limits of a SIEM by providing comprehensive monitoring of the entire attack surface. Having this broader visibility means that an XDR can identify more patterns in your data to detect potential threats. The goal is to help security teams correlate seemingly disconnected events, to take immediate action and mitigate cybersecurity threats.

Organizations should carefully assess their specific requirements and resources before choosing ELK as a SIEM or consider alternative SIEM solutions like the ones presented above. For example, you might ask yourself these questions before building your own SIEM with the ELK stack:

  • Do you have the resources to appropriately configure your ELK Stack for security workflows? For example, can you connect your ELK-powered SIEM with the right incident management tools for your team?
  • Do you have enough storage capacity to handle the indexing and retention complexity of the ELK Stack?
  • Is your organization prepared for the fluctuating costs associated with this storage?
  • Are you able to get the historical data you need from your SIEM solution to conduct threat hunting?

If your answer to these questions is “no,” a modular security data lake may be a more cost-effective solution, augmenting long-term workflows like threat hunting, while leaving real-time workflows to a purpose-built SIEM.


Witness the Power of ChaosSearch: Live Analytics on Streaming Data at Scale. Book your Demo today!

About the Author, David Bunting

David Bunting is the Director of Demand Generation at ChaosSearch, the cloud data platform simplifying log analysis, cloud-native security, and application insights. Since 2019 David has worked tirelessly to bring ChaosSearch’s revolutionary technology to engineering teams, garnering the company such accolades as the Data Breakthrough Award and Cybersecurity Excellence Award. A veteran of LogMeIn and OutSystems, David has spent 20 years creating revenue growth and developing teams for SaaS and PaaS solutions. More posts by David Bunting