The average cost of a data breach is expected to hit $5 million in 2023. For many organizations, it is a matter of when, not if, a cybersecurity incident will occur. Attackers are becoming more sophisticated and relying on weak links to exploit company applications and infrastructure.
Combine this phenomenon with the fact that the traditional network security perimeter has changed (and all but disappeared). Cloud computing and remote work have driven this trend. As a result, the sheer volume of telemetry data security teams must analyze and retain for threat detection, incident response, and threat hunting can easily overwhelm legacy security analytics tools.
This blog post will look at how tools like Security Information and Event Management (SIEM) systems perform in cloud-native environments, and compare them with emerging approaches such as combining an Extended Detection and Response (XDR) with a modular security data lake.
As threats and attack vectors multiply and increase in complexity, it is essential to store data longer and to bring in more data sources. A security data lake can help teams sift through the noise, investigate, respond, and mitigate real threats as they emerge, as well as look at the entire lifecycle of an incident comprehensively.
On top of that, a flexible layer of automation can drive analysis of the many data sources in a security data lake, assess risk, and engage security teams when necessary to provide human review of conditions.
Some tools, like SIEMs, face challenges when it comes to scale, cost and root cause analysis capabilities. We’ll cover those in more depth below. That’s why many security teams choose a data lake to separate storage from compute. A modular security data lake can be built on top of low-cost cloud object storage like Amazon S3 or GCP. From there you can index and search log data and other application telemetry data at a lower cost and at scale.
Security data lakes help you centralize and store unlimited amounts of data so analysts don’t have to access logs across several different sources. This supports many security use cases, particularly threat hunting and detecting advanced persistent threats.
A security data lake can be used in tandem with an XDR (or a SIEM, in many cases) to detect and respond to cloud-based security threats.
Depending on the vendor, an XDR and a SIEM may have some overlapping capabilities. Let’s review the similarities and differences, including how an XDR could potentially replace a SIEM with the reinforcement of a security data lake. In some cases where a SIEM is already deeply embedded, complementing a SIEM (like Splunk) with a security data lake can be more cost-effective and performant — especially for security investigation workloads that require a large volume of historical log data.
Many organizations use a SIEM for security analytics and threat hunting. A SIEM analyzes log data and telemetry, and provides real-time alerts about potential incidents. Key capabilities of a SIEM may include:
The sheer volume of cloud telemetry data can make SIEM systems impractical as the sole security analytics tool for many organizations. Some of the most common challenges include:
Security attacks today are increasingly sophisticated and rarely exploit a single endpoint. An XDR can move beyond the limits of a SIEM by providing comprehensive monitoring of the entire attack surface. Having this broader visibility means that an XDR can identify more patterns in your data to detect potential threats. The goal is to help security teams correlate seemingly disconnected events, to take immediate action and mitigate cybersecurity threats.
Both an XDR and a SIEM are designed to collect and analyze security data within a central location. However, a SIEM and XDR are different in a few key ways:
Between SIEM, SOAR (security orchestration, automation and response), XDR and other security monitoring and analytics technologies, it can get confusing and costly for organizations — fast. In many cloud-native environments, an XDR plus a security data lake can be better than a SIEM (if a SIEM is not already in place).
As mentioned above, a security data lake is often used in tandem with a SIEM to reduce the cost associated with ingesting and storing a high volume of log data and telemetry data, which is typical in cloud-native environments. Using a SIEM for real-time detection and a security data lake for deep investigation can be a great way to reduce costs and play to the strengths of each of these security analytics solutions.
Some of the advantages of a security data lake include:
Layering an XDR with a security data lake can result in better coverage than a SIEM alone, by providing multiple layers of security. Remember how SIEMs often cause alert fatigue, due to the overwhelming amount of individual alerts triggered by the system? These can make it difficult to understand which threats need immediate attention. An XDR can solve these issues by correlating and connecting log data to gain context into a security event.
The deep activity data within an XDR can be fed into a security data lake for more extensive threat-hunting and investigation capabilities. This reduces the total cost of ownership (TCO) of a security analytics solution, making it more cost efficient and performant to conduct deeper searches of log data that extend beyond 30 days.
A security data lake powered by solutions like ChaosSearch can reduce the TCO of tools like a SIEM. Sending critical security logs to ChaosSearch can reduce storage and maintenance costs of a SIEM, without limits on ingest or retention. For companies that don’t have a SIEM in place, or are frustrated by the limitations of a SIEM, it may be more economical and effective to use an XDR with a security data lake for contextually aware alerting paired with deeper threat hunting and investigation capabilities.
As an added benefit, a security data lake can improve compliance by retaining data beyond the 30-day window many SIEMs implement. Ultimately, there’s no one tool to rule all in a cloud native world. A modular security data lake can dramatically reduce costs of traditional security solutions like a SIEM, while allowing for added context of an XDR.
Want to learn more about threat hunting?