ChaosSearch revolutionizes the AWS Security Lake ecosystem with real-time data indexing and analytics, transforming the way security data is processed and analyzed.
ChaosSearch's capabilities empower organizations to utilize their security data more efficiently and effectively, significantly bolstering their security posture within the AWS Security Lake ecosystem.
Amazon Security Lake is a powerful data gathering platform that centralizes security data from AWS and third-party sources into a data lake stored in your AWS S3 account. Security Lake stores the data in Apache Parquet format and the standard open-source schema called the Open Cybersecurity Schema Framework (OCSF).
The ChaosSearch Data integration for Amazon Security Lake cost-effectively ingests and activates that important security object store for investigative analytics. ChaosSearch offers multi-model data access to users via open APIs such as Elasticsearch and SQL, or via the Kibana and Superset UIs included natively. Monitor, alert, and threat hunt using all your Security Lake data—without retention limits—to face today’s complex security environments and persistent threats. Combine the valuable Security Lake data with your application or infrastructure logs to improve your organization-wide observability and security posture.
With Amazon Security Lake, important AWS applications and key third-party source partners have united to home their security information into one reliable, secure, and accessible location, with common formats and an OCSF schema. Security Lake is a new way to confront the persistent threats to global business. ChaosSearch brings a new approach to increase your ability to monitor and analyze that security content at scale, while reducing operational costs of text search and relational analytics.
If you already have ChaosSearch set up and indexing your log and event data, you can add Amazon Security Lake S3 buckets for indexing and analysis. (If you are a new ChaosSearch customer, go to the next section.)
Configure Amazon Security Lake for your AWS account using the Getting Started steps in the AWS user guide. This will create a new IAM role with two policies.
Copy those policies and attach them to the existing ChaosSearch role ARN following the ChaosSearch help topic Attach the Policy to the ChaosSearch IAM Role. NOTE: If your existing ChaosSearch read-only policy has permissions to read and get all buckets on the account that Security Lake has created, and you want ChaosSearch to be able to read/index all Security Lake S3 data, you do not need to make any changes to existing permissions. This change allows ChaosSearch to see the new Security Lake bucket(s).
On the ChaosSearch Storage tab, create a live object group to index the Parquet files in the security lake S3 bucket. ChaosSearch has built-in support for the Parquet format.
Create a refinery view to enable searching and visualization for the indexed data created for the object group.
If you are new to ChaosSearch and want to start using ChaosSearch to index Security Lake S3 files, follow these steps.
In AWS S3, in the same account as the role, create a new bucket aws-security-data-lake-cs-<CUSTOMER-UID> (match the name used in the policy above). This is the read-write bucket for the ChaosSearch configuration and indexed data.
Connect your AWS resources to the ChaosSearch environment following the ChaosSearch help topic Add Role ARN to ChaosSearch AWS Credentials.
Import the Security Lake bucket(s) into ChaosSearch using a command similar to the following. NOTE: The bucket endpoints currently require an AWS V4 signature for authentication.
The following sample images show an example of the ChaosSearch Discover results for VPC Flow Logs, which you can use to hunt for potential threats on network traffic, and a VPC Flow Dashboard, which you can use to monitor network traffic, with the underlying VPC Flow Log data coming from Amazon Security Lake.
Common setup issues typically include errors or changes in the AWS role/policy definition and ARN information, or configuration of the object group(s) that ingest and index the data from the Security Lake S3 bucket (such as filtering/prefix changes or unexpected format errors). See the ChaosSearch documentation for the required IAM policies and configuration best practices. ChaosSearch Customer Success works with you to identify and correct any setup issues, and to refine RBAC policies for access to ChaosSearch object groups and views. Reach out to your ChaosSearch Customer Success representative via the dedicated Slack channel created for each customer or via email.
ChaosSearch Customer Success supports customers via dedicated Slack channels and email. Customer Success works with you to resolve configuration and querying issues, and can also help to tune settings to maximize economy and performance based on ingest volume and querying activity.
ChaosSearch offers a regular cadence of software updates for enhancements and fixes. Customer Success works with you to schedule upgrades, which can include enhancements to adapt for changes such as OCSF schema updates as well as evolutions to the Security Lake implementation.