Solution Brief

Improve Security Incident Detection and Compliance Logging

The Challenge: Balancing data compliance needs and cost efficiency

Most corporate data retention policies range from 3 to 7 years

Compliance frameworks, such as SOC2, HIPAA, PCI, and ISO 27001 force companies to maintain more data than ever. This data needs to be, not only retained, but it must also be easily accessible for analysis and queries in the event of an incident or legal request. However, unless you continually process your data into a database, you’ll need to resort to time-consuming and expensive ETL (extract – transform – load) processes or indexing into databases like Elasticsearch to get the information and answers you need from your data.

Storing years of security event logs is cost prohibitive in Elasticsearch

Elasticsearch and Lucene were never designed to hold significant amounts of data for long periods. Elasticsearch allows for the millisecond speed of queries but at the cost of storing additional structure to the data – through increasing the size of the indexed data on disk. This increase in data on disk means additional cost as you will be required to scale out hot Elasticsearch servers or continue to add more volumes to existing servers. In the age of cloud – every server and every disk has an associated cost. So, when your data volume grows, your cloud infrastructure bills do too.

Running software, open-source or proprietary, on cloud providers can have a high operational cost in servers, disk, and network. Leverage SaaS to focus engineering on what matters to grow your business.

It’s time-consuming to make log and event data searchable in S3

Many users leverage the durable, secure, and inexpensive storage platform of Amazon S3 to meet their company’s compliance requirements. Also, they harness the long-term cold storage of Amazon Glacier for data they need to keep but may never query. Both these data storage locations have only limited ways to query and visualize this data. Using tools like Amazon Select for S3 & Glacier or Amazon Athena only allow you limited insights into your data, and lack any native data aggregation, data visualization, or more importantly, text search. To get high-level and granular insights into your data, you would need to move the data out of Amazon S3, index or transform that data into a database like Elasticsearch, and finally install tools like Kibana to create data visualization insights into your data.

A New Strategy with CHAOSSEARCH

Leverage the power of Amazon S3 – store events securely and inexpensively

With CHAOSSEARCH you can use your S3 to store data for long term retention – cost effectively. You simply enable Amazon IAM access for the CHAOSSEARCH platform and immediately begin to index your source data and gain insight and answers to your questions. We make it easy for you to maintain control and ensure security and compliance with company policy because we never hold or store your data. CHAOSSEARCH indexed data writes to your existing S3 buckets.

Move compliance logs to Glacier – save money and still search your data

Never pay for an Amazon Glacier retrieval request again. With CHAOSSEARCH once your data is indexed, you never need to return to the source. Set up an Amazon S3 lifecycle policy to push your source data to Glacier, while leaving the fully indexed data in your S3 bucket. The patent-pending CHAOSSEARCH technology stores your data in a highly compressed format AND makes it available for search, query, and visualization. We enable you to reduce your S3 spend by moving the source data to Glacier, while at the same time query the compressed CHAOSSEARCH indexes via Elasticsearch APIs or our embedded Kibana interface.

Get answers to your questions instantly – without expensive ETL or reindexing

CHAOSSEARCH continually and fully indexes the data within your Amazon S3 buckets, ensuring that your data is always available for query anytime. CHAOSSEARCH is a fully managed service that provides endless scalability to handle even the most complex questions across the most extensive data sets. CHAOSSEARCH decouples the storage and compute, which allows the platform to scale independent to the data, allowing us the ability to ensure you the ability to store everything and ask anything.