Solution Brief

Managing Amazon Web Services (AWS) Application Log and Event Data at Scale

The Challenge: Logging across AWS services is easy – but how do you analyze, search, and query that data

Amazon Cloudtrail logs, Cloudfront Logs, ELB/ALB/NLB logs, VPC Flow Logs, and S3 Access logs — AWS users have an endless stream of logging data available to them. With the click of a toggle, you enable logging for services that will automatically stream log data to your Amazon S3 bucket – without writing any code or running additional services. However, the challenge you now face, is where and how are you going to analyze, search, and query the exponential growth that’s happening in your data sets?

Amazon Log Management tools pose challenges

Many companies leverage the Amazon service ecosystem and use tools like Redshift, or Elastic Map Reduce. Some even move their data into Dynamo to better organize, categorize and query data. The problem is that these tools lack essential data visualizations, and you are left to build your own internal user interface and reporting framework.

Amazon Athena isn’t designed for searching log files

Another common solution used to query data is Athena. However, with Athena, you need to know what you are looking for specifically. Athena doesn’t provide hunting and searching capabilities, with full-text searching and wildcard support. Like the other solutions, Athena also lacks a native, easy way to generate visualizations on your data. You have to be prepared to bring third-party tools, such as Tableau and you will also be required to spend time creating the database and table schemas.

The difference between a $3 and a $300 query on Athena is as simple as a poorly formed query. The ability to easily make costly mistakes means it can be very challenging to control, understand, and manage the cost of data analysis on Athena.

Elasticsearch deployments are complex and costly in time and capital

Engineers frequently resort to using the Logstash, Amazon S3 input plugin to read logs from AWS S3 and index them to an Elasticsearch cluster. However, gaining meaningful data insight is capital intensive and time-consuming. Technical operators need to setup Logstash, define the appropriate Grok filters, and deploy, configure, and tune an Elasticsearch cluster. Companies are forced to make significant engineering investments, spending hours setting up the pipelines to support ingesting this data. Since all compute resources on AWS have associated costs when running these systems, your Amazon expenses for compute, storage, and networking increases along with your data volume and usage.

A New Strategy with CHAOSSEARCH

Use your existing Amazon S3 infrastructure

The CHAOSSEARCH platform lets you search, query, and visualize your Amazon application logs without moving or transforming your data for services like Dynamo or Redshift. We eliminate transfer charges and security risks by enabling you to keep your data within the confines of your own S3 bucket. The CHAOSSEARCH platform uses Amazon standard, cross-account IAM read-only access to index your S3 bucket. We give you the ability to immediately ask questions and gather information to drive better business outcomes.

Reduce Amazon Athena spend and find the exact information you need

CHAOSSEARCH allows for full text and wildcard queries across all fields, enabling you to find what you are searching for more quickly and easily. The CHAOSSEARCH platform includes access to the powerful open-source visualization tool, Kibana – allowing you to leverage the same dashboard and queries as your existing Kibana interface. Don’t spend time learning new tools. Index your data with CHAOSSEARCH and improve your data insights to drive business value.

Leverage the power of Kibana, on your own S3 account

Take advantage of the full potential of Kibana and Elasticsearch APIs on all your AWS log and event data in your own S3 bucket. Continually process AWS Cloudtrail to get visibility into AWS account usage and understand usage patterns and identify risky behavior. Stream AWS load balancer logs (ELB/ALB/NLB) directly into your S3 bucket to more quickly search and analyze them. With CHAOSSEARCH you gain insight into application logs without running AWS Elasticsearch or other expensive SaaS search tools. Most importantly, you can do all this without ever moving your data.