I started using Amazon S3 in 2006 to store CD images (iso’s) as it was an easy to use, cost-effective way to make CD images publicly available for download. In 2006, the cost of object storage was high — S3 was disruptive right out of the gate. Since those days, S3 has become the bedrock of storage for the cloud as we know it. S3 is used to store everything from static web content, application log and event data, AWS logs, backups, images, movies, metadata about these things, satellite data, IoT and mobile data, essentially, everything.
Reading CHAOSSEARCH founder Thomas Hazel’s post Data is Cheap, Information is Expensive – Part 2, two things jump out. First, CHAOSSEARCH “could disrupt the analytics space and go a long way to solving the cost and complexity dilemma of big data.” Second, S3 is the perfect storage platform on which CHAOSSEARCH can prove that you can “Store everything. Ask anything.”
To illustrate how perfect the marriage of CHAOSSEARCH and S3 is and how quickly you can unlock value from data in S3, below are the steps along with a video walkthrough I shared with a new customer. The customer is in the video advertising space whose platform is based on Microsoft technologies running in AWS and needed assistance getting their platform logs to S3 for the IIS and .NET servers.
If your data is already in S3, great — we’d love the opportunity to help you unlock value from it. If not, as you can see from the Logstash example, it is very easy to do. There are a wide variety of ways to get your data into S3 from the AWS CLI, Fluentd, to AWS Kinesis, to the Filebeat use Patrick Flaherty of CHAOSSEARCH describes, to S3 export features built-in to products that I describe in #6 of my recent blog post. Once you have your data in S3, it’s incredibly easy and exciting to see how quickly you can extract value with CHAOSSEARCH.