ChaosSearch Blog - Tips for Wrestling Your Data Chaos

Kubernetes is eating the world; you can digest K8’s plume

Written by Dave Armlin | Apr 24, 2020

Innovation in hypervisor technology in the early 2000’s from both commercial and open source projects was the genesis for the public cloud as we know it today. Virtualization and Moore’s law, together with advances in storage technology, mobile and wireless, created a data explosion that continues to accelerate through today.

 

Containerization has brought yet another wave of growth and fed the microservice movement that is driving digital transformation and rapid creation of new services. Kubernetes emanated out of the Google Borg project in the early 2000’s and has evolved into the Defacto standard for deployment and orchestration of containers. Stealing from Marc Andreessen's quote on software, you can say that containers and Kubernetes are eating the world and have become the backbone of modern infrastructure. The 451 Group has projected the Container market to be a $4.3B market by 2022.  KubeCon’s growth to 12,000 attendees in 2019 is testament to Kubernetes role in this market. As a Boston based company we are hopeful that continued progress with the Covid-19 crisis will allow the 2020 KubeCon to occur here in Boston in November.

 

Kubes, or K8’s, gives enterprises incredible power to build, deploy, and scale production systems with great efficiency. While Kubernetes has been a great catalyst in the growth of cloud-native, multi-cloud microservices, it comes with a virtual firehose of log data. This torrent can quickly become unwieldy, difficult to manage, and hard to extract value from at scale.  Enterprises are challenged deciding how to manage their Kubernetes logs and wrestle with the ability to store more than a few days or weeks of logs at a time.


ChaosSearch’s core mission is to revolutionize access to data at scale stored in cloud object storage. ChaosSearch is a SaaS service that delivers on this mission, requiring zero administration and enabling instant value realization from data stored in object storage. Another revolutionary piece of the ChaosSearch story is that its DataFabric architecture empowers this on the customer’s cloud object storage, directly, without requiring movement of data. 

 

Kubernetes logs at scale are yet another great use-case for ChaosSearch.

 

ChaosSearch allows you to harness immediate insights out of Kubernetes log data, providing automating indexing, full text search and analytics at scale, quickly and easily.  Get the data to S3 and ChaosSearch does the rest, with disruptive price performance at TB to PB scale.  Chaos enables long tail unlimited retention of data that offers unique opportunities for you to get more value out of operational and application log and event data, opening up the ability to do trending, or analyze long term performance or security issues.

 

In the Kubernetes world we benefit from the fact that there are a rich set of configurations that allow you to log and get data to S3.  Our customers are using Fluentd to ship Kubes data to S3 and then using ChaosSearch to quickly index and provide access for search and visualizations with our embedded Kibana 7.2.4 from the Amazon Open Distro for Elasticsearch.

 

 

ChaosSearch will empower you to:

  • Monitor and analyze Kubernetes logs at scale
  • Quickly identify and resolve issues in Production
  • Alert on events in your K8’s environment 
  • Integrate your Kubernetes environment with business-critical systems like Slack, PagerDuty, OpsGenie, Jira, ServiceNow, Zendesk and more. See all integrations.

 

See ChaosSearch and Kubernetes live on Wednesday, April 29th as Director of Solutions Architecture Kevin Davis and I show the power of these two platforms together.

 

Register here