ChaosSearch Creates a Streamlined Centralized Logging Solution for Sixth Street Partners

Founded in 2009, Sixth Street is a global investment firm with over $60 billion in assets under management and more than 400 team members. As a global investment firm, with a variety of bespoke and commodity off the shelf applications used by customers and employees across the world, Sixth Street faces the difficult challenge of building a cloud presence on Amazon Web Services (AWS) with a cohesive approach to application lifecycle management, alerting, monitoring, and logging. Sixth Street partnered with Logicworks and ChaosSearch to build solutions to these problems.

Centralizing Application Log Sources

Logicworks, an AWS Premier Partner, deployed a Control Tower account management plane to help ease the burden of Identity and Access Management (IAM), security and operational guardrails, and log aggregation. Although Control Tower aggregates logs for native log data generated by AWS CloudTrail and AWS Config, it does not provide an out of the box solution for application logging.

To help build out and maintain Sixth Street’s centralized logging stack, Sixth Street enlisted the cloud-native log analysis service, ChaosSearch. “We were looking for a cost-effective centralized logging solution that could provide us with the effectiveness and flexibility of an Elasticsearch, Logstash and Kibana (ELK) stack without the additional overhead of maintaining such a solution in house,” said Dutko.

To aggregate the application logs, Logicworks and ChaosSearch collaborated and got creative. Logicworks built a solution to aggregate and centralize those logs, but Sixth Street still needed a process to query and analyze log data. Additionally, they needed a solution for monitoring and alerting on aggregate log data. ChaosSearch provided a platform for indexing, searching, and analyzing log data in a scalable and efficient manner.

Solution

The Control Tower environment built by Logicworks for Sixth Street spans three regions. It is segmented across accounts allocated by business unit and the Software Development LifeCycle (SDLC) environments of development, test, and production.

The standard Control Tower deployment automatically centralizes CloudTrail and Config logs from all accounts and all regions into the log archive account. Logicworks went a step further by deciding to consolidate all application logs in the central log archive account for consumption.

Sixth Street development teams writing cloud-native serverless applications using AWS Lambda could now send their logs to a central account. However, a significant amount of plumbing was required to not only get logs from their source account into the log archive account but also to enrich the data with invaluable metadata and prepare for ingestion into ChaosSearch.

Sixth Street desired to unburden both their development and infrastructure teams regarding set up for plumbing for each application. By automating the log collection and centralization process, Sixth Street developers were freed up to do what they do best - develop world class business critical applications.

The requirements provided to the Logicworks and ChaosSearch teams allowed developers to seamlessly consume logs in the ChaosSearch platform from their serverless applications built in AWS. Once deployed, a Lambda Function’s logs should flow into the ChaosSearch platform with no involvement from either the developers or the Sixth Street infrastructure team.

To accomplish this, Logicworks and ChaosSearch developed a serverless automated workflow to move logs from a CloudWatch Log Group through to a centralized S3 bucket where a cross account IAM role for ChaosSearch could ingest the logs into the ChaosSearch platform.

The overall log centralization solution made use of EventBridge, CloudWatch Log Groups, Lambda functions, Kinesis Data Firehose, and S3. The only requirement for developers was to tag their Lambda CloudWatch Log Groups on creation or after creation when logging needed to be activated.

Once a Lambda function was deployed with the appropriate tag and began generating logs, it only took a few minutes for the enriched log data to flow into ChaosSearch, and no further human interaction was required.

Results

As a result, the streamlined centralized logging solution saved Sixth Street five hours, per engineer, per week, as well as additional headcount costs, enabling the engineers to focus their efforts on more revenue generating projects. “Logicworks provided an invaluable service when it came to architecting our original deployment to AWS. The architecture guidance helped us shape our account structure for maximum flexibility and maintainability,” said Dutko. “Logicworks continues to work with us through bespoke projects, ad-hoc support requests and professional interactions between our other valued partners like ChaosSearch,” he added.