Logs are one of the three key “pillars” of observability, and cloud environments are no exception. You can’t know what’s happening in your cloud without analyzing cloud service logs, which allow you to audit and monitor workflows within your cloud.
That said, cloud logging is a unique beast in certain respects. Cloud service and application logs may not be fundamentally different in form from other types of logs, but cloud log management requires a special approach, due to the unique challenges that arise when accessing, securing and managing logs within a cloud environment.
This article discusses those challenges. It also explains how to address them in order to ensure you have complete observability into your cloud environment no matter which type of cloud architecture you use (single-cloud, multi-cloud, or hybrid cloud) or which workloads you run in it.
Why cloud logging is different
Again, cloud service logs typically don’t look very different from a log that you’d see in any other type of environment. For example, a basic log from Amazon’s EC2 service might look like this:
{"Records": [{
"eventVersion": "1.0",
"userIdentity": {
"type": "IAMUser",
"principalId": "EX_PRINCIPAL_ID",
"arn": "arn:aws:iam::123456789012:user/Alice",
"accessKeyId": "EXAMPLE_KEY_ID",
"accountId": "123456789012",
"userName": "Alice"
},
"eventTime": "2014-03-06T21:22:54Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "StartInstances",
"awsRegion": "us-east-2",
"sourceIPAddress": "205.251.233.176",
"userAgent": "ec2-api-tools 1.6.12.2",
"requestParameters": {"instancesSet": {"items": [{"instanceId": "i-ebeaf9e2"}]}},
"responseElements": {"instancesSet": {"items": [{
"instanceId": "i-ebeaf9e2",
"currentState": {
"code": 0,
"name": "pending"
},
"previousState": {
"code": 80,
"name": "stopped"
}
}]}}
}]}
As you can see, this is a JSON-formatted log file (which is borrowed from Amazon’s documentation) that tells us that an IAM user named Alice started an EC2 instance called i-ebeaf9e2. Apart from the JSON encoding, this log isn’t that different from a syslog file that you might collect on your Linux server, for example, or from a log file for an on-prem application.
READ: The New Best Way to Index and Query JSON Logs
However, if you think beyond the contents and structure of the log, you’ll realize that cloud logs are different from other types of logs in several key respects:
- Generation: In some cases, cloud services don’t generate logs at all unless you explicitly configure them to do so. For example, with Amazon EC2, you need to set up a logging agent before you can start collecting EC2 logs.
- Access: You typically can’t access the log file for a cloud service by simply tailing it or opening it with a local tool like journalctl. Instead, you have to collect the log with an aggregation tool or observability platform that is capable of integrating with your cloud services and extracting the log files they produce.
- Security: Depending on how your cloud service provider configures its logging services, logs may or may not be stored securely by default. For instance, on Amazon S3, log files are stored by default in S3 buckets. Those buckets could be configured in such a way that anyone on the Internet could view your log files – which is generally a very bad idea.
- Log storage and retention: Unlike most local log files, some logs for cloud services and applications are not stored permanently by default. Logs for containers will disappear when the container shuts down, for example, and Kubernetes in most cases will start overwriting log data once your log files exceed 10 megabytes in size unless you configure different log management behavior.
The above is to say that cloud logs themselves are not different from any other type of log, but cloud log management needs to be different to accommodate the unique challenges with regard to how cloud logs are generated, stored and managed.
READ: Managing the Mess of Modern IT: Log Analytics and Operations Engineering
Simplifying cloud logging with a third-party observability platform
There are two basic ways to simplify the complexity that comes with cloud logging.
One is to use the tooling that your cloud provider offers for aggregating and centralizing logs. For example, on AWS, you can use Amazon CloudWatch and CloudTrail to collect most of the logs generated by AWS cloud services.
The main downside of this approach is that it only works within the AWS cloud. You can’t use CloudWatch and CloudTrail to manage logs from other clouds or from on-prem applications or infrastructure, which is a major challenge if you operate a multicloud or hybrid cloud environment. In addition, CloudWatch and CloudTrail offer only basic analytics functionality. They can surface some insights about your cloud applications’ performance and health, but they generally won’t pinpoint the root cause of complicated performance problems.
That’s why a better solution for cloud log management in many cases is to use a third-party log analytics and observability platform to collect your logs and run analytics on them. Third-party platforms offer the ability to work across multiple clouds (and within a hybrid cloud environment). At the same time, they provide more sophisticated log analytics features than most cloud vendors’ own analytics tools.
Depending on which third-party log analytics platform you use, you may still choose to leverage a service like CloudWatch to build a pipeline that sends cloud service logs to the platform. Or, you may be able to collect the logs directly from the location where they are stored, like S3.
Either way, your goal should be to centralize log management and analytics across all of your cloud services and cloud environments. When you do that, you get a cloud logging process that is simpler, more secure, more comprehensive and more consistent than one that requires you to manage logs for each cloud service separately. In turn, you gain deep visibility into your cloud environment – which means you can discover what’s hidden in your data lake, no matter how complex your data lake is or which log files are floating within it.
READ: Going Beyond CloudWatch: 5 Steps to Better Log Analytics & Analysis
Conclusion
Cloud log management is inherently more challenging in some respects than is managing logs for on-prem applications or infrastructure. But with the right approach and tools – which include, above all, a cloud log analytics and observability platform that can work with any and all cloud logs – you can effectively manage the various logs that your cloud services produce, no matter how complex your cloud architecture is.
Additional Resources
Read the Blog: Troubleshooting Cloud Services and Infrastructure with Log Analytics
Listen to the Podcast: Differentiate or Drown: Managing Modern-Day Data
Check out the Report: 2022 Cloud Data & Analytics Survey Report
