Best Practices for Effective Log Management

Can following log management best practices help digital retailers drive more sales and revenue for this upcoming Cyber Monday?

Absolutely.

Cyber Monday is like the Super Bowl of eCommerce, with digital retailers projected to bring in nearly $12 billion on the busiest virtual shopping day of the year.

And for retailers hoping to maximize their Cyber Monday revenue, following log management best practices can provide significant competitive advantages when it comes to streamlining the digital checkout process, optimizing customer experiences, troubleshooting errors that impact revenue, and managing spikes in website traffic.

In this week’s blog, you’ll discover eight log management best practices that can help digital retailers optimize customer experiences and capitalize on the full revenue potential of Cyber Monday.

8 Log Management Best Practices

1. Implement Structured Logging
2. Build Meaning and Context into Log Messages
3. Avoid Logging Non-essential or Sensitive Data
4. Capture Logs from Diverse Sources
5. Aggregate and Centralize Your Log Data
6. Index Logs for Querying and Analytics
7. Configure Real-Time Log Monitoring and Alerts
8. Optimize Your Log Retention Policy

1. Implement Structured Logging

The traditional way of logging is to write event logs as plain text into a log file. The problem with this method is that plain text logs are an unstructured data format, which means they can’t easily be filtered or queried to extract insights.

As an alternative to traditional logging, digital retailers should implement structured logging and write their logs in a format like JSON or XML that’s easier to parse, analyze and query. Logs written in JSON are easily read-able by both humans and machines, and structured JSON logs are easily tabularized to enable filtering and queries.

Structured logging saves time, accelerates insight development, and helps digital retailers maximize the value of their log data as they optimize eCommerce properties for Cyber Monday.

2. Build Meaning and Context into Log Messages

Log messages should include meaningful information about the event that triggered the log, as well as additional context that can help analysts understand what happened, find correlations with other events, and diagnose potential issues that require further investigation.

Meaningful logs are descriptive and detailed, providing DevSecOps teams with useful information that can help streamline the diagnostic process when an error occurs.

Valuable context for log messages can include fields like:

• Timestamps - Knowing the exact date and time that an event occurred allows analysts to filter and query for other events that happened in the same time frame.
• User Request Identifiers - Requests from client browsers to the eCommerce web server have a unique identifier code that may be included in logs for events triggered by the request.
• Unique Identifiers - Digital retailers assign unique identifiers for individual users, products, user sessions, pages, shopping carts, and more. These data points can be written into event logs, providing valuable context and insight into the state of the application when the event occurred.

3. Avoid Logging Non-essential or Sensitive Data

Deciding what to include in log messages is just as important as determining what can be left out. Logging non-essential information that doesn’t help with diagnostics or root cause analysis results in increased time-to-insights, data volumes, and higher costs.

It’s also important to avoid logging sensitive data, especially proprietary data, application source codes, and personally identifiable information (PII) that may be covered by data privacy and security regulations or standards like the European GDPR, HIPAA, or PCI DSS.

Digital retailers can optimize customer experiences in preparation for Cyber Monday by logging data from individual user sessions, but instead of logging the user’s name and email with each event, we recommend assigning each User/Session a unique identifier that conceals their identity while still enabling analysts to effectively correlate events by session or user.

4. Capture Logs from Diverse Sources

As IT environments grow in complexity, DevOps teams have the potential to capture logs from tens or even hundreds of different sources. And while not all of these logs may be deemed essential, capturing the right logs can provide meaningful data and valuable context when it comes to detecting and diagnosing errors.

Digital retailers should think about capturing logs from:

• Infrastructure Devices - Logs from switches, routers, and network access points can help digital retailers diagnose misconfiguration issues that might be causing slow-downs for their customers.
• Security Devices - Security log analytics is essential during the Cyber Monday traffic spike. Logs from firewalls and intrusion detection systems enable SecOps teams to quickly detect and respond to security concerns before they result in costly unplanned downtime.
• Web Servers - Web server logs are essential for capturing information about how users interact with digital retail properties. They can help both DevOps and marketing teams optimize the customer experience by understanding when users visit the site, where they come from, and the actions they take upon arrival.
• Applications - Logs from payment gateways, analytics tools, databases, and mobile shopping apps can help DevOps teams pinpoint errors for rapid resolution.
• Cloud Infrastructure - The logs generated by cloud infrastructure and services can help DevOps teams gain insight into cloud service availability and performance, resource allocation, and latency issues.

When it comes to optimizing Cyber Monday results, digital retailers should focus their logging efforts on operations that are closely tied to revenue and customer experience, including the shopping cart, checkout process, email registration system, and authentication.

5. Aggregate and Centralize Your Log Data

Log data is generated at many different points in the IT infrastructure, but it must be aggregated in a centralized location before it can be used effectively for data analysis.

As IT systems generate logs, your log aggregator tool (e.g. Logstash, Graylog, etc.) should automatically ingest those logs and ship them out of the production environment and into a centralized location (e.g. public cloud storage, or a log management tool).

Aggregating and centralizing log data gives eCommerce developer teams the ability to investigate security or application performance issues without having to manually extract, organize, and prepare log data from potentially hundreds of different sources.

Read: Centralized Log Management and APM/Observability for Application Troubleshooting and DevOps Efficiency

6. Index Logs for Querying and Analytics

As enterprise IT environments increase in complexity, they generate massive volumes of log data that can take a long time to query. Indexing your logs creates a new data representation that’s optimized for query efficiency, enabling enterprise DevOps and data teams to more readily solve problems and extract value from their logs.

DevOps teams may choose log indexing engines like Elasticsearch or Apache Solr to index their logs, but these engines may encounter performance issues or data retention trade-offs when analyzing logs at scale.

Shameless plug: ChaosSearch’s proprietary Chaos Index® technology indexes logs directly in Amazon S3 with up to 95% data compression, enabling text, relational, and ML queries that help digital retailers get the most value from their log data.

Read: New Report Shares Best Practices for Modern Enterprise Data Management in Multi-Cloud World

7. Configure Real-Time Log Monitoring and Alerts

When the stakes are high, issues in the production environment need to be discovered and addressed right away. That’s never more true than on Cyber Monday, when even a few minutes of unplanned service interruption can result in thousands of dollars in lost revenue.

Read: 10 DevOps Tools for Continuous Monitoring

DevSecOps teams can configure their log management systems or SIEM tools to monitor the stream of ingested logs and alert on known errors or anomalous events that could signal a security incident or application performance issue.

Alerts can be routed directly to the mobile phones and/or Slack accounts of incident response teams, enabling rapid detection, diagnosis, and resolution of errors, and minimizing their impact on the customer journey.

8. Optimize Your Log Retention Policy

Enterprises should set different retention policies for different types of logs, depending on their unique needs and circumstances.

In some cases, preserving logs for the long-term is required to comply with local data protection regulations. You may also want to retain certain logs past the standard 90-day retention period to support long-term analysis of application performance or user behaviors.

Digital retailers can use historical logs and trend data to anticipate traffic spikes that take place on Cyber Monday, forecast the number of expected shoppers, and optimize their architecture, systems, and staffing to deliver the best possible customer experience during peak demand periods.

Future-proof Your Log Management Strategy

Hopefully these eight tips will help you plan for log data spikes as we head into Black Friday or other peak times for your industry. And as you think through a long-term strategy for log analytics, consider partnering with ChaosSearch to give your SRE team peace of mind.

The ChaosSearch cloud data platform enables log analytics at scale, with less toil and at lower cost, while taking advantage of all the reliability and security that comes with the cloud.

ChaosSearch indexes logs directly in your Amazon S3 or Google Cloud Storage buckets, preserving every detail of your log data with up to 95% compression, no data movement, and low cost of ownership. The platform enables multi-API data access, making your logs available for text search, relational (SQL) analytics, and machine learning queries using the tools your team already knows and loves (for instance, Kibana).

Digital retailers and FinTech companies like Agilence and Digital River use ChaosSearch to detect and investigate errors that impact the customer journey, forecast peak demand times using historical log data, and analyze user session logs to improve overall customer experience so they’re prepared to maximize Cyber Monday revenue. You’re welcome to give the platform a try to see how ChaosSearch can help you future-proof your business.

Additional Resources

Read the Blog: 2021 Benchmark Report | Log Management and Analytics

Watch the Webinar: Why and How Log Analytics Makes Cloud Operations Smarter

theCube Interview: Digital River talks their Modern, Cloud-Native Approach to Analytics