DevOps has become the dominant application development and delivery methodology today, embraced over traditional software development methods by teams striving for lightning-fast innovation and more frequent releases without compromising on quality, stability, or productivity.
As DevOps teams work to build, improve, and automate the release pipeline, specialized software tools are used to monitor application performance, troubleshoot software bugs, and evaluate user experience across staging, testing, development, and production environments. These tools provide crucial feedback and insights that drive future product iterations and support business decision-making. They deliver features and functionality that cover three capabilities: observability, monitoring, and analysis.
- Observability - Making data from within the application available and accessible for monitoring and analysis.
- Monitoring - Collecting, aggregating, and displaying application data for consumption by DevOps teams.
- Analysis - Manually or automatically investigating application data to extract insights that support a variety of DevOps use cases.
This blog post explores how the combination of two kinds of software tools -- Application Performance Monitoring (APM) and Centralized Log Management (CLM) -- can deliver DevOps capabilities covering all three bases (observability, monitoring, and analytics). You’ll discover the strengths and limitations of APM and CLM software, and how they complement each other to satisfy DevOps use cases that range from application monitoring and troubleshooting to real user monitoring, distributed tracing, and root cause analysis.
What is an APM Tool?
APM software solutions allow DevOps teams to track key application performance metrics by capturing telemetry and log data from applications in every stage of the DevOps release pipeline.
APM tools ingest logs, traces, and metrics from an application and present a near-real-time view of application behavior in feature-rich dashboards that can be reviewed by DevOps teams to ensure application uptime, enhance performance, and optimize the user experience.
The defining capabilities of APM software solutions are:
- Application Discovery - DevOps teams can use APM tools to automatically discover and map an application and its infrastructure components, ensuring real-time observability and awareness in complex IT environments.
- Telemetry - Telemetry is the practice of collecting data and measurements from applications or IT infrastructure and transmitting that information to a centralized monitoring platform. DevOps teams use telemetry to gather data on applications and IT infrastructure in the form of logs, metrics, and traces. APM tools offer several options for adding telemetry instrumentation and increasing observability of the DevOps environment.
- Application and Infrastructure Monitoring - Installing telemetry instrumentation allows DevOps teams to monitor predefined and custom metrics, log and event data, transaction traces, SQL queries, and stack-trace details. Telemetry from applications, cloud services, and IT infrastructure is sent to the APM software where it can be queried, reviewed, and analyzed by machine or human users. Telemetry data can also trigger alerts, notifying DevOps teams in real time when an error or anomaly is detected.
- Digital Experience Monitoring - APM software solutions are used by DevOps teams to facilitate both synthetic and real user experience monitoring. While synthetic monitoring uses automation scripts that simulate real user behaviors, Real User Monitoring (RUM) allows DevOps teams to understand how live users are accessing and utilizing the application.
- Reporting and Analytics - APM tools present data from logs, metrics, and traces in reports and visual dashboards that can be reviewed and analyzed by DevOps teams.
APM software solutions give DevOps teams the ability to measure the performance of application services, dependencies (databases, web services, caching, etc.), requests, and transactions, using telemetry data from installed agents that collect metrics and traces.
But despite their ability to monitor application performance and user experience, APM solutions aren’t the only tool that DevOps teams need to achieve comprehensive application observability, monitoring, and analytics.
Where are the Gaps in APM Software Solutions?
As containerized microservice architectures, such as Kubernetes (also known as K8s), have rapidly become commonplace, log data volumes have exploded and APM tools have been exposed for their deficiencies in delivering log analytics at scale. Here’s where DevOps teams are identifying gaps in their APM tools:
- Limited Data Retention - APM tools typically cap log data retention. Since APM tools are real-time or near-real-time operational solutions, they are designed for fast analysis of recent (< 30 days old) and smaller data sets. If a DevOps team wants longer retention, it’s common for APM tools to cap retention at 12 to 13 months. While DevOps teams can use APM tools for short-term analysis of operational data, limiting data retention creates analytical blind spots that negatively impact use cases like root cause and long-term trend analysis.
- High Storage Costs - APM tools often add metadata and parse log data into record fields to support analytics initiatives. This practice inflates the aggregate size of log data, leading to higher data storage costs. DevOps teams are ultimately forced to choose between shouldering these inflated costs or reducing them by decreasing log retention.
- Data Movement + Multiple Data Copies = Increased Costs - APM tools often require DevOps teams to capture and aggregate logs in cloud storage before sending it to a SaaS APM tool. This results in data storage and egress costs from the cloud service provider, plus data ingest and storage costs from the APM SaaS provider. As log volumes grow, these costs become prohibitive which eventually leads to data retention trade-offs.
- Data Privacy and Security Compliance - DevOps teams that are required to retain ownership and control of sensitive data can encounter compliance challenges that result from moving data between cloud storage and a SaaS APM platform.
APM tools provide the instrumentation (installed agents) that creates application observability and allows DevOps teams to capture metrics and traces for application troubleshooting and other use cases.
But when it comes to log management and analytics, APM tools simply aren’t optimized to efficiently aggregate, query, and analyze the massive volume of logs generated in today’s complex application environments.
To cover these gaps in log coverage, and better support use cases like root cause and long-term trend analysis, DevOps teams can implement a log analytics solution that supports cost-effective storage, querying, and analysis of log data.
READ: How to Move Kubernetes Logs to S3 with Logstash
What is Centralized Log Management for DevOps?
Logs are computer-generated documentation of events that happen within an application, host, or system. While metrics capture just a few data points, logs are created by every event and provide more comprehensive insights into the state of an application at any given time. As a result, logs are extremely useful for troubleshooting application issues.
DevOps teams can implement a CLM platform to collect, aggregate, analyze, and visualize log data from their application, services, and throughout the IT environment. Modern CLM platforms allow for in-depth querying and analysis with unlimited data retention and no data movement, extending application error detection and troubleshooting capabilities while minimizing costs.
The defining capabilities of CLM solutions include:
- Collecting and Aggregating Log Data - CLM tools collect and centralize log data from multiple sources so it can be analyzed together. Sources include the application and its related services, along with operating systems, network infrastructure, and endpoint devices.
- Normalizing Log Data - Log data is computer-generated and may be written in different formats depending on the source. CLM software tools automate the process of normalizing log data so it can be indexed in a unified format for querying and analytics.
- Indexing Log Data - CLM software solutions index aggregated and normalized log data, reducing its storage footprint and making it faster and more efficient for querying.
- Querying and Analytics on Log Data - CLM software allows DevOps teams to analyze log data and run queries to debug services, troubleshoot application errors, and determine the root cause of application performance issues.
- Visual Reports and Dashboarding - CLM software provides visualization and dashboarding capabilities that make it easier for DevOps teams to consume the data and assess application performance in near real time.
- No Data Movement or Duplication - Modern CLM platforms are available as a SaaS product that activates log data in cloud storage, enabling full analytics capabilities with no data movement, no egress/ingress, and no need for storing duplicate data.
While APM solutions are optimized for storing telemetry data like metrics and traces, CLM solutions are optimized for cost-efficient storage and querying of log data at scale. When deployed together, these tools optimize log coverage and offer comprehensive observability, monitoring, and analytics across the entire DevOps pipeline.
LEARN: ChaosSearch for DevOps Analysis
CLM vs. APM Use Cases for DevOps
APM and CLM solutions are complementary solutions for application troubleshooting and improving DevOps efficiency. Below, we highlight the specific benefits and applications of both APM and CLM software tools across five application troubleshooting use cases.
Application Observability
Application observability is the ability to make data from within an application accessible for monitoring and analysis.
- APM tools create application observability by installing agents that capture metrics and traces from within the application. Some APM tools also ingest logs, but in most cases, they are not cost-optimized to ingest, retain, and query large volumes of log data.
- CLM software captures event logs from the application, its related services and dependencies, as well as network endpoints, infrastructure components, and other sources in the IT environment. With unlimited data retention, DevOps team can store large volumes of log data to facilitate in-depth observability of applications over time.
Application Monitoring
Application component monitoring is the collection of application logs, metrics, and traces that can yield insights into application performance.
- APM tools monitor application components by aggregating metrics and traces from the application environment. Metrics provide insight into measurable dimensions of application performance, such as throughput and average response times, while distributed tracing and transaction performance monitoring provide insight into transactional behaviors between application services.
- CLM software monitors applications by aggregating log and event data from throughout the DevOps environment. Logs provide a timestamped record of every event that takes place within the application, enabling DevOps teams to fully monitor the application environment and related infrastructure.
Application Troubleshooting and Root Cause Analysis
When a request fails, a transaction takes too long, or an application returns an error, DevOps teams can use APM and Log Analytics tools to troubleshoot the application, diagnose the issue, and implement a solution.
- APM tools support troubleshooting by aggregating metrics, traces, and logs from the production environment. Metrics provide insight into the performance of applications, services, and system components, while traces provide insight into the health of transactions between application services, and logs provide records of application events.
- CLM software provides detailed records of events that take place within the application and throughout the IT environment. While metrics and traces help DevOps teams determine what is broken or where an error is happening, logs reveal a deeper level of detail that can help engineers troubleshoot and debug their applications.
CLM software is optimized for long-term retention of indexed log data, enabling DevOps teams to keep using their older logs when needed for application troubleshooting, root cause analysis, and forensic investigations.
LIVE WORKSHOP: Unleash Analytics on Your Cloud Object Storage
Digital Experience Monitoring
While application troubleshooting focuses on measuring performance and diagnosing errors in back-end systems, digital experience monitoring focuses on monitoring interactions between front-end client interfaces and either real or simulated users. Digital experience monitoring allows DevOps teams to better understand the impact of application updates on the customer experience.
- APM tools frequently include digital experience monitoring as a core capability. They can use individual session traces to follow the path of a request from user initiation, through the application’s front-end, and into the back-end architecture while evaluating the success of every transaction along the way.
- CLM software can also be used for real user monitoring, empowering DevOps teams with insights into how many visitors are using the app, where they’re spending the most time, and where they’re encountering friction or experiencing bottlenecks. When deployed together, APM tools and CLM software provide a comprehensive approach to monitoring the digital experience.
Summary
At scale, a DevOps team will optimize their log coverage, application performance, and user experience by deploying both APM/observability and centralized log management.
APM tools are optimized for real-time and near real-time operational data. They feature sub-second query responses and present application performance data in simple UIs and dashboards. However, they are not designed for storing, querying, and analyzing large volumes of log data.
CLM complements APM and observability tools by economically increasing and optimizing log coverage. Increased log coverage empowers DevOps teams to fully monitor the production environment while eliminating log data egress/ingress costs, supporting long-term investigations and root cause analysis, and retaining possession and control of data for compliance purposes.
The two solutions, APM and CLM, work well together to provide the most complete visibility into application performance and the long-term health of applications, infrastructure, and user experience.
Learn how ChaosSearch can deliver a centralized log management solution that delivers massive time, cost and complexity savings over the Elasticsearch stack, while complementing your APM tool.
