The Hidden Costs of Your ELK Stack [VIDEO]
At first glance, there may seem to be little not to love about the ELK Stack. It’s open source. It’s free (if you set it up and manage it yourself, at least). It’s a widely used solution with a thriving ecosystem surrounding it.
But if you’ve ever actually built and managed an ELK stack environment, you have probably found that the theory doesn’t match the reality. The ELK stack is full of hidden costs, and it often fails to deliver real value over the long term.
In fact, the ELK stack can fall so short of its promises that Thomas Hazel, CTO at ChaosSearch, and Andrew Kenney, VP of Engineering at Fairmarkit, recently dedicated an entire webinar to the topic of the ELK stack’s hidden costs - A New Vision for Log Analytics. Here’s an overview of what they had to say about problems with the ELK stack and alternative approaches to ELK.
ELK Stack Benefits: The Theory
On the surface, the ELK stack offers several benefits that make it a great data analytics solution for businesses:
- Scalability: For businesses that expect their data to grow steadily over time, the ELK stack appears to offer the scalability they need over the long term.
- Flexibility: The ELK stack can ingest and analyze virtually any type of data, which appeals to companies that need to support a variety of data analytics use cases.
- Open source: The technologies that power the ELK stack are open source. That’s attractive for organizations that interpret open source to mean “lock-in free” and “low cost.”
- Easy to deploy: The fact that you can build your own ELK stack using open source components makes it appealing for companies that want a low-risk solution for data analytics.
These, at least, are the theoretical promises behind ELK stack.
ELK Stack Drawbacks: The Reality
The reality of ELK stack is more complex. The platform is subject to several key limitations that undercut the value it theoretically provides.
Scaling is Harder than It Looks
Although in theory your ELK environment can grow seamlessly along with your data, the reality is that scaling ELK usually requires provisioning more infrastructure. You will also need to update your configuration in many cases in order to accommodate new types of data and new analytics needs.
In practice, then, the ELK stack hardly provides the set-it-and-forget-it scalability that businesses need to ensure their data analytics solution keeps pace with their data analytics needs.
High Management Costs
The ELK stack has a free and open source core, but that hardly means it’s free to use. Implementing an ELK environment yourself using the vanilla open source code requires a lot of engineering expertise, which translates to a lot of money.
If, alternatively, you use a managed ELK service, like those offered by public cloud vendors, you have to pay for the software and infrastructure that comes with it.
In short, the ELK stack is one of those open source solutions that can easily break the bank, despite a nominal price tag that reads “free.”
Centralization is Hard to Achieve
The ELK stack may be capable in theory of analyzing any kind of data from any kind of source.
But in practice, different types of logs or other data sources often require different configurations. They may also need different ingestion processes.
What this means is that it can be very difficult to build a truly centralized ELK environment that is capable of supporting all use cases. You are likely to end up with multiple environments -- one for, say, business intelligence and another for application performance analytics.
And, of course, more environments only increase the scalability, management and cost challenges described above.
Are ELK Costs Worth It?
For most businesses and use cases, then, the ELK stack ends up costing much more money and delivering much less value than it might seem. Unless you have narrow and specific data analytics needs, and you don’t expect your data to increase significantly in volume and type over time, you are likely to find that you spend much more on the ELK than you get out of it.
This isn’t because of any fundamental problems in ELK, by the way. The idea of combining a data lake and data analytics into a single platform, which is one of the defining features of the ELK stack, is a powerful one. It’s the management difficulties and the fragmentation between different ELK implementations that make ELK a less effective solution than many businesses envision.
An Alternative to ELK Stack
To translate the theoretical benefits of a solution like ELK Stack into reality, you need a platform that combines a data lake with data analytics, but that removes the complexity, scalability and cost challenges.
Read the Series
Part 1A: Data Lake Challenges: Or, Why Your Data Lake Isn’t Working Out
Part 1B: Data Lake Opportunities: Rethinking Data Analytics Optimization
Part 2: AWS Monitoring Challenges: How to Approach AWS Management
Part 3: The Hidden Costs of Your ELK Stack