New Video Series: The 7 Challenges of Big Data Analytics with Thomas Hazel... Watch Now ->
New Video Series: The 7 Challenges of Big Data Analytics with Thomas Hazel... Watch Now ->
Start Free Trial

Elasticsearch Replacement for Log Analytics at Scale

TCO Calculator Widget - Assumptions

Elasticsearch configurations

Shard size: AWS recommends shard size between 30-50GB as optimal size for shards. However, for PB-scale cluster this limit is increased to 100GB. For our estimation, we are considering the following shard sizes based on total storage for the cluster:

  • Up to 200TB of total storage -> Shard size: 30GB
  • From 200TB to 500TB of total storage -> shard size: 50GB
  • Over 500TB of total storage -> shard size: 100GB

vCPUs per Active Shard: 1 vCPU per shard, as recommended by AWS for large scale clusters.

Dedicated master nodes: 3

Availability zones: 3

Reserved instance: Considering 1yr/reserved instances for all nodes (data and master) and average discount of 31% on instance price.

Retention: all data retained is considered hot (no ultrawarm). This makes a fair comparison with ChaosSearch where all data stays hot all the time, no matter the size.

Replicas: 1 replica considered. Although using more replicas increase reliability, it becomes cost-prohibitive for large scale deployments.

 

Operations and Support

Staffing: High-skilled, experienced professionals are required to plan, deploy, monitor and troubleshoot Elasticsearch at scale. For this estimation we are considering $130,000/yr for each FTE responsible for managing the cluster. The number of FTEs necessary is estimated based on the size of the cluster, as follows:

  • Self-hosted Elasticsearch on AWS: 1xFTE per 5TB/day ingested
  • Amazon OpenSearch: 1xFTE per 10TB/day ingested
  • ChaosSearch: True SaaS solution. Fully managed by ChaosSearch.

Training: Cost of ongoing internal user training and support per year : $5,000 (not required for ChaosSearch)

Cluster deployment/setup effort: For this estimation we are considering the cost related to the design, plan and deployment of a brand new cluster. Cost is calculated by the estimated number of hours (check below) multiplied by the hourly cost of the FTE ($130,000/yr):

  • Self-hosted Elasticsearch on AWS: 60hrs
  • Amazon OpenSearch: 40hrs
  • ChaosSearch: 8hrs

Support: The following costs were considered for support:

  • Self-hosted Elasticsearch on AWS and Amazon OpenSearch: AWS Enterprise support (3% of total cost)
  • ChaosSearch: Support is included on the license

Want to dig deeper?

While retention, ingestion and overhead are all basic inputs for a cost estimation, properly sizing an Elasticsearch cluster involves other parameters that can dramatically influence the total cost, especially when dealing with large clusters.

To get a personalized cost projection based on your current technical and business requirements, schedule a 30-min session with one of our experts to walk you through our full Elasticsearch cost calculator, and get a complete breakdown.

Schedule Now!