Shard size: AWS recommends shard size between 30-50GB as optimal size for shards. However, for PB-scale cluster this limit is increased to 100GB. For our estimation, we are considering the following shard sizes based on total storage for the cluster:
vCPUs per Active Shard: 1 vCPU per shard, as recommended by AWS for large scale clusters.
Dedicated master nodes: 3
Availability zones: 3
Reserved instance: Considering 1yr/reserved instances for all nodes (data and master) and average discount of 31% on instance price.
Retention: all data retained is considered hot (no ultrawarm). This makes a fair comparison with ChaosSearch where all data stays hot all the time, no matter the size.
Replicas: 1 replica considered. Although using more replicas increase reliability, it becomes cost-prohibitive for large scale deployments.
Staffing: High-skilled, experienced professionals are required to plan, deploy, monitor and troubleshoot Elasticsearch at scale. For this estimation we are considering $130,000/yr for each FTE responsible for managing the cluster. The number of FTEs necessary is estimated based on the size of the cluster, as follows:
Training: Cost of ongoing internal user training and support per year : $5,000 (not required for ChaosSearch)
Cluster deployment/setup effort: For this estimation we are considering the cost related to the design, plan and deployment of a brand new cluster. Cost is calculated by the estimated number of hours (check below) multiplied by the hourly cost of the FTE ($130,000/yr):
Support: The following costs were considered for support:
While retention, ingestion and overhead are all basic inputs for a cost estimation, properly sizing an Elasticsearch cluster involves other parameters that can dramatically influence the total cost, especially when dealing with large clusters.
To get a personalized cost projection based on your current technical and business requirements, schedule a 30-min session with one of our experts to walk you through our full Elasticsearch cost calculator, and get a complete breakdown.