--- title: Thank You - Reaching the Elasticsearch Wall Webinar description: Watch this recorded webinar to learn what organizations can do when they hit the ELK Stack wall using ChaosSearch, an advanced technology platform --- Revinate leaves their ELK stack behind to find huge gains with ChaosSearch -- Read More! [Revinate leaves their ELK stack behind to find huge gains with ChaosSearch -- Read More! ](https://www.chaossearch.io/resources/customer-stories/revinate) [![ChaosSearch](https://www.chaossearch.io/hubfs/2021%20Website/logo.svg) ](https://www.chaossearch.io/) ![Gartner Cool Vendor 2023](https://www.chaossearch.io/hubfs/C2020/Logos/Gartner%20Cool%20Vendor%202023.png) [![Start Free Trial](https://no-cache.hubspot.com/cta/default/4020721/a1eeee12-14a2-41d7-a38b-11e32980b916.png)](https://cta-redirect.hubspot.com/cta/redirect/4020721/a1eeee12-14a2-41d7-a38b-11e32980b916) # Reaching the Elasticsearch Wall ## Thank You The webinar recording is available for viewing below. Enjoy!   ## Interested in scheduling a brief intro call to see how ChaosSearch can accelerate your analytics? Yes - show me the calendar! ## Future-Proof Your Analytics at Scale [![Get a Demo](https://no-cache.hubspot.com/cta/default/4020721/0ae4b1b0-9ec0-4169-80cc-64fda5fd56db.png)](https://cta-redirect.hubspot.com/cta/redirect/4020721/0ae4b1b0-9ec0-4169-80cc-64fda5fd56db) ©2024, ChaosSearch®, Inc. [Legal](https://www.chaossearch.io/legal) Elasticsearch, Logstash, and Kibana are trademarks of Elasticsearch B.V., registered in the U.S. and in other countries. Elasticsearch B.V. and ChaosSearch®, Inc., are not affiliated. Equifax is a registered trademark of Equifax, Inc. Contact Us Phone: [(800) 216-0202](tel:+8002160202) Email: [teamchaos@chaossearch.io](mailto:teamchaos@chaossearch.io) Follow Us - - - - ```json { "@context" : "https://schema.org", "@type" : "VideoObject", "description" : "Reaching the Elasticsearch Wall: When Expanding Log and Event Data Threatens to Topple Your ELK Strategy", "name" : "Reaching the Elasticsearch Wall: When Expanding Log and Event Data Threatens to Topple Your ELK Strategy", "thumbnailUrl" : "https://i.vimeocdn.com/video/827111528_100x75.jpg?r=pad", "transcript" : "DAVID BUNTING: Hello, everyone, and thanks so much for joining us today. We really appreciate it. I'm David Bunting with ChaosSearch, and I'd like to welcome you to the AWSInsider webcast entitled \"Reaching the Elasticsearch Wall-- When Expanding Log and Event Data Threatens to Topple Your ELK Strategy.\" So let's jump right in. First, just a few quick housekeeping items. If you're having any technical issues, audio or otherwise, please post a question in the Q&A panel, and someone from the AWSInsider team will assist you. We expect today's presentation to last approximately 30 minutes. The session will be recorded, and you'll receive an email with a link to the recording. Finally, we will be taking audience questions throughout the webcast as well as at the end, during the Q&A session. Please use the Q&A panel to submit your questions at any time. For any that we don't get to during the webcast, we'll be sure to follow up afterwards via email. Now I'd like to have you meet today's presenters, and I'm actually going to let them introduce themselves. Greg, can you say hello to the crowd? GREG SCHULZ: Thanks, David. Hello, everybody. This is Greg Schulz, Founder of independent IT analyst and consultancy firm Server StorageIO. I've worked as the customer in various IT organizations in different roles as well as a vendor, consulting analyst, and author of several books, including Software-Defined Data Infrastructure Essentials via CRC Press. My experience includes hands-on, real-world, across applications, data infrastructures, hardware, software, cloud, among other topics as well as being a multi-year Microsoft MVP-- Cloud Data Center Management-- and 10-time VMware vExpert. And look forward to chatting with everybody today. David? DAVID BUNTING: Great. Thanks, Greg. And Betsy? BETSY BILHORN: Thank you. Hi, everybody. I'm Betsy Bilhorn. I'm the Senior Vice President of Products at Jitterbit. For those people who are not familiar with Jitterbit, we're an enterprise integration platform as a service company. We're [INAUDIBLE] technologies for businesses to be able to orchestrate their data across all of their systems and their business processes. I'm also the person responsible for running our engineering and our operations. So I have a lot of experience working with data, both in the data integration space but also managing that data from a log perspective and providing that to our customers. David? DAVID BUNTING: Super. Thanks, Betsy. Tom? TOM O'CONNELL: Thanks, David. Hi, this is Tom O'Connell. I'm the Chief Sales Officer at ChaosSearch. I've been with the company now for a year or so, but I've got about 25 or 30 years of experience selling technology, specifically database technology, in the industry. I want to thank everyone for joining today. We've got a very large audience, and we appreciate you guys taking your time to attend. DAVID BUNTING: Great. And finally, Kevin? KEVIN DAVIS: Thanks, David. This is Kevin Davis, Director of Solutions Architecture at ChaosSearch. I've, along with Tom O'Connell, been with the organization for about a year now. Previously, my background is working with other small-staff startups and helping them, either through acquisition or IPO, ultimately achieve the success that they're looking for from a presale standpoint as well as product standpoint in the market. DAVID BUNTING: Super. Thanks, everyone. So let me just cover the agenda briefly, and then we'll get started. First, we're going to be sharing insights into the state of the state-- the data explosion unfolding all around us and why it's happening. Next, we'll take a look at the current trends and approaches to managing this data volume. And then we'll talk about the looming Elasticsearch wall, where growing data volume can drive your ELK strategy to its knees. Finally, we'll wrap up with a quick overview of ChaosSearch and then take any additional audience questions. So now without further ado, I'll turn it over to our speakers. Greg, why don't you take it away? GREG SCHULZ: Great. Thanks, David. Next slide, let's take a look at hopefully what's obvious-- but sometimes what's obvious is not so obvious. And just to help set the stage that-- look, one of the biggest challenges out there is that there's more data arriving from more sources at a faster rate, whether it be actual data, whether it be log data, event, telemetry, metadata-- different sorts of data. And the big net of it all is that the volume, the quantity is increasing as well as the size, the velocity, the speed of which things are arriving, as well as longevity and the dependencies. We've got a deeper, longer dependency-- call it an addiction for some-- on having that data and having that information available. After all, there is no such thing as an information or data recession. Granted, there are adjustments to what are going on from a economic consideration. But the key emphasis here, and what we're going to really focus on in around today, is that growth of unstructured data, particularly log data, event data, telemetry, meta, coming from a variety of different sources-- traditional sources, from servers, from virtual machines, from containers, from IoT devices, from just all kinds of different sources, at a faster pace. And one of the challenges that we have with all of this is that-- how are you going to leverage it? How can you unlock that value? And so we're going to look at this. And while this is occurring, both on-prem as well as in the cloud as well as across cloud and in hybrid, our focus today, besides being on that unstructured, is going to also be on unlocking that value and how to address unlocking that value, finding new value in that data. Which, if we jump to the next slide, takes a look at unlocking the value of all that data and that information. Again, we're going to focus on the cloud, we'll focus on unstructured data being stored in buckets, containers, objects, and blobs. But one of the challenges out here is that, if you think about it, we've got this data growth that's occurring, and some data has known value. There's data with no value, which means get rid of it, but where one of the biggest growth areas occurring is in this mid space. And for some it might be called the purgatory, but it's transitional space, in that there's data out there, you've got data in your AWS-- Amazon Web Services-- S3 buckets as objects, but you know what the value is. How do you go about finding that? How do you go about leveraging that? How do you go about unlocking new value? In other words, besides that return on investment, how are you maximizing your return on information, your return on innovating, as well as improving the response in inquiry? So what this does is sets up the discussion of finding value-- leveraging that known data value but finding and unlocking, creating new value of what do you have out there? Identifying, finding, removing barriers to search that impacts productivity, that also limit how much you can search in a given amount of time as well addressing cost. So at this point, David, I'm going to hand off over to Betsy and have Betsy tell us a little bit about what they're encountering over at Jitterbug. Betsy? BETSY BILHORN: Great. Thank you, Greg. So as Greg alluded about this massive explosion about data and we've also seen in our market sector, there's a massive explosion of applications. So there's over 100,000 SaaS apps alone, not counting your on-prem apps and everything else out there. So what that means for us, as a company, and our customers is they're putting all those systems together, they're pumping more and more data, and they're just creating more and more integrations. And that's throwing off-- when you think about us managing our cloud platform and just generating all the logs, just from a platform management perspective, that grew exponentially. So it was getting to the point where we were really not having a good handle just from the sheer volume of data that was being thrown off from the platform itself. The other thing that we saw is, because we have all this really great data and, as Greg alluded to, unlocking value-- so analytics is a big deal. We have customers that are saying to us, well hey, we want to have really robust analytics, and we want to unlock the value of what you're logging for us, and we expect that to be in the product. We expect you to be able to give it to us via streaming API or some other mechanism so I can take all that data from your platform, and I can put it in my platforms. And then when-- we wanted to provide all this stuff. We've been in business now for 15 years, and a lot's changed, technology-wise. 15 years, a lot's changed about how we think about storing data, using data, analyzing data. So as you can imagine, we're looking at this problem and we're looking at, well, over 15 years, we've had about 13 different ways of logging data. And the kind of data that we're doing, some still being logged others not. So this is a really big problem for us. And I'm just talking about managing our platform for our customers. Now, internally, I'm also getting pressure from our own business, of-- hey, we're collecting all this really great data. Can't we collect more? Because the technology is there. How can we get these reporting for product usage, for entitlement usage, license usage? And it just goes on and on and on. So I think you can imagine being in my position. I'm literally drowning in data, and everybody wants it every which way for us, and we really needed to find a solution that was going to work for us and handle all of those different situations. So Greg, I'm going to hand it back to you. GREG SCHULZ: OK, great. Thanks, Betsy. And I want to build off of that, which ties right into my next slide here, which is about enabling search and unlocking data value. Just to go back there for a minute, Betsy, you mentioned something I think ties in. And it's really that transitional point, which is, there's a lot of data out there, and as you're able to show and demonstrate that value of that data-- more importantly, transform it into information. You get requests to say, hey, well, can we have more of it? Historically, people say, well, there's this cost to keeping all this. Well, there is if you don't know the value. But it sounds like, Betsy, in your organization you have found the value or are finding that value that offsets that overhead of keeping it, hence this addiction to having more and more. Does that resonate? BETSY BILHORN: Yeah, that absolutely does resonate, and I think, too, you talk about an addiction, right? And again, a challenge for us is, how far do we feed that addiction? When does the addiction become a problem? And kind of tap dancing between how much data we store and how we provide it to the point where it does provide value. And now it's not really providing value-- it's just there for being there. GREG SCHULZ: And that ties right into what we talked about a couple of minutes ago, which is, if you know the value, take care of it. And it might mean reassigning it to a different level, leveraging different cloud resources, leveraging different cloud object storage tiers. But ultimately what it comes back to is that today, storage is relatively cheap-- particularly, cloud storage is relatively cheap. Managing it and searching it is not cheap. Likewise, open-source technology is free-- the software-- however, running it, maintaining it, supporting it is not free. You've got to pay the piper there somewhere. And that ties into-- there's different options as you navigate the sea of different search options that are out there if you're looking to unlock that value or derive more value to help offset and keep more information. And so as a part of that, you can certainly go out there and do it yourself, leverage different tools, different technology, but again, just because the software is free, running it isn't. Just because you've got resources on the cloud, those cloud resources-- Well, let's put it this way. Your credit card deserves a break, so do what you can to help it out. But one of the biggest challenges that we see running into, and Betsy, I think have you commented on this, is that there's different approaches to searching. And one of the keys is knowing what it is that you're searching for so that you can identify the right approach, the right type of tool that aligns with unlocking that value as well as leveraging different services. BETSY BILHORN: Right, exactly. And I don't know, Greg, if we want to go to the next slide, and I can talk about our experiences and what we went through. GREG SCHULZ: Absolutely. BETSY BILHORN: OK. Yeah, so as I mentioned, we had a lot of different places and a lot of different logs and, as we're building out the slide, these are just representative of some of those. So when you look at this, you immediately-- for folks in the audience, it was a little bit of crazy town. So what we said was, hey, we're going to break this project up into two phases. One, we need to do consolidation for internal use. And Greg, as you alluded to, what are we unlocking the value for? And for us, the most important thing was making sure that when we had problems, either helping our customer with performance problems on their own integration runs or just really making sure the platform having those really tight SLAs, we wanted to get on there and get really fast information and have it in one place. The second phase that we did was we said, OK next, once we get that dialed down, now we'll look at the product use piece. So our holy grail was-- let's get down to one. We looked at all the things, and it came down to three. One was Loggly, and we looked at that because we had already had it. We had relationships with the executive at the company. And we ended up not using it for two reasons. One was we do a lot of work in health care, and there are certain health care governance requirements that Loggly, at the time, did not meet for us. And then the second was that we had performance issues, and we were beginning to tip over Loggly, so we didn't think long term that that might be a best option for us. SQL Server, we had some expertise in-house, but as you know, SQL Server-- Much of our data we store is not relational, and so trying to kind of fit a square peg in a round hole with SQL Server would have been really difficult. We also didn't have super crackerjack database architect administrators who could really run SQL at the performance and the scope and scale that we would need for that. So that got nixed. And so we decided on ELK. And the reason why was we thought that it was going to give this performance we needed. It really was the hot, new thing. It checked off all the boxes for us. We also was very happy about the visualization tooling as well. So that's how we ended up picking ELK and consolidating down. So Greg, I'm going to hand it back to you if you want to talk a little bit more. But that was our selection process and how we got down there. GREG SCHULZ: Yeah, I think, Betsy, just to elaborate that on a minute, certainly the-- I think you hit on a key point there, which is that if you're searching, particularly on that unstructured data, which is typically what you're going to find being stored in buckets and objects, containers and blobs that use that right tool, and if you're looking for more structure, that's where the SQL, the NoSQLs come into play. But the other tools like the ELK, that's where that comes into play for searching those other sorts of type data. But, Betsy, as a part of that, in your experience, since you mentioned the whole SQL Server approach, searching SQL you're leveraging the indices and things like that. Part of the benefit of having structured data, nonbenefits are certainly having that restrictive constructive-type environment. The benefit of being in the unstructured is it's unstructured. So could you tell us a little bit more about your experiences that, as you moved in with ELK-type solutions, that role of the indexing, the indices to help speed up the search but also to help start addressing the resource cost associated with searching? BETSY BILHORN: Yeah, so that was a really big one for us. And when you mentioned about unstructured data, we literally had all different kinds, and we also didn't know what we were going to have in the future. So having a very consistent way where things were indexed, we could do a search very, very quickly but-- I also really cannot emphasize enough-- for us to be able to do that in a very consistent way. So we wanted to have the benefit where we could do search, and we could do queries, we can do visualization, and that would be applicable to a variety of different people, both in our DevOps and engineering team. So we have that team learning one skill and being able to use that. So that was a big thing for us. And not having to go into different places or going into the SQL Server. And then learning different languages or different ways of doing indexes and queries was not going to work for us. GREG SCHULZ: And to play off that a little bit real quick, Betsy, is that, yeah, you could run a brute force search against-- whether it be an AWS3 bucket and objects or an Azure container and blobs, you could run brute force against those and do the search, but that's going to take time. And it's going to hit your credit card pretty hard for your CPU time, your I/O, your networking, not to mention all those API calls. What did you run into with that? BETSY BILHORN: Yeah, so we actually were able to avoid that. So we had somebody who wanted to do that, and fortunately for us, we sat back and we said, OK, how long is that really going to take? And we had them do a prototype. And so for us it didn't really hit the credit card as much, but we saw that the prototype took way too long. It was really painful, and it ended up being an utter failure. We said, no, that that's not even under consideration. GREG SCHULZ: Outstanding. So I think at this point we hand it back to David, who is going to take us into another area. DAVID BUNTING: Great. Thanks, Greg and thanks, Betsy. Betsy, it would be great now if you could talk a little bit about the effort to stand up ELK and elaborate on the challenges that you ran into and ultimately how you ended up solving for those challenges. BETSY BILHORN: Yeah, absolutely. I'd be happy to talk about that. All right, so I have a little bit of a build here. We hit our ELK wall pretty early. So I'll talk first about-- we made what I would call some rookie mistakes. We felt like, oh, ELK would be this-- hey, we've done SQL Server and Loggly and all these different things, so how hard could ELK be, right? And there was a pretty steep learning curve, and it ended up at one point where it was so steep that we had to hire ELK consultants. And so that was a little bit of a shocker for us. The other thing that we found out as we were planning-- so we started our initial ELK implementation, but then we were looking at a 18-to-24-month trend of how much data we were going to put in there and what was going to be our costs over that time and from a budgeting standpoint, and then we layered on what we want to do with product. And it became very quickly obvious to us that we would pretty much almost double our entire AWS cost. So when you're talking about doubling your overall operational cost for the platform, that definitely gave me pause. It's not something that I want to go to the CFO or the board of directors about. So we went, huh, OK we're going to have to rethink that a little bit. And then, again, we go back to the retention and the speed. So it only worked for seven days. And so that was going to be a problem for us, as far as things like reporting and trending analysis. And then what do we do? OK, so we have to take all the ELK data and put it someplace else. Let's put it in S3. As Greg mentioned before, having to do brute force searches on S3 just-- it was not going to work. And it wasn't going to work for the product in the future either. The next thing that we ran into, because we were having this very steep learning curve, the DevOps team began to migrate to other tools. So unbeknownst to us, all of a sudden we're hearing the words \"data dog.\" And we're like, well, wait a minute, I thought we had an ELK strategy. So there was a little bit of a low-grade mutiny where we were trialing New Relic and we were trialing Datadog and whatnot. As I mentioned, the performance really degraded after seven days' worth of data. That wasn't going to work for us. We have to look at things anywhere from seven to 90 days. So you can imagine the issues that we were having there. And what we discovered was for both of our internal reporting and, to some degree, to provide some of the robust analytics that we wanted to put in the product long term, we would have to still have a data lake strategy. So here I am, staring down this huge ELK cost, I'm staring down-- what am I going to do with this warm storage on S3? And now I'm going to have to put in a data lake, with all the attendant costs and resources and everything else that goes with that. So that was definitely something where we were just like, what is going on there? And then lastly, it was really great for troubleshooting, however, like I said, we had to go back a ways. So if we started to see brownouts or little weirdnesses and performance issues with the platform, some of those things might start months ahead of time, and they're just tiny little indicators. So that was really critical for us. The other thing is that our customers at the same time are coming to us and saying, hey, I want two years' worth of data. And so we're building out the requirements for the product part of this, and we're going to have to drive that visualization and that search. They start saying they want two years, and we're saying, how we're going to do that? And you can do that with ELK, but it was incredibly expensive. And as I mentioned here in the slide, ELK just became really untenable for our future product needs to be able to provide differentiation and delight our customers there. So that's when we knew we really had to look at something very different, and we were going to have to take a completely different strategy as far as our search for all of that log data, both from an expense standpoint, performance standpoint, and a future-proofing standpoint. So I'll hand it back to you, Greg or David? Tom? DAVID BUNTING: Thanks, Betsy. This is David. One question actually came in from the audience while you were talking. The question is, why didn't you consider AWS Athena as part of your search, or did you? BETSY BILHORN: Well, actually we didn't, and there were really two reasons. For us, as a company, we were trying to be cloud neutral, and there was discussion about how deep did we want to get into vendor lock-in with AWS-native based Athena, and then how quickly could we get out of that if we needed to? And there's just a lot of nuanced reasons why, but that was peculiar to our company. DAVID BUNTING: Great. Thanks, Betsy. And can you talk a little bit about where you ultimately ended up landing and how you solved the problem? BETSY BILHORN: Yeah, so we were casting about and really couldn't find anything, and we were introduced to ChaosSearch-- and that was about, I would say almost two years ago, about a year and a half ago-- and had some great discussions. We saw that this was going to be a really groundbreaking thing for us to enable some of the things that we wanted to do, and we worked with the ChaosSearch team in preview, and the product delivered what you promised, and we were extremely pleased with the results for that. So we've been using ChaosSearch now in production since the spring. DAVID BUNTING: Great. Thanks, Betsy. And this is a good transition time to turn it over to Tom to talk a little bit about ChaosSearch. TOM O'CONNELL: Great. Thanks, Betsy. Really appreciate that overview. I think it's helpful as an entree into the ChaosSearch description. So I'm just going to take a couple of minutes to give a snapshot of the company and an overview of the technology and the service that ChaosSearch is providing. We're an emerging software company based in Boston. We're currently delivering a search as a service solution for our customers. And it's very unique and innovative in the markets. Such so unique and innovative that we've actually filed four patents with the US Patent Office. We're also a venture-backed company. We're fortunate enough to have attracted investments from three forward-thinking venture capital firms-- .406 Ventures, Glasswing Ventures, and Stage 1 Ventures. In addition to the funding that is in the capital that they've provided us, they've also given us some great advice and guidance as we've entered the market. We're currently a member of the AWS Partner Network because of our support of the AWS platform, but we fully expect to support other cloud providers as we move forward, such as GCP and Microsoft Azure. The product-- or the service that we're offering right now has been available for about nine months. And the interest has been incredible, which tells us that we're addressing a significant problem in the marketplace. So if we look at some of the challenges that we address, they're very, very consistent with what Betsy described at Jitterbit. No pun intended, but we're finding that data is chaotic, and it's growing significantly. Just to quantify what we mean by that, I'll use a statistic from IDC. They predict that the collective sum of the world's data will grow from 33 zettabytes this year to 175 zettabytes by 2025. That's a compound annual growth rate of 61%. Very, very significant. And what's even more important than that is half of that data will be stored in the cloud. And S3 is currently the de facto standard for cloud data storage, so that's the main reason why we picked AWS S3 as the data platform of choice to develop our current solution on. We're also seeing many siloed services in the cloud that require data to be extract or ETL process to take place and data movement from one environment to another based upon the workload. This is time-consuming-- it's costly and complex. The whole data movement process is cumbersome and opens itself up to data loss and security issues. So what we are providing is a fully-managed platform, software as a service platform that allows you to focus on search and analytics directly on S3. So our philosophy is that you should not move data from one environment to another. You've stored that data within S3, and we provide you to search on it and derive value from that data. There are some statistics on the table on the right-hand side. I'm going to ask Kevin Davis just to talk a little bit about some of those advantages with ChaosSearch over Elasticsearch. KEVIN DAVIS: Thanks, Tom. Yeah, I think one of the biggest advantages from the customers that I've worked with around the unique approach that we've taken where we've separated storage from compute, which allows us to have not just increased efficiency in our indexing rates but also our query and search rates and responses as well. Essentially, what we're able to do is to add additional compute at any time to tackle either larger data sets or return queries at an even faster rate than you would typically expect in the normal Elasticsearch deployment. And because of this and the indexing technology that our organization has developed, instead of either in the Elasticsearch world like we've seen, where you would explode the data up 5x potentially to get to do sharding or potentially not ensure-- or ensure that you don't lose any data. ChaosSearch takes the inverse approach, where we actually shrink the data down potentially around 80%, or one-to-four ratio, similar type compression to a gzip or a snappy. So as you're creating and deriving value from these indices, you're actually not increasing the value or the amount of data that's being stored, you're decreasing it. And as part of that where it's being backed by S3, one of the unique components is we never go back to the original source or raw files being stored within S3. We're always querying the indices that we create, which allows you to then continue to use the tiering or the aging policies provided by AWS S3 to potentially realize some additional cost savings there while moving that data off to a glacier or longer-term storage. TOM O'CONNELL: Great. Thanks, Kevin. So this slide actually doesn't do justice to all the use cases for ChaosSearch because all of your data will reside in S3, and we provide a layer on top of S3 to allow you to derive value from that data. Essentially, what we're providing you is a platform for unlimited data retention. So imagine the possibilities of being able to search all of your S3 data with trend analysis over a longer period of time, historical analysis from various lines of business, and analytics for the long tail of data. It's really powerful in terms of extracting value from the data that you've been storing. There's no need now to delete any of that data and move data off into another environment. We provide all queries on a single source of your S3 data. So our motto is, store everything, ask anything on S3. It's very, very powerful. So what we'd like people to do is, essentially, go to our website, chaossearch.io. Specifically, take a look at the platform page that we have listed there. I think you'll find a wealth of information in terms of what ChaosSearch can do for you and address some of your search and analytic issues. DAVID BUNTING: Great. Thanks, Tom. Thanks, Greg. Thanks, Betsy. This is David again, and we're going to take some questions from the audience now. We have a few in the queue, and I'm going to raise those questions. The first one, Kevin, I'm going to pose to you. Does ChaosSearch-- it says to you, but does ChaosSearch support GCP and Park? KEVIN DAVIS: So currently, the only cloud provider that we work with is AWS. As Tom had mentioned, S3 seem to be the de facto standard within cloud storage and where data ended up ultimately residing at the end of the day. The plan at the end of this year is to support GCP and have a multi-cloud support model and approach. And then the second part of that question, Parquet, at this point we do not support reading from the Parquet format. But again, at the end of this year, our engineering team has committed to delivering that and being able to read from those file formats as well. DAVID BUNTING: Great. Thanks, Kevin. The next question, actually, I'll put to Tom, is just a general question about how ChaosSearch is priced, if you can speak to that. TOM O'CONNELL: Thanks, David, sure. So pricing is one of the pillars of value that we provide our customers. We feel that we've got the most economical solution on the market, specifically for the log and event management space. We price our solution based upon the amount of data that is indexed on a daily basis. So at this point in time, I'm sure people will go to the website and see it, but we're at $25 per gigabyte of indexed data on a monthly basis. So it's very, very attractive when compared to solutions on the market like Elasticsearch. In addition to the index data, we provide our customers with unlimited, or no data limits when it pertains to data retention. So you can search data that's weeks, months, or even years old for that single monthly fee. DAVID BUNTING: Great. Thanks. Here's another question from the audience. Is security not compromised in the back-and-forth? TOM O'CONNELL: No, it's not. Because we've separated, again, storage from compute or S3 from EC2, the approach that we take is to deploy the service as close to the S3 bucket as possible, so always in the same region. So that helps with not just the ingress and egress cost but also any potential security challenges. During the indexing process if, for some reason, there were to either not be an object index or index not to complete successfully, the approach is fine because it's all backed and stored directly in S3 anyways. In the indices that we're creating are essentially logical representations of a physical index, so there would be no potential back-and-forth security compromises or issues that would arise from that. KEVIN DAVIS: And in addition to what Kevin has mentioned around security, we're also GDPR-compliant because what we do is, essentially, take the compute and put it in the same data center as the S3 storage. So there is no data movement outside of the data center, and therefore there's no there's no data movement outside of the boundaries of a geographic location. DAVID BUNTING: Great. Thanks, guys. Here's another question, and this one's actually for Betsy. And the question is, did you have to change any of the Elasticsearch end user tooling to leverage ChaosSearch? BETSY BILHORN: That's a easy question. No. We didn't have to do anything to change our tooling there. Just fit in just beautifully. DAVID BUNTING: OK, great. And Kevin, this question just came in. You may have spoken to this, but I'm going to ask it again anyway. Do you need to access our EC2 to deploy ChaosSearch? TOM O'CONNELL: No. Great question. So we actually don't. The way that we go about accessing our customers' S3 environments is through a simple IAM role and policy. It allows us read access to either the specific bucket or buckets, or you can get as granular as defining just the prefix within the policy itself, and then from there, we'll need separate write access. And the reason for that is, as we're doing the indexing, we write the indices directly back into our customers' S3 environment, and that would be the only access that we would need. Ultimately our customers control that, so they would be able to define either how they would want that access to work, or we can certainly make recommendations based on best practices we've seen so far with our customers. DAVID BUNTING: Great. Give another 15 seconds here to see if any more questions come in. 10, 9, 8, 7, 6. Nope. So I think we're going to wrap up now. Thanks to everyone who joined today. We really appreciate it. Thank you, Greg. Thank you, Betsy. Thanks, Kevin. Thanks, Tom. Great presentation, great information. As I mentioned at the start of the webcast, we will be circulating a recording. You'll get a link by email following the end of this webcast, and thank you again. As Tom mentioned, please come to the ChaosSearch website. You can find lots of great information there about our service and ask any questions to our experts through the chat interface on our website as well. So thanks, everyone. And have a good afternoon. ", "uploadDate" : "2021-03-25T17:21:50.000-04:00" } ```