ChaosSearch helps BAI apply large-scale log analytics to build transit of the future

Coverage from SiliconANGLE - June 16, 2021 –

Smart cities are no longer just science fiction elements, as “internet of things: devices and self-driving cars and trucks are already a reality and produce a lot of data. The ability to capture, store and analyze the immense volumes of log data generated across these networks is no small feat.

This is a challenge for companies such as BAI Communications Pty. Ltd., which provides state-of-the-art communications infrastructure (cellular, Wi-Fi, broadcast, radio and IP networks) in major commuter cities, including New York, Toronto, Tokyo, Hong Kong, Sydney and London. In other words, it keeps commuters around the globe connected while they’re in transit, driving the evolution of smart cities.

“There was an explosion of data, and a lot of our insights aren’t coming from one system anymore; it’s coming from collecting data from all of the different pieces, the different infrastructure, whether that’s your fiber infrastructure or your wireless infrastructure,” said Jeremy Foran, head of data analytics at BAI Communications. “And then, to solve problems, you need to correlate data across those systems.”

Foran and Thomas Hazel, founder and chief technology and science officer of ChaosSearch Inc., spoke with John Furrier, host of theCUBE, SiliconANGLE Media’s live-streaming studio, during the AWS Startup Showcase: The Next Big Things in AI, Security & Life Sciences. They discussed the challenge surrounding the capture, storage and analysis of increasing amounts of data, ChaosSearch’s solutions to solve these problems and how it enables BAI Communications to generate value for its customers.

Data compression means less cost

With different equipment to provide public Wi-Fi, its 5G networks, and the need to be able to adhere to rules like PCI compliance and ISO 27000, BAI holds a tremendous amount of data. To store and handle this in a cost-effective way, the company took on the solutions of the log data analytics platform company ChaosSearch.

“We wanted to use Amazon Simple Storage Service (Amazon S3), but when we were doing some projections, we don’t really have the budget for all of these places,” Foran said. “Working with ChaosSearch, using their compression, brought those costs down drastically.”

The ChaosSearch Data Platform, which is available in the AWS Marketplace, enables search and analysis of multi-model cloud data on Amazon S3.

ChaosSearch is leveraging Amazon S3 as the data storage, which allows customers to take advantage of the industry-leading scalability, data availability, security, and performance provided by Amazon S3. All the data is read from and kept in customer's buckets and under their control. Through combining Amazon S3’s advanced Identify and Access Management (IAM) functionality and roles with a Role Based Access Control (RBAC) overlay, ChaosSearch customers can enjoy exceptional security and granular data access control. ChaosSearch combined with AWS provides customers with the fastest, most scalable log analytics solution in the market.

BAI streams its devices and services into one centralized data lake on its cloud outer storage. ChaosSearch then connects to that storage and turns it into an analytic database for log analysis. All new workloads also flow into this lake.

Although an essential aspect of the ChaosSearch solution is data retention, there is no chance that the data lake will become a “data swamp,” according to Hazel. The reason is that the data is discovered, organized and automatically indexed, so businesses can get insights and therefore value from them. The solution also provides real-time notifications, alerting, monitoring and will soon offer machine learning services.

“The idea is that you use this lake offering to store all your data in a cost-effective way, but our service allows you to analyze it both in a long retention perspective as well as real-time perspective,” Hazel explained. “Bringing those two worlds together is so key, because typically you have silo solutions, whether it’s real-time at scale or retention scale.”

Another fundamental feature of ChaosSearch is that data remains fully controlled by customers.

“Unlike a lot of solutions where you move the data into them and now they are responsible, actually BAI owns everything,” Hazel explained. “They provide access so that we could provide an analysis that they could turn off at any point in time.”

Getting unpredictable insights

The benefit of having a large amount of data stored is getting more and more insights, even unpredictable ones, from it, according to Foran. For example, by providing Wi-Fi in a subway infrastructure, BAI was able to get that Wi-Fi data to begin to understand the flow of people entering and leaving the subway network. This resulted in insights for rail operators to get passengers from one place to another quicker.

“When we built the Wi-Fi, it wasn’t with the intention of getting Torontonians across the city faster, but that was one of the values that we were able to get from the data,” Foran stated.

Relying on the ChaosSearch compression solution, BAI is storing all of its data. In addition, the company believes that the services offered by the platform will make it easier to use technologies such as machine learning.

“Starting from scratch and trying to build out models that have value, that takes a fair amount of work, and that landscape keeps changing,” Foran said. “Being able to push our data into an Amazon S3 bucket and then retain that data and then get anomaly detection on top of it, that’s something special and that unlocks a lot of ability for our teams.”

About ChaosSearch

ChaosSearch helps modern organizations Know Better® by activating the data lake for analytics. The ChaosSearch Cloud Data Platform indexes customers’ cloud data, rendering it fully searchable and enabling analytics at scale with massive reductions of time, cost and complexity.

ChaosSearch was purpose-built for cost-effective, highly scalable analytics encompassing full text search, SQL and machine learning capabilities in one unified offering. The patented ChaosSearch technology instantly transforms your cloud object storage (Amazon S3, Google Cloud Storage) into a hot, analytical data lake.

For more information, visit ChaosSearch.io or follow us on Twitter @ChaosSearch and LinkedIn.