How to Create a Dashboard in Kibana
Wondering how to create a dashboard in Kibana to visualize and analyze your log data?
In this blog post, we’ll provide a step-by-step explanation of how to create a dashboard in Kibana. You’ll learn how to use Kibana to query indexed application and event log data, filter query results to highlight the most critical and actionable information, build Kibana visualizations using your log data, and incorporate those visualizations into a Kibana dashboard.
What is Kibana?
Kibana is an open-source data analytics and visualization tool used by CloudOps and IT security teams to explore, monitor, and extract insights from structured log data. Once log data has been centralized (with a log aggregation tool, e.g. Fluentbit, Logstash or Amazon CloudWatch) and indexed (e.g., with Elasticsearch, Amazon OpenSearch Service, or ChaosSearch®), IT analysts can use Kibana to perform queries and filtering, construct data visualizations, and combine those visualizations into dashboards that reveal valuable insights into system security and application performance.
Some organizations use Kibana in combination with Logstash and Elasticsearch as part of the open-source ELK stack log analytics solution. Logstash is used to aggregate, parse, and transform log data before indexing the data in Elasticsearch. Once data is indexed, Elasticsearch allows users to query the data in different ways and produce tables that can be visualized using Kibana.
At ChaosSearch, we’ve integrated the OpenSearch version of Kibana (a.k.a OpenSearch Dashboards) directly into our comprehensive log analytics solution. ChaosSearch transforms Amazon S3 into a data lake repository for log and event data, allowing DevSecOps teams to aggregate, index, and analyze log data in real-time, with no data movement and no ETL process.
With ChaosSearch, log data is aggregated in Amazon S3 buckets whose full functionality is available from within the ChaosSearch console. Log data in S3 buckets can be indexed using our proprietary Chaos Index® format, and DevSecOps teams can enable Live Indexing to index newly-created objects in specific Object Groups, enabling near real-time querying for data as it becomes available.
Learn more about getting your data into S3:
Once log data has been indexed by ChaosSearch, users can leverage the ChaosSearch Refinery® to clean, prepare, and transform the data without writing or executing any code. This results in the creation of an Index View: a logical index, based on a physical index, ready for discovery, visualization, dashboarding, and analysis using Kibana.
Before we dive into all of the details and the how-to portion of this guide, let’s start with a brief overview of the main Kibana functionalities. Opening the left sidebar of Kibana, you’ll notice five key capabilities that you can leverage to support your log analytics program:
How Kibana Can Support Your Log Analytics
Kibana’s Discover tool is usually the starting point to data exploration. It gives you the ability to interact quickly with your data through search and filters. After choosing which Index View you’d like to work with, Kibana will allow you to submit search queries, filter the results, and view document data from your log files. If the data is timestamped, a histogram representation of your data will be automatically generated at the top of the page.
Watch: Searching with Kibana
Kibana’s Visualize tool allows you to construct different kinds of visualizations using the ChaosSearch indices you have created.
Kibana’s Dashboard tool gives you the ability to combine log data visualizations built from your ChaosSearch indices into functional dashboards. Using dashboards with real-time indexing allows you to automatically update dashboards in near real-time as events are logged in your Amazon S3 buckets.
With the ChaosSearch Kibana integration, you’ll notice an added feature called “Alerting.” This feature gives you the ability to monitor log and event data in real-time, create customized alert triggers, and set a destination (Slack channel, email, AWS chime, etc.) where notifications may be sent when an alert is triggered.
Finally, Kibana’s management interface gives you the ability to adjust Kibana’s runtime configuration and tweak advanced settings to change how Kibana behaves as you Discover, Visualize, and Dashboard your log data.
Next, we’ll take you through the process of data discovery, visualization, and how to create a dashboard in Kibana.
How to Discover Your Data in Kibana
Kibana’s Discover screen is where you’ll be able to perform queries and create summary tables based on your indexed log data. As a prerequisite for using the Discover tool, you’ll need to have ingested some log data into your S3 buckets and have created ChaosSearch Index Views using the Refinery tool.
Having done so, you’ll be able to query those Index Views and apply filters on the Discover screen.
In this example, we’re using Kibana’s Discover screen to explore log data from AWS CloudTrail, which allows you to monitor and record all account activity across your AWS infrastructure.
Using the drop-down menu on Kibana’s left sidebar, we could select the Chaos View we created previously using the ChaosSearch Refinery® tool. Beneath our selection, Kibana shows us all the fields present in the View that we might choose to query from. For this analysis, we’re interested in exploring AWS console login events present in the CloudTrail.
Looking under the heading “Available Fields,” we can search for a field called “Records.eventName,” which contains the name of the logged event. To set up our query, we’ll need to use the Query/Search Bar at the top of the Kibana Discover page to express our query using the Kibana Query Language (KQL). In this case, we’re looking for ConsoleLogin events, so our query in KQL reads Records.eventName: "ConsoleLogin."
With this query, we can isolate all of the ConsoleLogin events from our CloudTrail logs. We can also specify an absolute or relative data range for our query - here, we’ve chosen to exclusively analyze data from the last 90 days. Finally, we hit the Refresh button to see the results from our query.
Once you’ve entered your query, Kibana will return a list of results that match your query parameters. You can click on each result to expand it and view all data from that log file in table or JSON format.
ChaosSearch is built from the ground up for analyzing vast amounts of data, usually on the Terabyte and Petabyte scales. For this reason, our version of Kibana includes some additional features required when exploring these large datasets. The first one is a Query Progress Bar which monitors the status of your query. There is also a Cancel Query button used to abort queries when for example, you detect an error on your search syntax. Finally, there is a counter that indicates the Total Queried Data, which gives users instant feedback on how much data the query is running against.
Another useful feature of Discover is the ability to select specific fields to be listed on the event details table, giving you a clear view of the fields you're interested in for your query. All other fields for the event are now hidden but can still be accessed by expanding any particular event.
When you’re happy with your query, you’ll need to click the Save button and enter a name for your query before you can turn your query results into a Visualization.
How to Build Visualizations in Kibana
Once you’ve queried your data, it’s time to transform your results into a visualization. Clicking the Visualize button on the left sidebar will bring you to the Visualizations screen, where you can create a new visualization or access visualizations you have previously created. To build a new data visualization, click on the Create Visualization button.
Kibana allows you to create many different types of visualizations, including area plots, data tables, gauges, goal trackers, heat maps, bar and line graphs, and more. The right option for you will depend on the specific goals of your analysis. You can read a short description of each visualization format to help determine the best choice for your intended use case.
After choosing a format for your visualization, you’ll need to identify a data source. You can choose either a saved search (like the one we created above) or perform a new search by selecting the Chaos View with the data you want to visualize.
In this example, we’re using our indexed CloudTrail log data to create a visualization of event names generated over the past 30 days.
In our previous example, we focused specifically on Console Login events, but now we’re interested in all types of events and how frequently they appear in our log data. To shed light on that, we chose to organize our events data into a Pie Chart.
After selecting a visualization format and data source, it’s time to configure the visualization builder to display your data in a useful way.
Configuration options in the visualization builder will vary depending on the visualization format you choose for your data. For a pie chart, you’ll need to use two aggregations: the Metrics aggregation and the Buckets aggregation.
The Metrics aggregation establishes how the size of each slice in your pie chart will be determined. In this example, we’re using the Count aggregation, so the size of each slice will reflect the total number of events for a particular event name within our source data (relative to the whole, since it’s a pie chart).
The Buckets aggregation determines what information is being retrieved from your data set to create the graph. In this example, we use the Terms aggregation, which allows us to display the top 10 most frequently occurring Event Names in our data.
Finally, we created a filter to remove events without information on the EventName field from our pie chart related to CloudTrail log file integrity events.
You can modify visualizations by adding more filters, changing the date range, or even changing the colors. If you hover your mouse over a slice of the pie chart, you’ll see a small pop-up window with details about the data for that specific slice. When you’re done creating your visualization, you’ll need to click Save and give it a name before using it to create a dashboard in Kibana.
How to Create a Dashboard in Kibana
Creating functional and informative dashboards is a great way to start monitoring applications and AWS services in real-time. To get started, you’ll need to click “Dashboard” on the left sidebar, then click “Create new dashboard.” Next, you’ll be asked to Add Panels to your Kibana dashboard.
Panels are the building blocks of dashboards in Kibana. Any visualization you have previously created and saved in Kibana can be added as a Panel in your dashboard. You can also add a Text panel with static information about your dashboard or a Controls panel that allows you to filter your data in real-time. Panels can also be used to display a saved search table from the Kibana Discover tool, a table of live-streaming logs, or the results of machine-learning anomaly detection jobs.
You can also clone panels within a dashboard, copy panels to a separate dashboard, and even apply filters until every panel in your dashboard shows the data and information you want to see.
Once you’ve specified what data should be included in each panel on your dashboard, you’ll be able to arrange and reorganize panels and other dashboard elements and apply design options to make your dashboard more visually appealing and functional for users.
Once you have finished, you can click the Save as button to add a title to your completed dashboard and save it for future reference.
You can share your newly created Kibana dashboard in several ways: by embedding it as an iframe on a webpage, sharing a direct link, or generating a PDF or PNG report of your dashboard.
As you get more familiar with how to create dashboards in Kibana, you’ll be able to build more visually appealing and informative dashboards to support your application monitoring and security log analytics initiatives.
Use Kibana for Real-Time Insights into Your Log Data
Now that you’ve learned how to create a dashboard in Kibana, what’s next?
If you haven’t yet, now would be a great time to configure real-time indexing for the log data included in your dashboard. With real-time indexing, the log data you generate is used to automatically update your indices, visualizations, and dashboards in Kibana.
As a result, you’ll be able to gather real-time insights into network security and application performance, and you can even configure customized Alerts to notify you when anomalies are detected in your newly ingested log data.
Read the Blog: The Power of ChaosSearch Alerts
Watch the Webinar: Advanced Analytics - Data Architecture Best Practices for Advanced Analytics
Check out the Whitepaper: DevOps Forensic Files: Using Log Analytics to Increase Efficiency