Wondering how to create a dashboard in Kibana to visualize and analyze your log data?
In this blog post, we’ll show you how to create a dashboard in Kibana, step-by-step. You’ll learn how you can use Kibana to query indexed application and event log data, filter query results to highlight the most important and actionable information, build Kibana visualizations using your log data, and incorporate those visualizations into a Kibana dashboard.
What is Kibana?
Kibana is an open-source data analytics and visualization tool used by DevOps and IT security teams to explore, monitor, and extract insights from structured log data. Once log data has been centralized (with a log aggregation tool, e.g. Logstash or Amazon CloudWatch) and indexed (e.g., with Elasticsearch or ChaosSearch®,), IT analysts can use Kibana to perform queries and filtering, construct data visualizations, and combine those visualizations into dashboards that reveal valuable insights into system security and application performance.
Some organizations use Kibana in combination with Logstash and Elasticsearch as part of the open-source ELK stack log analytics solution. Logstash is used to aggregate, parse, and transform log data before indexing the data in Elasticsearch. Once data is indexed, Elasticsearch allows users to query the data in different ways and produce tables that can be visualized using Kibana.
At ChaosSearch, we’ve integrated the Open Distro version of Kibana directly into our comprehensive log analytics solution. ChaosSearch transforms Amazon S3 into a data lake repository for log and event data, allowing DevSecOps teams to aggregate, index, and analyze log data in real time, with no data movement and no ETL process.
With ChaosSearch, log data is aggregated in Amazon S3 buckets whose full functionality is available from within the ChaosSearch console. Log data in S3 buckets can be indexed using our proprietary Chaos Index® format and DevSecOps teams can enable Live Indexing to index newly-created objects in specific Object Groups, enabling near real-time querying for data as it becomes available.
Once log data has been indexed by ChaosSearch, users can leverage the ChaosSearch Refinery® to clean, prepare, and transform the data without writing or executing any code. This results in the creation of an Index View: a logical index, based on a physical index, ready for discovery, visualization, dashboarding, and analysis using Kibana.
Before we get into the details and how-to portion of this guide, let’s start with a brief overview of the main Kibana functionalities. Opening the left side-bar of Kibana, you’ll notice five key capabilities that you can leverage to support your log analytics program:
Kibana’s Discover tool gives you the ability to interact with your data. After choosing which Index View you’d like to work with, Kibana will allow you to submit queries, filter your search results, and view document data from your log files. If the data is timestamped, a histogram representation of your data will be automatically generated at the top of the page.
Kibana’s Visualize tool gives you the ability to construct different kinds of visualizations using ChaosSearch indices you have created.
Kibana’s Dashboard tool gives you the ability to combine log data visualizations built from your ChaosSearch indices into functional dashboards. Using dashboards with real-time indexing allows you to automatically update dashboards in near real-time as events are logged in your Amazon S3 buckets.
With the ChaosSearch Kibana integration, you’ll notice an added feature called “Alerting”. This feature gives you the ability to monitor log and event data in real-time, create customized alert triggers, and set a destination (Slack channel, email, AWS chime, etc.) where notifications may be sent when an alert is triggered.
Finally, Kibana’s management interface gives you the ability to adjust Kibana’s runtime configuration and tweak advanced settings to change how Kibana behaves as you Discover, Visualize, and Dashboard your log data.
Next, we’ll take you through the process of data discovery, visualization, and how to create a dashboard in Kibana.
How to Discover Your Data in Kibana
Kibana’s Discover screen is where you’ll be able to perform queries and create summary tables based on your indexed log data. As a prerequisite for using the Discover tool, you’ll need to have ingested some log data into your S3 buckets and have created ChaosSearch Index Views using the Refinery tool.
Having done so, you’ll be able to query those Index Views and apply filters on the Discover screen.
In this example, we’re using Kibana’s Discover screen to explore log data from Amazon ELB, a software-based load balancer that distributes incoming traffic across a collection of AWS EC2 instances.
Using the drop-down menu on Kibana’s left side-bar, we were able to select “aws-elb-logs”, a Chaos Index View we created previously using the ChaosSearch refinery tool. Beneath our selection, Kibana shows us all the fields present in the index that we might choose to query from. For this analysis, we’re interested in exploring status codes that were generated by the ELB application.
Looking under the heading “Available Fields”, we can see that our Chaos Index View contains a field called “elb_status_code” that we can query. To set up our query, we’ll need to use the Query/Search Bar at the top of the Kibana Discover page to express our query using the Kibana Query Language (KQL). In this case, we’re looking for instances where ELB produced a 404 status code, so our query in KQL reads “elb_status_code:404”.
With this query, we can isolate all of the ELB logs from our Chaos Index View that have a value of “404” in the “elb_status_code” field. We can also specify an absolute or relative data range for our query - here, we’ve chosen to exclusively analyze data from the last 3 years. Finally, we hit the Refresh button to see the results from our query.
Once you’ve entered your query, Kibana will return a list of results that match your query parameters. You can click on each result to expand it and view all data from that log file in JSON format.
At ChaosSearch, we’ve upgraded the Open Distro version of Kibana with additional features, including a Query Progress Bar that helps you monitor the status of your query, and a Cancel Query button you can use to abort queries that are taking too long.
When you’re happy with your query, you’ll need to click the Save button and enter a name for your query before you can turn your query results into a visualization.
How to Build Visualizations in Kibana
Once you’ve queried your data, it’s time to transform your results into a visualization. Clicking the Visualize button on the left sidebar will bring you to the Visualizations screen where you can create a new visualization or access visualizations you have created in the past. To build a new data visualization, click on the blue Create New Visualization button.
Kibana allows you to create many different types of visualizations, including area plots, data tables, gauges, goal trackers, heat maps, bar and line graphs, and more. The right option for you will depend on the specific goals of your analysis. You can read a short description of each visualization format to help determine the best option for your intended use case.
After choosing a format for your visualization, you’ll need to identify a data source. You can choose either a saved search (like the one we created above), or perform a new search by selecting the Chaos Index View with the data you want to visualize.
In this example, we’re using our indexed ELB log data to create a visualization of ELB status codes generated over the past 3 years.
In our previous example, we focused specifically on 404 error codes, but now we’re interested in all types of ELB status codes and how frequently they appear in our log data. To shed light on that, we chose to organize our ELB status code data into a Pie Chart.
After choosing a visualization format and data source, it’s time to configure the visualization builder to display your data in a useful way.
Configuration options in the visualization builder will vary depending on the visualization format you choose for your data. For a pie chart, you’ll need to make use of two aggregations: the Metrics aggregation and the Buckets aggregation.
The Metrics aggregation establishes how the size of each slice in your pie chart will be determined. In this example, we’re using the Count aggregation, so the size of each slice will reflect the total number of instances of each ELB status code within our source data (relative to the whole, since it’s a pie chart).
The Buckets aggregation determines what information is being retrieved from your data set to create the graph. In this example, we use the Terms aggregation, which allows us to display the most frequently occurring status codes in our data. We also specify which field we’re querying, and we’re telling Kibana to highlight the five most frequently occurring status codes in descending order.
Finally, we created a filter to remove instances of ELB status code 200 from our pie chart. This status code normally indicates that ELB is healthy and functioning normally, but we’re more interested in other status codes indicating errors that we may want to investigate. The pie chart we created here shows that the most common error codes were:
- HTTP 404 (Not found)
- HTTP 500 (Internal server error)
- HTTP 503 (Service unavailable)
You can modify visualizations by adding more filters, changing the date range, or even changing the colors. If you hover your mouse over a slice of the pie chart, you’ll see a small pop-up window with details about the data for that specific slice. When you’re done creating your visualization, you’ll need to click Save and give it a name before using it to create a dashboard in Kibana.
How to Create a Dashboard in Kibana
Creating functional and informative dashboards is a great way to start monitoring network security and application performance in real-time. To get started, you’ll need to click “Dashboard” on the left side-bar, then click on “Create new dashboard”. Next, you’ll be asked to Add Panels to your Kibana dashboard.
Panels are the building blocks of dashboards in Kibana. Any visualization you have previously created and saved in Kibana can be added as a Panel in your dashboard. You can also add a Text panel with static information about your dashboard, or a Controls panel that gives you the ability to filter your data in real-time. Panels can also be used to display a saved search table from the Kibana Discover tool, a table of live-streaming logs, or the results of machine-learning anomaly detection jobs.
You can also clone panels within a dashboard, copy panels to a separate dashboard, and even apply filters until every panel in your dashboard shows the data and information you want to see.
Once you’ve specified what data should be included in each panel on your dashboard, you’ll be able to arrange and reorganize panels and other dashboard elements, and apply design options to make your dashboard more visually appealing and functional for users.
When you’re finished, you can click on the Save as button to add a title to your completed dashboard and save it for future reference.
You can share your newly created Kibana dashboard a number of ways: by embedding it as an iframe on a webpage, sharing a direct link, or generating a PDF or PNG report of your dashboard.
As you get more familiar with how to create dashboards in Kibana, you’ll be able to build more visually appealing and informative dashboards to support your application monitoring and security log analytics initiatives.
Use Kibana for Real-Time Insights into Your Log Data
Now that you’ve learned how to create a dashboard in Kibana, what’s next?
If you haven’t yet, now would be a great time to configure real-time indexing for the log data included in your dashboard. With real-time indexing, the log data you generate is used to automatically update your indices, visualizations, and dashboards in Kibana.
As a result, you’ll be able to gather real-time insights into network security and application performance, and you can even configure customized Alerts to notify you when anomalies are detected in your newly ingested log data.