Integrating Humio with Grafana For A Comprehensive Dashboarding Experience

Using the Humio Web UI, you can create dashboards that display the results of your queries in real time and in a way that makes them easy to understand. This is important, as having a comprehensive dashboard can mean the difference between spotting an important development in your infrastructure and letting it pass by unnoticed.

While we are proud of our UI, it is not your sole option for creating dashboards with Humio data. We know that some of our customers like to create their dashboards in Grafana, a popular open source project that has been around since 2014. With Grafana, you may create dashboards with data extracted from various data sources, including Prometheus, Sensu, Splunk, Elasticsearch, and of course Humio.

You can use Humio as a data source within Grafana by using our Humio plugin for Grafana, which we have developed with help from our open source community. To install and set up this plugin, please follow the guide in the corresponding documentation, which also explains general usage of the add-on.

On this page, we wish to show you the many ways in which you can use Humio query results within Grafana, thanks to our plugin. This should hopefully inspire you to try it out, and start including Humio data within your existing Grafana dashboards!

Panels

A Grafana dashboard may contain many panels, each showcasing data from a different type of data source. After installing the Humio plugin, you will be able to create a Humio data source within Grafana, and start adding panels to your dashboards. These panels can then be populated with results from Humio queries, which you can type directly into the Grafana UI.

Here is a showcase of some the panels that you’ll be able to display your Humio data in.

Time Series Graphs

When you use the timechart()function in one of your queries, it creates one or more time-series based on the event data fed into the function. Time-series data can easily be plotted as graphs in Grafana by using the Graph panel type, which can be seen below.

This specific panel showcases the number of incoming connections over time across different hosts in a network. It was created by querying a Humio repository filled with Windows logs, coming from different Windows PCs on the same network. The query used can be seen below. It filters for all events that indicate a network connection, and then pipes those into timechart(), generating a time series for each unique hostname discovered in the filtered events.

event.action = "Network connection detected (rule: NetworkConnect)" |
timechart(series=host.name)

Like in the Humio UI, you are able to zoom into specific parts of your graph in Grafana, and you may plot multiple time-series from multiple different queries on the same panel.

Bar Gauges

Humio supports the use of the groupby() function in its queries, following the same concept as the GROUP BY found in SQL. The grouped data can be visualized using Grafana’s Bar Gauge panel type, as shown below.

This specific panel is generated from a HaProxy server, which sits in front of a Humio instance. It shows the average number of bytes read per Humio query job, created on that particular Humio instance. The query used to create the panel can be seen below. As the logs from HaProxy only leave details about web requests, we have to use Humio’s regex feature to filter for the events, which describe a web requesting polling from a query job. These events are then grouped by query job, summing up the number of bytes read from the job. Finally the data is grouped across the different repos on the Humio instance.

method=GET |
"status_code"=200 |
regex("/api/v1/dataspaces/(?<refdomain>.+?)/queryjobs/1-(?<jobId>.{24})",field=path) |
groupby([refdomain, jobId],  function=sum(bytes_read)) |
groupby(refdomain, function=avg(_sum))

As with all other Grafana panels, the Bar Gauge panel is visually customizable, and you can change its look and feel by editing the settings of the panel. In this example we have chosen a retro LCD look for each of our gauges.

Tables

Humio generates tabular data, when the table() function is used within a query. Naturally, this type of data can be displayed in Grafana’s Table panel type, as shown below. As a neat feature, if you do not want to use the header names generated by the query, Grafana allows you to change them through the Grafana UI.

This particular panel was created from the query below, which looks at Windows logs in a Humio repository. First, we use filters to get ahold of all the unsuccessful login attempts. Afterwards we group the logons by username before finally piping the grouped data into the aforementioned table() function.

winlog.task = Logon |
winlog.keywords[0] = "Audit Failure" |
groupby(winlog.event_data.TargetUserName) |
table([_count, winlog.event_data.TargetUserName], sortby=_count, limit=5)

Single Stats

While some queries produce very simple results, they can still be very useful. As an example, take queries that use the count() function, which only returns a single number. When displaying this in a Singlestat panel, as shown below, you can add thresholds values. Now each time the value breaks a threshold the panel will change color.

Again, this panel has been populated by querying a Humio repo filled with Windows logs. It uses a few filters to get all unsuccessful login attempts, and then pipes these into count().

winlog.task = Logon  |
winlog.keywords[0] = "Audit Failure"  |
count()

Worldmap

Humio has a great feature in the worldmap() function, which can take input from log events, such as IP addresses, and map them into a data format that can be visualized on a map.

While Grafana doesn’t have a native Worldmap panel, there is a third party Worldmap plugin. This can be combined with the Humio plugin to showcase locational data on your Grafana dashboard, as shown below.

The query to create this panel was very simple. It only required us to filter on http events and then pipe those into the worldmap() function.

type=http | worldmap(ip=client_ip)

Please note that in order to get the results shown on the panel above, you will need to configure the panel a bit. This is however covered in our documentation.

Query Variables

In Grafana, a variable is a named set of values, which you can use to dynamically generate dashboard content. While you can of course hard-code your own set of variable values, i.e. a list of system host names, the Humio plugin allows you to populate your variables with the results of Humio queries using the Query Variable feature in Grafana. This saves you from having to type in each variable value, and it ensures that the values of your variables can be updated automatically over time, depending on Humio query results.

If you take a look at the figure below, you can see how we configure a query variable Hostnames, by setting up a * query against our WinLogs repository. To turn the query results into variable values, we specify that we want to extract the values from the host.name field of the results. Finally, we click the Execute Humio Query button, which yields the following set of variable values: (LAPTOP-OEMCKUCU, shinkiro.corp.atom-it.dk). Note that while the query result can include up to 200 events, duplicate values are discarded when populating our variable, leaving us with the names of the only two hosts ingesting logs into WinLogs.

Variables have various uses within Grafana, but we want to highlight how they can be used to template Humio queries within dashboard panels. This will let you avoid having to define multiple near identical queries for multiple panels. Let us say you want to create a panel for each host showing the number of windows logs generated by the user SYSTEM over time. We can template the query using the variable Hostnames like so:

host.name=$Hostnames 
| winlog.user.name = SYSTEM 
| timechart(series=host.name)

The same principle works if we want to display a panel for each host, showcasing the number of failed logins:

host.name=$Hostnames 
| winlog.task = Logon
| winlog.keywords[0] = "Audit Failure"
| count()

With these queries plugged into their respective panels in a dashboard, we can use the Hostnames variable dropdown on the dashboard to switch between which host we are displaying data from. In the following figure, you can see how our dashboard shows data from LAPTOP-OEMCKUCU:

And by just picking another option in the Hostnames dropdown, we can update the dashboard to show data from shinkiro.corp.atom-it.dk instead:

Annotations

Grafana annotations allow you to visually display important log events across your Grafana graph panels.

Let us take the graph from the Query Variables example, which showcases a graph over the number of windows logs generated by the SYSTEM user. Using annotations, this graph can be annotated to show each Windows Update event that has occurred across the time period displayed by the graph.

In the figure below, you can see how we configure a new annotation Windows Update Events with the following filter query:

host.name = $Hostnames | winlog.task = "Windows Update Agent"

Which yields log events for each Windows Update event that has occurred on a particular host. Note that this is another example of using a query variable to template a query. In addition to the query, we also specify the text displayed alongside each annotation as corresponding to the @rawstring field of that particular log event.

With the annotation configured, our graph panels will now be marked with an annotation for each log event returned by our annotation query, which is executed alongside the graph query, marking each Windows Update event: