Alerting

Humio® has support for alerting and can be configured to notify external systems about error conditions or other events. Every repository has its own set of alerts.

The alerting concept in Humio consists of two parts: Notifiers and Alerts.

Alerts

Alerts are standard Live Queries that run continuously, and trigger whenever there are one or more rows in the search result.

For example, an alarm can be configured to trigger whenever there are more than five status 500s in the accesslog.

#type=accesslog statuscode=500
| count(as=internal_server_errors)
| internal_server_errors > 5

If there are fewer than five events in the time window, the search will be an empty result and nothing will happen. If there are more than five events, a non-empty result will be returned and then the alert will trigger the notifier.

Types of alerts

Alerts can be divided into two groups

  • Single events that can affect one or more users’ experience with the product. Usually not something that should wake engineers up at night over, but could result in a ticket on your issue tracker.
  • Faulty state is when one or more components have reached a bad state and are unable to function properly. This usually affects most users and is something that should wake engineers up at night.

Creating alerts

The easiest way to create a new alert is by building up your query in the Search view.

  1. Don’t forget to set a live time window for the search.
  2. Select the Save As… > Alert option on the right.
  3. Give it a name, select a notifier, and finally notification frequency. The notification frequency is the minimum time before the same alert will be triggered again.

For notifiers like email and Slack, you want a lower notification frequency (more time in-between triggers) since the triggers don’t de-duplicate.

Notifiers

A notifier is a module that sends notifications when alerts trigger.

Built-in notifiers

Humio currently supports the following notifier types:

Configuring a notifier

  • Go to Alerts > Notifiers > New Notifier.
  • Select a type of notifier from the Notifier Type dropdown list.

All notifiers must be assigned a name.

For on-site installations, remember to set the PUBLIC_URL field in the Humio config. This will ensure that links in notifications will go to the correct URL.

Custom Notifiers

If the built-in notifiers are not enough and you need to make something custom, Humio supports webhooks that allow you to call an external service with HTTP. You can add headers and customize body of the message as seen below.

Message templates

Notifier templates are used to create the messages sent by notifiers. They currently apply to Email and Webhook notifiers. The template engine is a simple “search/replace” model, where the {…} marked placeholders are replaced with contextually-aware variables.

See the list for an explanation of the placeholders:

Placeholder Description
{field:$FIELE_NAME} Extracts the value of $FIELD_NAME from the alert result set. If there are multiple rows in the result, the first result is used. Put field names with spaces in double quotes, {field"My Field"}.
{alert_name} The user-made name of the alert fired.
{alert_description} A user-made description of the alert fired.
{alert_triggered_timestamp} The time at which the alert was triggered.
{event_count} The number of events in the query result.
{url} A URL to open Humio with the alert’s query.
{query_string} The query that triggered the alert.
{query_result_summary} A summary of data in the query result.
{query_time_interval} The time interval for the alert’s query. (10m > now)
{warnings} Any warnings that were generated by the query.