Humio has built-in support for alerting. Every repository has it’s own individual set of alerts. The alert concept in Humio consists of two parts: Notifiers and Alerts.
Notifiers are the integration between Humio and other systems. Currently e-mail notifications and webhooks are supported along with a list of dedicated integrations e.g. Slack.
For instance, when an alert detects your accesslog has reached a set threshold for Internal Server Errors, it will trigger a notifier that will send a message informing about the issue .
Alerts are standard Live Queries that run continuously. Alerts trigger whenever there is one or more rows in the search result.
Example For instance an alarm can be configured to trigger whenever there’s more than 5 status 500s in the accesslog.
#type=accesslog statuscode=500 | count(as=internal_server_errors) | internal_server_errors > 5
If there’s less than 5 events in the time window the search will be an empty result and nothing will happen. On the other hand, if there’s more than five events a non-empty result will be returned and then alert will trigger the notifier.
Generally speaking, alerts can be divided into two groups:
When alerts trigger they are configured to send notifications using a notifier.
The easiest way to create a new alert is by building up your query in the Search view.
Don’t forget to set a Live time window for the search. And then hit the
Save As… →
Alert option on the right.
Then give it a name, select a notifier and finally Notification Frequency. The Notification Frequency is the minimum time before the same alert will be triggered again.
For notifiers like E-mail and Slack you want a lower Notification Frequency (more time in-between) triggers since they don’t do de-duplication.
A notifier is a module that sends notifications when alerts trigger.
Our list of notifiers is ever growing and currently we do support the following services.
Creating a new Notifier is pretty simple. On the Alerts Page there’s a Notifiers menu item on the left. To create a new one hit the “New Notifier” button on the top right.
First you’ll need to select a type of notifier from the “Notifier Type” dropdown list
All notifiers must be assigned a name.
Make sure you give your notifiers a meaningful name, i.e. “OpsTeam”, “Backlog issues” etc. We will make sure that the type of the notifier is also displayed.
If the built-in notifiers are not enough and you need to make something custom, Humio supports webhooks that allow you to call an external service with HTTP. You can add headers and customize body of the message as seen below.
Notifier templates are used to create the message sent by notifiers.
They currently apply to Email and WebHook notifiers.
The template engine is a simple “search/replace” model, where the
placeholders are replaced with contextual aware variables.
See the list for an explanation of the placeholders:
||Extracts the value of
||The user made name of the alert fired.|
||A user made description of the alert fired.|
||The time at which the alert was triggered.|
||The number of events in the query result.|
||A URL to open Humio with the alert’s query.|
||The query that triggered the alert.|
||A summary of data in the query result.|
||The time interval for the alert’s query. (e.g. 10m -> now)|
||Any warnings that was generated by the query.|