Data sent to Humio usually has some structure. You can use parsers to extract this structure. For example, a standard web server log has the status code, method, and URL fields for each log line.
When sending data to Humio, for example using Filebeat, you must specify a parser telling Humio how to parse the incoming data.
Humio has some built-in parsers for common formats like access logs from Apache and Nginx web servers. It also allows for custom parsers.
To list all parsers for a given repository:
GET /api/v1/repositories/$REPOSITORY_NAME/parsers
[
{
"builtIn": true,
"parseKeyValues": false,
"parser": "(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?",
"id": "accesslog",
"dateTimeFields": [
"@timestamp"
],
"kind": "regex",
"dateTimeFormat": "dd/MMM/yyyy:HH:mm:ss Z"
},
{
"id": "json",
"kind": "json",
"parseKeyValues": false,
"dateTimeFields": [
"@timestamp"
]
}
]
The output format is similar to the input format in Add Parser,
where $YOUR_HUMIO_URL
is the URL for your Humio Cloud Account.
Here’s an example_
curl https://`$YOUR_HUMIO_URL`/api/v1/repositories/$REPOSITORY_NAME/parsers \
-H "Authorization: Bearer $API_TOKEN"
To create a parser for a given repository:
POST /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID
or to updated an existing parser use
PUT /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID
The JSON request body has the following attributes:
Name | Type | Required | Description |
---|---|---|---|
kind |
String | Yes | Controls which parser kind to create. You can set this to regex , or json . |
parser |
String | Yes | The parser specification. The contents of this field vary depending on the type of parser you are creating. See the details below |
parseKeyValues |
Boolean | No | Sets whether you want the parser to parse ‘key=value’ pairs in the log line. The default value is false . |
dateTimeFields |
Array | Yes | Specifies the fields which contain the timestamp of the event. You can specify multiple fields, for example, a date field and a time field. The values of these fields are concatenated with whitespaces. Humio parses these fields with the format that you specify in the dateTimeFormat attribute. |
dateTimeFormat |
String | No | The format string that Humio should use to parse the fields identified by the dateTimeFields attribute. This attribute uses the Java DateTimeFormatter syntax. The default value is the ISO-8601 format, for example, yyyy-MM-dd'T'HH:mm:ss.SSSZ , with milliseconds as an optional addition. |
timezone |
String | No | This field is only used if the timestamp of the event is in localtime and does not have a timezone. In that case, you can use it to set a timezone. Do not use this field if the timezone is part of the dateTimeFormat .Examples: UTC , Z , or Europe/Copenhagen . |
tagFields |
Array | No | Specify fields in events generated by this parser that should be turned into tags. For example it could be specified that the host field in the events from this parser should be treated as a tag. |
Standard HTTP response codes, where $YOUR_HUMIO_URL
is the URL for your Humio Cloud Account.
Here’s an example:
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
-XPUT \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"parser": "^(?<date>\\S*) (?<time>\\S*) (?<host>\\S*) (?<appname>\\S*):",
"kind": "regex",
"parseKeyValues": true,
"dateTimeFormat": "yyyy-MM-dd HH:mm:ss.SSS",
"dateTimeFields": ["date", "time"],
"timezone": "UTC",
"tagFields": ["host"]
}'
Humio currently supports two types of parsers:
When using the “json” parser type, Humio expects data to be in the JSON format.
The only required field is dateTimeFields
. This designates which
field has the timestamp.
Here’s an example:
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
-XPUT \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"kind": "json", "dateTimeFields": ["@timestamp"] }'
The “regex” parser type allows you to specify parsers using regular expressions, where named capture groups specify fields.
This example shows how to parse Nginx access logs with regular expressions.
Note the use of \\
to escape backslashes:
(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?
To delete a parser from a given repository, make the following request:
DELETE /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME
Standard HTTP response codes, where $YOUR_HUMIO_URL
is the URL for your
Humio Cloud Account.
Here’s an example:
curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
-XDELETE \
-H "Authorization: Bearer $API_TOKEN"