Parser API

Data sent to Humio usually has some structure. You can use parsers to extract this structure. For example, a standard web server log has the status code, method, and URL fields for each log line.

When sending data to Humio, for example using Filebeat, you must specify a parser telling Humio how to parse the incoming data.

Humio has some built-in parsers for common formats like access logs from Apache and Nginx web servers. It also allows for custom parsers.

List Parsers

Request

To list all parsers for a given repository:

GET     /api/v1/repositories/$REPOSITORY_NAME/parsers

Response

[
  {
    "builtIn": true,
    "parseKeyValues": false,
    "parser": "(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?",
    "id": "accesslog",
    "dateTimeFields": [
      "@timestamp"
    ],
    "kind": "regex",
    "dateTimeFormat": "dd/MMM/yyyy:HH:mm:ss Z"
  },
  {
    "id": "json",
    "kind": "json",
    "parseKeyValues": false,
    "dateTimeFields": [
      "@timestamp"
    ]
  }
]

The output format is similar to the input format in Add Parser, where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account.

Here’s an example_

curl https://`$YOUR_HUMIO_URL`/api/v1/repositories/$REPOSITORY_NAME/parsers \
 -H "Authorization: Bearer $API_TOKEN"

Create or Update Parser

Request

To create a parser for a given repository:

POST     /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID

or to updated an existing parser use

PUT     /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_ID

The JSON request body has the following attributes:

Name Type Required Description
kind String Yes Controls which parser kind to create. You can set this to regex, or json.
parser String Yes The parser specification.

The contents of this field vary depending on the type of parser you are creating. See the details below
parseKeyValues Boolean No Sets whether you want the parser to parse ‘key=value’ pairs in the log line.

The default value is false.
dateTimeFields Array Yes Specifies the fields which contain the timestamp of the event.

You can specify multiple fields, for example, a date field and a time field. The values of these fields are concatenated with whitespaces.

Humio parses these fields with the format that you specify in the dateTimeFormat attribute.
dateTimeFormat String No The format string that Humio should use to parse the fields identified by the dateTimeFields attribute.

This attribute uses the Java DateTimeFormatter syntax.

The default value is the ISO-8601 format, for example, yyyy-MM-dd'T'HH:mm:ss.SSSZ, with milliseconds as an optional addition.
timezone String No This field is only used if the timestamp of the event is in localtime and does not have a timezone.

In that case, you can use it to set a timezone.

Do not use this field if the timezone is part of the dateTimeFormat.

Examples: UTC, Z, or Europe/Copenhagen.
tagFields Array No Specify fields in events generated by this parser that should be turned into tags.
For example it could be specified that the host field in the events from this parser should be treated as a tag.

Response

Standard HTTP response codes, where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account.

Here’s an example:

curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
 -XPUT \
 -H "Authorization: Bearer $API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"parser": "^(?<date>\\S*) (?<time>\\S*) (?<host>\\S*) (?<appname>\\S*):",
      "kind": "regex",
      "parseKeyValues": true,
      "dateTimeFormat": "yyyy-MM-dd HH:mm:ss.SSS",
      "dateTimeFields": ["date", "time"],
      "timezone": "UTC",
      "tagFields": ["host"]
     }'

Parser Types

Humio currently supports two types of parsers:

JSON Parsers

When using the “json” parser type, Humio expects data to be in the JSON format.

The only required field is dateTimeFields. This designates which field has the timestamp.

Here’s an example:

curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
 -XPUT \
 -H "Authorization: Bearer $API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{"kind": "json", "dateTimeFields": ["@timestamp"] }'

RegExp Parsers

The “regex” parser type allows you to specify parsers using regular expressions, where named capture groups specify fields.

This example shows how to parse Nginx access logs with regular expressions. Note the use of \\ to escape backslashes:

(?<client>\\S+)\\s+-\\s+(?<userid>\\S+)\\s+\\[(?<@timestamp>.*)\\]\\s+\"((?<method>\\S+)\\s+(?<url>\\S+)?\\s+(?<httpversion>\\S+)?|-)\"\\s+(?<statuscode>\\d+)\\s+(?<responsesize>\\S+)\\s+\"(?<referrer>.*)\"\\s+\"(?<useragent>.*)\"\\s*(?<responsetime>.+)?

Delete Parser

Request

To delete a parser from a given repository, make the following request:

DELETE     /api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME

Response

Standard HTTP response codes, where $YOUR_HUMIO_URL is the URL for your Humio Cloud Account.

Here’s an example:

curl https://$YOUR_HUMIO_URL/api/v1/repositories/$REPOSITORY_NAME/parsers/$PARSER_NAME \
 -XDELETE \
 -H "Authorization: Bearer $API_TOKEN"