Built-in Parsers

Humio® supplies built-in parsers for common log formats, including a parser for the widely-used accesslog format for web servers like Apache and Nginx.

You can examine each of the built-in parsers directly in the Humio Web UI. Open the parser’s page and check the supported regular expression and timestamp formats. When you paste in test data, Humio shows the result of that parsing.

When shipping data to Humio, you want to check to see if there is a built-in parser for the logs before writing a custom parser. The built-in parsers are a good starting point when creating custom parsers.

accesslog

This parser can handle the accesslog format, which is the default log format used by Apache and Nginx. The parser also supports putting the response time at the end of the log line.
If you have modified the logging of your web server, copy the built-in accesslog parser and modify it to suit your customizations.

Example Input

localhost - - [25/Feb/2017:21:05:16 +0100] "POST /api/v1/ingest/elastic-bulk HTTP/1.1" 200 50 "-" "Go-http-client/1.1" 0.000 848`
192.168.1.102 - - [25/Feb/2017:21:06:15 +0100] "GET /api/v1/repositories/gotoconf/queryjobs/855620e9-1d1f-4b0e-91fe-c348795e68c9 HTTP/1.1" 200 591 "referrer" "Mozilla/5.0" 0.008 995

bro-json

This parser can process JSON data generated from Bro. This parser is tailored to handle the output generated from the Bro script, and you can read about how to send Bro data to Humio here.

The name of the Bro log file will become a #path tag in Humio.

corelight-json

This is a built-in parser that supports Corelights Bro sensors. Corelight sensors have default support for streaming out Bro logs. Humio can receive the streaming data using this parser and ingest listeners.

json

This parser can process JSON data in log lines. It expects to find a JSON property called @timestamp containing an ISO 8601-formatted time string.

If you don’t have control over the JSON format, you can create a custom JSON parser.

Example Input

{
  "@timestamp": "2017-02-25T20:18:43.598+00:00",
  "loglevel": "INFO",
  "service": {
	"name": "user service",
	"time": 123
  }
}

kv

This parser is the key-value parser. It can process standard key-value patterns in log lines. It expects the log line to start with a date in ISO 8601 format. The parser processes the rest of the line for key-value pairs.

Example Input

2017-02-25T20:18:43.598+0000 created a new user user="John Doe" service=user-service as a freemium user

Given the above log line, Humio parses the fields user=John Doe and service=user-service.

kv-millis

This parser, like the kv parser, this is a key-value parser. However, it expects the timestamp in the log line to be UTC time in milliseconds.

Example Input

1488054417000 created a new user user="John Doe" service=user-service as a freemium user

Given the above log line, Humio parses the fields user=John Doe and service=user-service.

serilog

This parser can process log lines written by Serilog’s JsonFormatter.

Example serilog configuration:

 Log.Logger = new LoggerConfiguration()
                .WriteTo.File(formatter: new JsonFormatter(renderMessage: true), path:logPath, rollingInterval: RollingInterval.Day)

Note the required renderMessage: true part of the configuration. Humio will display the rendered message output by Serilog instead of the raw event.

Example Input

{"Timestamp":"2019-01-21T13:26:25.1354930+01:00","Level":"Information","MessageTemplate":"Processed {@Position} in {Elapsed:000} ms.","RenderedMessage":"Processed { Latitude: 25, Longitude: 134 } in 034 ms.","Properties":{"Position":{"Latitude":25,"Longitude":134},"Elapsed":34,"ProcessId":"15133"},"Renderings":{"Elapsed":[{"Format":"000","Rendering":"034"}]}}

Properties output by Serilog are available within the parsed event, such as Properties.Position.Latitude from the above example input.

syslog

This parser aims to be compatible with a variety of syslog formats. This includes RFC 3164 and RFC 5424. The parser does not implement every aspect of the syslog RFCs, but is instead liberal in what it accepts.

Example Input

<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for foo on /dev/pts/8
<34>1 2003-10-11T22:14:15.003Z server1.com sshd – – pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.
<34>Oct 11 22:14:15 mymachine su: 'su root' failed for foo on /dev/pts/8
Oct 11 22:14:15 su: 'su root' failed for foo on /dev/pts/8

If no timezone is specified, as in the last two examples, the parser defaults to UTC time. To change that, you may create a new parser by copying this parser and modifying timezone="UTC" to your desired timezone.

The parser also leverages Humio’s built-in key-value parser.

The parser syslog-utc will have better performance when the logs display this specific format.

syslog-utc

This parser can process standard lines generated by the syslog system.

The parser expects lines to starts with a timestamp, followed by the optional fields host, app, and pid. It also expects the timestamp to be in the UTC time zone. If your timestamps are in your local timezone, and that is NOT UTC, you will want to create a new parser by copying this parser and modifying timezone="UTC" to your desired timezone.

This parser also leverages Humio’s built-in key-value parser.

Example Input

Feb 25 19:17:01 Myhost CRON[24886]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Feb 25 06:35:01 Myhost CRON[24272]: (root) CMD (command -v deb-sa1 > /dev/null && deb-sa1 1 1)