Logstash

Logstash is an established open source tool for collecting logs, parsing them and outputting them to other systems.

You can use Logstash alongside Humio to process and analyze logs together. In this scenario, you use Logstash as the log collection and parsing agent, and instruct it to send the data to Humio.

Humio supports the ElasticSearch bulk insertion API Just point the Elastic outputter to Humio.

The benefit of this approach is that you can take advantage of the extensible architecture of Logstash to parse many kinds of data:

  • You can install one of the many available plugins that can parse many well-known data formats.
  • You can use the Grok language to build custom parsers for unusual data formats. Grok has many built-in patterns to help parse your data.

Installation

To download Logstash, visit the Logstash downloads page.

You can find the complete documentation for Logstash at the Reference page of the official Logstash website.

Configuration

Because Humio supports parts of the ElasticSearch insertion API, you can use the built-in elasticsearch output in the Logstash configuration.

The following example shows a very simple Logstash configuration that sends data to Humio:

input{
  exec{
    command => "date"
    interval => "5"
  }
}
output{
  elasticsearch{
    hosts => ["https://$BASEURL/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch/"]
    user => "$INGEST_TOKEN"
    password => "notused" # a password has to be set, but Humio does not use it
  }
}

Adding tags to events

Please read the section on tags before adding tags to your events. Add tags by including them in the “inputs/exec” section:

input{
  exec{
    command => "date"
    interval => "5"
    add_field => { "[@tags][customer]" => "CustomerName" }
  }
}
Where:

  • $BASEURL - is the base URL of your Humio server (e.g. https://cloud.humio.com:443 or http://localhost:8080)
  • $REPOSITORY_NAME - is the name of your repository on your server (e.g. sandbox)
  • $INGEST_TOKEN - is the ingest token for your repository, (e.g. a string such as fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii).

Logstash uses 9200 as the default port, if no port is specified. So if Humio is listening on the default ports 80 or 443, these ports should be explicitly put in the $BASEURL

In the above example, Logstash calls the Linux date command every five seconds. It passes the output from this command to Humio.

Field mappings

When you use the ElasticSearch output, Logstash outputs JSON objects. The JSON for an event sent to Humio with the above configuration looks like this:

{
  "@timestamp": "2016-08-25T08:34:37.041Z",
  "message": "Thu Aug 25 10:34:37 CEST 2016\n",
  "command": "date"
}

Humio maps each JSON object into an Event. Each field in the JSON object becomes a field in the Humio Event.

Humio treats some fields as special cases:

Name Description
@timestamp This field must be present, and contain the timestamp in ISO 8601 format. This format is: yyyy-MM-dd'THH:mm:ss.SSSZ.

You can specify the timezone (like +00:02) in the timestamp. Specify the time zone if you want Humio to save this information. Logstash adds the @timestamp field automatically.

Depending on the configuration, the timestamp can be the time at which Logstash handles the event, or the actual timestamp in the data. If the timestamp is present in the data, you can configure logstash to parse it, for example, by using the date filter.
message If present, Humio treats this field as the rawstring of the event.

Humio maps this field to the @rawstring field, which is textual representation of the raw event in Humio.

If you do not provide the message or rawstring field, then the rawstring representation is the JSON structure as text.
rawstring This field is similar to the message field.

If you provide both fields, then Humio uses the message field. The reason for having both is that some Logstash integrations automatically set a message field representing the raw string.

In Humio, we use the name rawstring.

Dropping fields

Logstash often adds fields like host and @version to events. You can remove these fields using a filter and the drop_field function in Logstash.