Vector

Vector is a lightweight and ultra-fast tool for building observability pipelines. It can be used to replace Logstash, Fluent, Telegraf, Beats, or similar tools.

Vector has built-in support for shipping logs to Humio through the humio_logs sink.

Installation

Vector can be installed on Linux, Windows, and MacOS. The Vector documentation includes several methods of installation.

Configuration

Sending data to Humio with Vector is very easy using the humio_logs sink. We only need the URL of the Humio cluster and an ingest token.

In the example below we configure Vector to read from standard input (stdin) and send each line to the Humio cluster. Messages entered at the command line after starting Vector will be sent to Humio.

  1. Create the file vector.toml

    data_dir = "/var/lib/vector"
    
    # stdin source: https://vector.dev/docs/reference/sources/stdin/
    [sources.my_stdin_source]
    type = "stdin"
    
    # humio sink: https://vector.dev/docs/reference/sinks/humio_logs/
    [sinks.my_humio_cluster]
    # set type to "humio" in version 0.8.0
    # https://github.com/timberio/vector/issues/1971
    inputs = ["my_stdin_source"]
    type = "humio_logs"
    encoding.codec = "json" # optional, possible values = json, text, default = json
    host = "${HUMIO_URL}"
    token = "${HUMIO_INGEST_TOKEN}"
    

By default Vector sends events to Humio as JSON. Vector version 0.9.1 added the option to send logs to Humio in the raw text format by setting the encoding.codec = "text".

  1. Run Vector with the environment variables HUMIO_URL and HUMIO_INGEST_TOKEN set appropriately and enter test messages

    HUMIO_URL=http://localhost:8080 HUMIO_INGEST_TOKEN=KL95YdaSYEWJ1tV9CPEqWGdMi4FVXghD0xxGrDAU3Wg5 vector --config vector.toml
    Mar 04 13:40:19.770  INFO vector: Log level "info" is enabled.
    Mar 04 13:40:19.770  INFO vector: Loading configs. path=["vector.toml"]
    Mar 04 13:40:19.773  INFO vector: Vector is starting. version="0.8.1" git_version="v0.8.1" released="Wed, 04 Mar 2020 15:11:57 +0000" arch="x86_64"
    Mar 04 13:40:19.773  INFO vector::topology: Running healthchecks.
    Mar 04 13:40:19.773  INFO vector::topology: Starting source "my_stdin_source"
    Mar 04 13:40:19.773  INFO vector::topology: Starting sink "my_humio_cluster"
    Mar 04 13:40:19.774  INFO source{name=my_stdin_source type=stdin}: vector::sources::stdin: Capturing STDIN
    Mar 04 13:40:19.781  INFO vector::topology::builder: Healthcheck: Passed.
    Example Message 1
    Example Message 2
    
  2. Search your Humio repository for the test messages

    The messages in Humio will have the following structure. Note that Vector adds timestamp and host to the messages.

    {"@timestamp":1583349673000,"#type":"none","host":"MacBook-Pro.local","#repo":"vector-example","@timezone":"Z","message":"Example Message 2","@rawstring":"{\"host\":\"Daniels-MacBook-Pro.local\",\"message\":\"Example Message 2\"}","@id":"mENFVMQVJyQ2M5pV4D1sFMB9_1_1_1583349673"}
    {"@timestamp":1583349669000,"#type":"none","host":"MacBook-Pro.local","#repo":"vector-example","@timezone":"Z","message":"Example Message 1","@rawstring":"{\"host\":\"Daniels-MacBook-Pro.local\",\"message\":\"Example Message 1\"}","@id":"mENFVMQVJyQ2M5pV4D1sFMB9_1_0_1583349669"}
    

Adding Fields

Vector makes it possible to add fields with static values using its transforms capability (https://vector.dev/docs/reference/transforms/add_fields/). In the example below a field called name will be added to the event sent to Humio with the value set to Name:

[transforms.sourcename_transform]
  type = "add_fields"
  inputs = ["sourcename"]
  fields.name = "Name"

You will need to update the inputs section of your sinks to point the transformation that you created in order for the new field to be added to the event (as illustrated below).

[sinks.humio_out]
  type = "humio_logs"
  inputs = ["sourcename_transform"]
  encoding.codec = "json"
  token = "$api-token"
  host = "$humio-url"

Multiline events

By default, Vector creates one event for each line in the in a file. However, you can also split events in different ways. For example, stack traces in many programming languages span multiple lines.

You can specify multiline settings in the Vector configuration. See Vector’s multiline configuration documentation

Often a log event starts with a timestamp, and we want to read all lines until we see a new line starting with a timestamp. In Vector that can be done like this:

  [sources.source_name.multiline]
    # Example: [4/28/20 14:59:25:783 EDT]
    start_pattern = "^\\[[0-9]{1,2}/[0-9]{1,2}/[0-9]{2}"
    mode = "halt_before"
    condition_pattern = "^\\[[0-9]{1,2}/[0-9]{1,2}/[0-9]{2}"
    timeout_ms = 1000

The start_pattern should match your timestamp format