Shipping Humio Logs to another Humio Cluster

When running a Humio cluster in production we highly recommend shipping Humio’s internal logs into another cluster. This is so if you run into any problems with your production cluster you are still able to debug what went wrong.

You can use the humio/insights package to monitor any Humio cluster. This comes with dashboards and saved queries that can be useful into debugging what went wrong with Humio.

This is a guide on how to ship Humio’s internal logs to another Humio cluster.

Prepare the Cluster

This guide assumes you have another Humio cluster ready and setup to receive another Humio clusters’ logs.

  1. Create a repository on your Humio monitoring cluster. This will be where we will ship Humio’s internal logs.

  2. Install the humio/insights package on the just created repository. This will include all the dashboards, queries and parsers used to setup and monitor the other Humio cluster.

  3. Create an ingest token and connect it to the parser named “humio”. This comes as part of the humio/insights package once installed.

  4. Configure a log shipper to send Humio’s logs. See below how to do this.

Configuring a log shipper to send Humio logs

Vector

We recommend sending logs using Vector. It is lightweight agent with built-in support for shipping logs to Humio via the humio_logs sink.

  1. Install Vector on all Humio nodes within the cluster we are going to monitor. There is more information here on how to install vector.

  2. Edit your vector.toml configuration file to the below:

[sources.logs]
  type = "file"
  include = ["${HUMIO_LOGS_DIR}/humio*.log"]
  [sources.logs.multiline]
    start_pattern = "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
    mode = "halt_before"
    condition_pattern = "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
    timeout_ms = 2000

# Humio Sink: https://vector.dev/docs/reference/sinks/humio_logs/
[sinks.humio_cluster]
    type = "humio_logs"
    inputs = ["logs"]
    compression = "gzip"
    host = "${HUMIO_URL}"
    token = "${INGEST_TOKEN}"

In the above configuration you need to replace the following:

  • ${HUMIO_LOGS_DIR} which will be the path to directory containing Humio’s internal logs. Note how globbing (*) is used to specify which files to collect. Example path can look like /data/humio-data/logs.
  • ${HUMIO_URL} with the URL of your Humio cluster being used for monitoring. An example URL should look like so https://cloud.humio.com or https://cloud.us.humio.com.
  • ${INGEST_TOKEN} The ingest token from the repository on the cluster we are going to be using to monitor our Humio cluster.
  1. Start Vector and check the repository for Humio internal logs.

Filebeat

We also support sending Humio’s internal logs via Filebeat. Follow these steps to setup Filebeat to ship Humio’s internal logs to another Humio cluster.

  1. Install Filebeat on all Humio nodes within the cluster we are going to monitor.

  2. Edit your filebeat.yml configuration file to the below:

filebeat.inputs:
- paths:
  - ${HUMIO_LOGS_DIR}/humio-*.log
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
queue.mem:
  events: 8000
  flush.min_events: 1000
  flush.timeout: 1s
output:
  elasticsearch:
    hosts: ["${HUMIO_URL}"]
    password: $INGEST_TOKEN
    compression_level: 5
    bulk_max_size: 200

In the above configuration you need to replace the following:

  • ${HUMIO_LOGS_DIR} which will be the path to directory containing Humio’s internal logs. Note how globbing (*) is used to specify which files to collect. Example path can look like /data/humio-data/logs.
  • ${HUMIO_URL} with the URL of your Humio cluster being used for monitoring. An example URL should look like so https://cloud.humio.com or https://cloud.us.humio.com.
  • ${INGEST_TOKEN} The ingest token from the repository on the cluster we are going to be using to monitor our Humio cluster.
  1. Start Filebeat and check the repository to see if logs have been received.

Sending Humio Logs to Humio Cloud

To assist in monitoring your on-prem instance of Humio it is possible to ship Humio’s logs into Humio Cloud This is convenient for not having to run and maintain another cluster. This also helps as you can share your internal logs with Humio Support.

When getting logs shipped into Humio Cloud you should already be in touch with Humio Support. If it is agreed that we can get your logs setup into Humio Cloud then this is what you should have setup.

Pre-Requisites

  • A Humio Cloud account setup.
  • A repository preferabbly under the format onprem_$orgName_debug. Please contact support if you need a repository created.
  • The humio/insights package installed on your repository.

Configure the log shippers

For both Filebeat and Vector you just need to ensure the ${HUMIO_URL} is set to https://cloud.humio.com for EU Cloud or https://cloud.us.humio.com for US Cloud depending on where your Humio Cloud account is.

The humio-debug.log can contain sensitive information. It contains logs of things like: E-mails of your Humio users, queries, names of repositories, views and parsers, IP addresses and access logs from your Humio nodes. It does not log any of your ingested events. Please ensure you are aware of this before shipping this log file into Humio Cloud.