Ansible Roles Used With Humio

Introduction

Ansible is an automation tool that simplifies configuration management, program setup, cloud computing, and intra-service operations using YAML to define automation jobs in plain terms. It communicates over SSH to send and retrieve and information to and from remote machines.

Ansible is also a great way of managing a Humio cluster. This document contains details on the Ansible roles used with Humio. For step-by-step instructions on installing Humio using Ansible, please see Installing Humio on bare metal using Ansible.

Below is a list of Ansible Galaxy roles, as well as a few sample projects that demonstrate how they are used.

Ansible Galaxy Roles

Humio is actively maintaining the following three roles

Additionally, Humio recommends using the following roles

These sample projects demonstrate how Ansible Galaxy roles are used

The easiest way to refer to these roles is using a requirements.yml file in the root of your Ansible project

- src: humio.java
- src: AnsibleShipyard.ansible-zookeeper
- src: humio.kafka
- src: humio.server
- src: https://github.com/elastic/ansible-beats

humio.java

The purpose of this role is to install Azul Zulu OpenJDK, which is required by Zookeeper, Kafka, and Humio. The defaults will work for most users, so no additional configuration is required.

AnsibleShipyard.ansible-zookeeper

This is a third-party role that the Humio team has used with good results.

The following variables should be configured for the Zookeeper role:

  • zookeeper_hosts, array consisting of objects containing
    • id, the Zookeeper host id, usually a number between 1 and 3
    • host, the IP address of the host

There are several options for automating. Please see the Cluster example.

We recommend having at least three Zookeeper nodes for high-availability.

humio.kafka

Kafka is at the heart of a Humio installation. Although the use of this exact role isn’t strictly necessary, it’s highly recommended since the Humio team will be maintaining the configuration defaults for Kafka.

For optimal performance, it is a good idea to have one Kafka instance per Humio server in your cluster.

The configuration of this role is similar to Zookeeper, with only a few required variables adjusted:

  • kafka_broker_id, a unique number identifying the instance
  • zookeeper_hosts, exactly similar to the same variable in the Zookeeper role

For additional details on configuration, please review Humio’s Ansible-Kafka GitHub repository.

humio.server

Humio is installed using the humio.server role.

The configuration of the role is similar to the Zookeeper and Kafka roles, with only a few required variables adjusted:

  • zookeeper_hosts, exactly the same variable in the Zookeeper role
  • kafka_hosts, exactly the same variable in the Kafka role

In addition, we recommend keeping the humio_version variable up-to-date to maintain compatibility.

For more details on configuration, please see Humio’s Ansible-Server GitHub repository.

Ansible beats

This is a third-party role maintained by Elastic, the makers of Beats. Currently it’s not pushed to Ansible Galaxy as an official role.

The configuration of this role is straightforward, but we strongly recommend reading its documentation.

Humio recommends the following configuration for Humio nodes:

- role: "ansible-beats"
  beat: "filebeat"
  beat_conf:
    "filebeat":
      "inputs":
        - paths:
            - /var/log/humio/*/humio-debug.log
            fields:
              "@type": humio
              "@tags": ["@host", "@type"]
            multiline:
              pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
              negate: true
              match: after
        - paths:
            - /var/log/kafka/server.log
            fields:
              "@type": kafka
              "@tags": ["@host", "@type"]
            multiline:
              pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
              negate: true
              match: after
      output_conf:
        "elasticsearch":
          "hosts": ["localhost"]
          "username": "developer" //Don't forget to replace this with a real ingest token