Removing a Node

When you want to remove a node from a cluster in Humio®, you need to make sure that any digest and archiving responsibilities are transferred to another node. This means removing the node from any digest and archive rules. This will stop the node from accepting any new work. The data stored on the node would be copied to another node to keep the cluster’s replication factor stable.

To safely remove a node from a Humio cluster you need to ensure that the data stored on the node is completely copied to another node before it leaves.

We will be using the Cluster Node UI in this guide, but everything can be automated using the Cluster Management GraphQL API We will also be listing the associated HTTP calls performed in each step.

The Cluster UI will indicate that a node is safe to remove from the cluster with a Yes in the Removable column.

Step 1: Un-assign digest rules

If the node is being used for digest (the Digest column has a value greater than 0), you need to stop accepting new digest work by removing the node from all Digest Rules.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes.
  2. Go to Actions, select Stop using this node for digest of incoming data > Remove node from Digest Rules.

This will automatically assign other suitable nodes to the digest partitions currently assigned to the node you want to remove.

Step 2: Un-assign archive rules

If the node is being used for archiving (the Archive column has a value greater than 0), you need to stop archiving new data by removing the node from all Archive Rules.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes.
  2. In the Actions panel, click Stop using this node for storing for incoming data > Remove node from Storage Rules.

This will automatically assign other suitable nodes to the Archive partitions currently assigned to the node you want to remove.

Step 3: Moving all data to other nodes

Finally, you need to move all data archived on the node to other nodes to ensure that the cluster’s replication factor is upheld before the node is removed.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes.
  2. In the Actions panel, click Move all existing data away from this node > Move data out of node.

Step 4: Shut down the Humio process on the node

You should shut down the Humio process on the node, but you must wait until the Size column of the Node List shows 0 B indicating that no more data resides on the node.

Step 5: Unregister from the cluster

Finally, you should see that the Removable column says Yes, and you can unregister the node from the cluster, telling other nodes that the node will not be coming back again.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes.
  2. In the Actions panel, click Remove Node > Remove Node.

Forcing removal

If a node has died and there is no backup and no way to retrieve the data, you can forcibly unregister the node. This means that you will have to accept potential data loss, if no replicas of the data existed.

You can forcibly remove a node by checking Force Remove.