Removing a Node

To safely remove a node from a Humio cluster you need to ensure that the data stored on the node is completely copied to another node before it leaves.

Steps to removing a cluster node

When you want to remove node from a cluster you need to make sure that any digest and archiving responsibilities. This means removing the node from any Digest and Archive Rules. This will stop the node from accepting any new work. The data stored on the node would be copied to another node to keep the cluster’s replication factor stable.

We will be using the Cluster Node UI in this guide, but everything can be automated using the Cluster Management GraphQL API we will be listing the associated HTTP calls performed in each step.

The Cluster UI will indicate that a node is safe to remove from the cluster with a Yes in the Removable column.

Step 1: Un-assign Digest Rules

If the node is being used for digest (the Digest column has a value greater than 0), you need to stop accepting new digest work by removing the node from all Digest Rules.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes
  2. In the panel labeled Actions click the item labeled Stop using this node for digest of incoming data and then Remove node from Digest Rules.

This will automatically assign other suitable nodes to the digest partitions currently assigned to the node you want to remove.

Step 2: Un-assign Archive Rules

If the node is being used for archiving (the Archive column has a value greater than 0), you need to stop archiving new data by removing the node from all Archive Rules.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes
  2. In the panel labeled Actions click the item labeled Stop using this node for storing for incoming data and then Remove node from Storage Rules.

This will automatically assign other suitable nodes to the Archive partitions currently assigned to the node you want to remove.

Step 3: Moving all data to other nodes

Finally you need to move all archived on the node other nodes to ensure that the cluster’s replication factor is upheld before the node is removed.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes
  2. In the panel labeled Actions click the item labeled Move all existing data away from this node and then Move data out of node.

Step 4: Shut down the Humio Process on the node

You should shut down the Humio process on the node. You must wait until the Size column of the Node List shows 0 B indicating that no more data resides on the node.

Step 5: Unregister from the cluster

Finally you should see that the Removable column says Yes, and you can unregister the node from the cluster - i.e. telling other nodes that the node will not be coming back again.

In the Cluster UI follow these steps:

  1. Select the node you want to remove in the list of nodes
  2. In the panel labeled Actions click the item labeled Remove Node and then click the button Remove Node.

Forcing Removal

If a node has died and there is no backup and no way of retrieval of the data you can forcibly unregister the node. This means that you will have to accept potential data loss, if no replicas of the data existed!

You can forcibly remove a node by checking the checkbox Force Remove in the UI.