Storage Rules

In Humio, data is distributed across the cluster nodes. Which nodes store what is chosen randomly. The only thing you as an operator can control is how big a portion is assigned to each node, and that multiple replicas are not stored on the same rack/machine/location (to ensure fault-tolerence).

Data is stored in units of segments, which are compressed files between 0.5GB and 1GB.

Storage Partitions

A cluster will divide data into partitions (or buckets); we cannot know exactly which partition a given data segment will be put in. Partitions are chosen randomly to spread the data evenly across nodes.

The number of partitions is configurable but that is not important initially — the default number is 48 partitions.

Configuring Storage Rules

Data is distributed according to the cluster’s storage rules. A Storage Rule is a relation between a storage partition and the set of nodes that should store all data written to that partition.

When a digest node completes a data segment file (the internal data unit in Humio), it is assigned to a random storage partition. Here is an example configuration:

Example Storage Rules

Partition ID Nodes
1 1,2
2 3,4
3 1,2

In this example the cluster is configured with three storage partitions and four nodes. Nodes 1 and 2 will receive 23 of all data written to the cluster, while nodes 3 and 4 only store 13 of all data. This is because 1 and 2 archive all data in partitions 1 and 3, while nodes 3 and 4 only archive the data in partition 2.

Replication Factor

Notice that in the example above there are the same number of nodes per partition. This is because we want a replication factor of 2, meaning that all data is stored on two nodes. If you had a partition with only one associated node, the replication factor would effectively be one for the entire cluster. This is because you cannot know which data goes into an given partition — and it does not make sense to say that a random subset of the data should only be stored in one copy.

If you want fault-tolerence, you should ensure your data is replicated across multiple node, physical servers, and geographical locations.

UI for Storage Nodes

The Cluster Nodes Administration Page can be found under the Account Menu by selecting Administration > Cluster Nodes. On the right-hand side of the screen the Storage Rules Table is displayed.

Storage Rules, a cluster of 3 nodes where each node is assigned to 2 out of 3 archive partitions leading to a replication factor of 2.

Storage Rules, a cluster of 3 nodes where each node is assigned to 2 out of 3 archive partitions leading to a replication factor of 2.

There is also a tab for Digest Rules and it is important to understand that the Digest Partitions and Storage Partitions are not related in any way. For example, a Digest Partition with ID=1 does not contain the same data as are written to the Storage Partition with ID=1.

Storage Divergence

Humio is capable of storing and searching across huge amounts of data. When cluster nodes join or leave the cluster, data will usually need to be moved between nodes to ensure the replication factor is upheld and that no data is lost.

If your system contains very large amounts of data you cannot simply shuffle it around whenever a node leaves or enters the system. That is because moving terabytes or petabytes of data over the network can take a very long time and potentially impact system performance if done at the wrong time.

Data is stored in Humio according to the cluster’s Storage Rules, but when these rules are changed, for example when a storage node fails and is removed from the cluster, data is not automatically redistributed to match the new ruleset.

In other words the Storage Rules only apply to new data that is ingested. This means that data can end up being stored in fewer replicas than the configured replication factor. This is not necessarily a bad thing — it depends on how strict your replication requirements are.

You can always redistribute it to match the current rules, but it is done as a separate step from changing rules.

At the top of the Cluster Node Management UI you can see the “Storage Divergence” indicated. This will in effect be the amount of data that will need to be sent between nodes in order to make the all the cluster’s data conform to the current rules.

Example: Changing Retention Factor only applies to new data

Say you have a cluster and want to increase your replication factor to four instead of the current two replicas.

This would require having four nodes in each storage rule — which sets the replication factor to four.

This change will ONLY APPLY TO NEW DATA ENTERING THE SYSTEM; all existing data will only be kept in two copies!

The reason for this is that the increased replication factor would mean that all data in the entire cluster would have to be transmitted between nodes. In a cluster with a large amount of data this might not be what you want.

Example: A node is removed uncleanly

If you remove a node from the cluster without first handing over the node’s data to other nodes, there will be one less version of whatever data it was holding.

In this case that effective data distribution will diverge from the current rules, indicated by the Too Low segment of the replication bar in the Cluster Management UI.

Redistribute Data to match the current Storage Rules

If you want to make your make your effective data distribution match the current storage rules you can use the Cluster Management UI.

At the bottom of the Storage Rules Panel on the right-hand side of the screen you can click Show Options, here you will be offered the option to Start Transfers.

If you click it you will see that the Traffic column of the nodes will indicate the shuffling of data around the cluster.

If you make a mistake, you can always undo the change and click “Start Transfers”, effectively undoing the change.

Storage Metrics within the Humio UI

These metrics are applicable to both cluster and single node installations.

On the front page of Humio, there is a Cluster Stats box that gives you information regarding how much data is in the cluster. These storage statistics are meant to represent the ‘searchable data’ within Humio. This means that

  • It includes the compressed ingested data that is found within Humio segment files.
  • It includes Humio’s own system data.
  • It does not include the duplicated/replicated data.

Administration – Cluster Nodes UI – Nodes

Under the Cluster Administration tab, you can see the list of Humio nodes that display Size information.

  • The green part of this bar is ‘Humio Data’.
  • The darker grey is all your other data on that node.
  • The lighter grey is free space.

What Humio Data means is this context is that

  • It includes the compressed ingested data that is found within Humio segment files.
  • It includes the duplicated/replicated data.
  • It does not include the Humio’s own system data.

Administration – Cluster Nodes UI – Replication

Under the Cluster Administration Tab at the top of the page under Replication, you can find information regarding your Replication. Perfect means the total size of segment files that meet the replication factor. Low is the total size of the segment files that are less than the replication factor. Absent means Humio knows about these segment files but can’t find them on any of the nodes. The total size that is displayed within these boxes

  • includes the compressed ingested data that is found within Humio segment files.
  • does not include additional duplicated/replicated data.
  • does not include Humio’s own system data.