When sizing Humio, your choice depends on your usage patterns, so we recommend first doing an example setup to see how Humio works with your data. The following provides some examples. For clustered setups, and/or setups with data replication, we currently recommend contacting Humio Technical Support for specific suggestions.
With Humio, the limiting factor is usually query speed, not ingest capacity. Query speed depends upon the number of CPUs, available RAM, and disk speed. The following numbers are all general recommendations; your exact circumstances may be different.
A running Humio instance will use something like 10-20% of available RAM — the rest of available memory
is used for OS-level file system cache. If you’re, for example, on an AWS
i3.8xlarge (244 GB RAM, 32 vCPUs, 4 x 1,900 NVMe SSD), then you would typically see
~32 GB used for running Humio, and the rest is available for file system caches.
i3.8xlarge has fast NVMe disks. In that case data can be read fast enough from disk to keep the CPUs busy. In case of slower disks memory is important for the file system cache.
Data is typically compressed 10-20x, depending on what it is. Your mileage may vary, but short log lines (HTTP access logs, syslog) compress better than longish JSON logs (such as those coming from Metricbeat).
For data available on fast NVMe disks or as compressed data in the OS-level cache, Humio generally provides query speed at 0.5 GB/s/vCPU,
or 0.5 GB/s/hyperthread. So, on a
i3.8xlarge instance with 32 vCPUs, you observe ~16GB/s queries.
This is the search speed when all data needs to be scanned. In many cases our hash indexes will improve the search speed significantly.
In cloud environments, like Google and Amazon, we recommend using fast ephemeral NVMe disks for good search performance. We do not recommend EBS storage as it will be slow and expensive (IOPS). Setup Humio with Bucket storage to keep data durable. Bucket storage stores data in Amazon S3 or Google Cloud Storage. Zookeeper and Kafka should not be run on ephemeral disks! It is important to keep that data durable.
Searches going beyond what fits in fast disks or the OS-level file system caches can be significantly slower, as they depend on disk I/O performance. If you have sufficiently fast NVME-drives or similar that can deliver the compressed data as fast as the CPU cores can decompress them, then there is virtually no penalty for doing searches that extend beyond the size of the page cache. We built Humio to run on local SSDs, so it is not (presently) optimized to run on high-latency EBS storage or slow spinning disks. But it will work, especially if most searches are “live” or within the page cache. Humio reads files using “read ahead” instructions to the Linux kernel, which allows for file systems on spinning disks to read continuous ranges of blocks rather than random blocks.
i3.8xlargeper TB/day data. Disclaimer - This of course depends on the scenario. How many users are using the system and how much data is being searched etc.
Your machine has 64 GB of RAM, 16 hyper threads (8 cores), and 256GB NVMe storage. In this case that means that 10 seconds worth of query time will run through 80 GB of data. So this machine fits an 80 GB/day ingest, with 1 months of data available for fast querying.
You can store 2 TB of data before your disk is 80% full.
For AWS, we recommend starting with these instance types. T
||488||64 (2 CPUs)|
||244||32 (2 CPUs)|
For instance an
i3.4xlarge would be suitable for 150 GB/day ingest, holding 5 days
of data in cache, and because of the SSDs this would be avoiding the “cliff” when
the cache runs full. The 3.8 TB SSD would hold ~150 days of ingest data.
Running many live queries/dashboards is less of an issue with Humio than most other similar products, because these are kept in-memory as a sort of in-memory materialized view. The time spent on updating them is part of the ingest flow and thus having many live queries increases the CPU usage for the ingest process. When initializing such queries, it does need to run a historic query to fill in past data, and that can take some time in particular if it extends beyond what fits within the compressed-data in memory horizon.
FIO is a great tool for testing the IO performance of your system. It is available as a APT package on Ubuntu too:
sudo apt install fio
fio either with all options on the commandline or through a “jobfile”.
sudo fio --filename=/data/fio-test.tmp --filesize=1Gi --bs=256K -rw=read --time_based --runtime=5s --name=read_bandwidth_test --numjobs=8 --thread --direct=1
Here are a sample contents for a jobfile that somewhat mimics how
Humio reads from the file system. Make sure to replace
/data/fio-tmp-dir with the path to somewhere on the disk where your
humio-data would be located, once installed.
[global] thread rw=read bs=256Ki directory=/data/fio-tmp-dir direct=1 [read8] stonewall size=1Gi numjobs=8
fio --bandwidth-log ./humio-read-test.fio # Clean tmp files from fio: rm /data/fio-tmp-dir/read8.?.?
and then look for the lines at the bottom of the very detailed report similar to…
Run status group 0 (all jobs): READ: bw=3095MiB/s (3245MB/s), 387MiB/s-431MiB/s (406MB/s-452MB/s), io=8192MiB (8590MB), run=2375-2647msec
This example is an NVME providing ~3 GB/s read performance. This allows for searching up to (9*3) GB/s of uncompressed data if the system has sufficient number of CPU cores according to the rules of thumb above. This NVME is well matched to a CPU with 32 hyper-threads (16 hardware cores.)