JVM Configuration

Humio runs on the Java Virtual Machine (JVM). In this section we describe things you should consider when selecting and configuring your JVM for Humio.

Which JVM?

Humio requires a Java version 11 or later JVM to function properly. We operate Humio Cloud using the Azul-provided Zulu JVM version 13. Our Docker container uses this JVM as well.

We recommend you use one of the following well-tested distributions of the Java runtime when operating Humio.

Java version 11

Provider Name Architectures
Amazon AWS OpenJDK 11 Corretto x86_64
AdoptOpenJDK.net OpenJDK 11 (HotSpot) x86_64
Azul Systems OpenJDK 11 Zulu x86_64
BellSoft OpenJDK 11 Liberica x86_64, ARMv8
Oracle Java SE 11 x86_64
Oracle OpenJDK 11 x86_64

Java version 12

Provider Name Architectures
Azul Systems OpenJDK 12 Zulu x86_64
AdoptOpenJDK.net OpenJDK 12 (HotSpot) x86_64
BellSoft OpenJDK 12 Liberica x86_64, ARMv8

Java version 13

Provider Name Architectures
Azul Systems OpenJDK 13 Zulu x86_64
AdoptOpenJDK.net OpenJDK 13 (HotSpot) x86_64
BellSoft OpenJDK 13 Liberica x86_64, ARMv8

What about…

  • Open J9 — We’ve not yet qualified Humio on OpenJ9 (vs HotSpot) and so cannot recommend using it as yet.
  • Azul Zing — We have tried Zing 11 and in our testing thus far the C4 (or GPGC) garbage collector and Falcon C2 JIT work, providing predictable, low pause collections and efficient execution.
  • Oracle’s Graal and SubstrateVM — Is an interesting alternative to the C2 HotSpot JIT in the OpenJDK. It is not yet supported for production use with Humio. We plan to investigate and support it as it matures.

Java memory options

We recommend that systems running Humio have as much RAM as possible, but not for the JVM. Humio will operate comfortably within 10 GB for most workloads. The remainder of the RAM in your system should remain available for use as filesystem page cache.

A good rule of thumb calculation for memory allocation is

  • (8 GB baseline + 1 GB per core) + that much again in off-heap memory

So, for a production installation on an 8 core VM, you would want about 16 GB of memory with JVM settings as follows

-server -Xms16G  -Xmx16G -Xss2M -XX:MaxDirectMemorySize=16G

This sets Humio to allocate a heap size of 16 GB and further allocates 16 GB for direct memory access (which is used by direct byte buffers). That will leave a further 32 GB of memory for OS processes and filesystem cache. For large installations, more memory for filesystem cache to use will translate into faster queries, so we recommend using as much memory as is economically feasible on your hardware.

For a smaller, two-core system that would look like this

-server -Xms10G  -Xmx10G -Xss2M -XX:MaxDirectMemorySize=10G

That sets Humio to allocate a heap size of 10 GB and further allocates 10 GB for direct memory access (as such, you would want a system with 32 GB of memory, most likely).

It’s definitely possible to run Humio on smaller systems with less memory than this, but we recommend a system with at least 32 GB of memory for all but the smallest installations.

To view how much memory is available for use as filesystem page cache, you can run the following command

$ free -h
			  total        used        free      shared  buff/cache   available
Mem:           125G         24G        1.7G        416K         99G         99G
Swap:           33G         10M         33G

The memory displayed in the available column is what’s currently available for use as page cache. The buff/cache column displays how much of that memory is currently being used for page cache.

If you’re installing on a system with two CPU sockets using our Ansible scripts, then you will end up with two Humio JVM processes running on your system. Under these conditions, the memory requirement will double, so keep that in mind when planning.

Garbage collection

We routinely test Humio against available garbage collectors including Garbage First (-XX:+UseG1), old parallel (-XX:+UseParallelOldGC), and parallel (-XX:+UseParallelGC) collectors, and are investigating others such as Shenandoah, and Zero GC. These work well with our standard workload. While optimized not to allocate objects unless necessary, Humio does still incur a good deal of memory pressure and class unloading due to the nature of Scala on the JVM. The preference, when considering GC options, is for throughput over predictable low latency, meaning that Humio as a solution is tolerant to pauses induced by GC collections. That said, newer concurrent GC algorithms are getting better at balancing these two competing requirements and offer a more predictable application experience.

A key requirement of any GC is that it return unused memory to the operating system, as we depend on the filesystem cache for some of the system performance.

Regardless of which collector you use, we recommend that you configure the JVM for verbose garbage collector logging and then store and monitor those logs within Humio itself.

-Xlog:gc+jni=debug:file=/var/log/humio/gc.log:time,tags:filecount=5,filesize=102400

There are a few helpful flags when running the JVM with Humio that should improve performance and stability. Our current configuration for testing consists of the following combination of flags running a single JVM (Azul Zulu OpenJDK 13) on a non-NUMA system with 256 GB of RAM.

-server -Xms32G -Xmx32G -Xss2M -XX:MaxDirectMemorySize=64G -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:+ClassUnloadingWithConcurrentMark -XX:+ScavengeBeforeFullGC -XX:+UseTLAB -XX:+ResizeTLAB -XX:+ExplicitGCInvokesConcurrent -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+ExitOnOutOfMemoryError -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Dsun.net.inetaddr.ttl=60 -Dnetworkaddress.cache.ttl=60 -Dsun.net.inetaddr.negative.ttl=10 -Dakka.log-config-on-start=on -XX:+UnlockDiagnosticVMOptions -XX:-UseBiasedLocking --add-exports java.base/jdk.internal.util=ALL-UNNAMED -XX:CompileCommand=dontinline,com/humio/util/HotspotUtilsJ.dontInline -Xlog:safepoint*,gc*,gc+ref=debug,gc+heap=debug,gc+age=trace:file=/data/logs/gc_humio.log:time,tags:filecount=5,filesize=102400

Shenandoah GC

ShenandaohGC is a new, fully concurrent, non-generational garbage collector developed for the OpenJDK. As of JDK 13 we find that it provides a low and predictable pause time while not impacting throughput and will return memory to the operating system when not in use. While not available in all Java distributions (most notably Oracle’s), it is a very good choice for Humio. To enable it, and again we recommend JDK 13 or later for this GC, use the following flags:

-XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC -XX:+ClassUnloadingWithConcurrentMark

The Z Garbage Collector (ZGC)

ZGC is also relatively new within the JVM and also has features favorable to Humio. We have discovered that the ZGC (JDK 11) reserves memory as “shared memory” which has the effect of lowering the amount available for disk caching. As Humio is generally IO bound, the ability to cache as much of the block device into RAM is related to providing lower latency and higher throughput. We recommend against using the ZGC until we have tested the implications of the JEP 351 which we hope addresses this issue.

-XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:+ClassUnloadingWithConcurrentMark

Verify physical memory is available for filesystem page cache

Once you have Humio (and perhaps also Kafka, Zookeeper, and other software) running on your server, verify that there is ample memory remaining for caching files using the command free -h. On a server with 128 GB of RAM we usually see around 90 GB as “available”. If the number is much lower, due to a large amount being either “used” or “shared”, then you may want to improve on that. However, if you have a fast IO subsystem, such one based on a RAID 0 stripe of fast NVMe drives, you may find that using memory for caching has no effect on query performance.

You can check by dropping the OS file cache using sudo sysctl -w vm.drop_caches=3 which will drop any cached files, and then compare the speed when running the same trivial query multiple times. Using the same fixed time interval, query of a simple count() twice on a set of data that makes the query take 5-10 seconds to execute is a good test. If you benefit from the page cache you will see a much faster response on the second and following runs compared to the first run.

Another way to validate that the IO subsystem is fast is to inspect the output of iostat -xm 2 while running a query after dropping filesystem page cached data as shown above. If the NVMe-drives are close to 100% utilized, then you will benefit from having memory for page caching.

Java’s file encoding and character set

Humio requires UTF-8 encoding to operate properly. Due to historical choices, the default encoding in almost all of the Java distributions available today is ANSI_X3.4-1968 for file encoding and US-ASCII for the character set (which, amongst other things, determines sort order).

Azul Systems ships two JVM versions, one commercial (Zing) and one open source (Zulu), both of which ship with these two properties defaulting to the more modern and commonly used UTF-8 encoding by default. If you are not running one of these two JVMs with Humio, be sure to include the following options in your configuration.

-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8

If you are unsure what encoding your JVM is using, you can set the flags anyway, or you can to test your distribution using the following instructions.

  1. Create a file in /tmp called HelloWorld.java with the following content.

    import java.nio.charset.Charset;
    
    	   public static void main(String[] args) {
    		   System.out.println(String.format("Hello, World! %s\n", System.getProperty("java.version")));
    		   System.out.println(String.format("file.encoding: %s\n", System.getProperty("file.encoding")));
    		   System.out.println(String.format("defaultCharset: %s\n", Charset.defaultCharset().name()));
    	   }
    }
    
  2. Compile it using javac, then launch the program using your installed JVM, or test using JVMs provided as Docker containers as shown below. Here’s how:

    $ cd tmp
    
    $ vi HelloWorld.java # add code below...
    
    $ javac HelloWorld.java
    
    $ docker run -it --rm -v `pwd`:/data bellsoft/liberica-openjdk-debian:latest java -classpath /data HelloWorld
    Hello, World! 13.0.2
    
    file.encoding: ANSI_X3.4-1968
    
    defaultCharset: US-ASCII
    
    $ docker run -it --rm -v `pwd`:/data bellsoft/liberica-openjdk-debian:latest java -classpath /data -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 HelloWorld
    Hello, World! 13.0.2
    
    file.encoding: UTF-8
    
    defaultCharset: UTF-8
    
    $ docker run -it --rm -v `pwd`:/data azul/zulu-openjdk:latest java -classpath /data -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 HelloWorld
    Hello, World! 1.8.0_242
    
    file.encoding: UTF-8
    
    defaultCharset: UTF-8
    
    $ docker run -it --rm -v `pwd`:/data azul/zulu-openjdk:latest java -classpath /data HelloWorld
    Hello, World! 1.8.0_242
    
    file.encoding: UTF-8
    
    defaultCharset: UTF-8
    
    $
    

Observe that in the tests above, the default encoding for the OpenJDK from Liberica is ANSI_X3.4-1968 and the character set is US-ASCII. This is normal and has been historically the JVM default: also it is not what Humio requires to operate properly in production.

Charsets and character encoding

Humio requires two properties (file.encoding and sun.jnu.encoding – which default to ANSI_X3.4-1968 and US-ASCII) to be set to UTF-8. This is done on the command line by adding -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8. Humio tests for these settings at start-up and fails to start when they are wrong.

What follows is a simple example that highlights this issue:

import java.nio.charset.Charset;

public class HelloWorld {
    public static void main(String[] args) {
	System.out.println(String.format("Hello, World! %s", System.getProperty("java.version")));
	System.out.println(String.format("file.encoding: %s", System.getProperty("file.encoding")));
	System.out.println(String.format("defaultCharset: %s", Charset.defaultCharset().name()));
    }
}

Below are a few sample runs using two different JDK distributions. The Azul Zulu distribution has deviated from the standard and ships with these properties defaulting to UTF-8 but most other distributions have preferred to remain in lock step with the standard.

$ docker run -it --rm -v `pwd`:/data azul/zulu-openjdk:latest java -classpath /data HelloWorld
Hello, World! 1.8.0_242
file.encoding: UTF-8
defaultCharset: UTF-8
 $ docker run -it --rm -v `pwd`:/data bellsoft/liberica-openjdk-debian:latest java -classpath /data HelloWorld
Hello, World! 13.0.2
file.encoding: ANSI_X3.4-1968
defaultCharset: US-ASCII
$ docker run -it --rm -v `pwd`:/data bellsoft/liberica-openjdk-debian:latest java -classpath /data -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 HelloWorld
Hello, World! 13.0.2
file.encoding: UTF-8
defaultCharset: UTF-8

Transparent Huge Pages (THP) on Linux Systems

Huge pages are helpful in virtual memory management in Linux systems. As the name suggests, they help in managing pages larger than the standard 4 KB.

In virtual memory management, the kernel maintains a table to map virtual memory addresses to physical addresses. For every page transaction, the kernel needs to load related mapping. If you have small sized pages, then you need to load more pages, which results in the kernel loading more mapping tables. This decreases performance.

Using huge pages means you will need fewer pages. This decreases the number of mapping tables loaded by the kernel, which increases your kernel level performance, ultimately benefitting your application.

In short, enabling huge pages means less system overhead to access and maintain them.

Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.

To find out what’s available and being used on your Linux system you can as root run some of the following commands:

# grep Huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       1024 kB
# egrep 'trans|thp' /proc/vmstat
nr_anon_transparent_hugepages 2018
thp_fault_alloc 7302
thp_fault_fallback 0
thp_collapse_alloc 401
thp_collapse_alloc_failed 0
thp_split 21
# cat /proc/sys/vm/nr_hugepages
1024
# sysctl vm.nr_hugepages
vm.nr_hugepages = 1024
# grep -i HugePages_Total /proc/meminfo
HugePages_Total:       1024
# cat /proc/sys/vm/nr_hugepages
1024
# sysctl vm.nr_hugepages
vm.nr_hugepages = 1024

Make sure that grub.conf doesn’t include: transparent_hugepage=never.

The JVM flag to enable the use of this feature is -XX:+UseTransparentHugePages.

One way to add configuration to your Linux system to allow applications to use huge pages, the simplest is to add the following to your /etc/rc.local file.

madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo advise > /sys/kernel/mm/transparent_hugepage/shmem_enabled
echo defer > /sys/kernel/mm/transparent_hugepage/defrag
echo 1 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

Note that if you are using containers (Docker) then you’ll have to configure huge page support in the operating system that is running the containers, not the containers themselves.

Docker/Containerized Execution

When running a JVM inside a container, such as when using Docker, you’ll want to ensure that the JVM doesn’t try to overallocate memory available to it. A new feature of the JVM allows for discovery of that limit by adding the following flags at startup.

-XX:+UseCGroupMemoryLimitForHeap
-XX:MaxRAMFraction=1

NUMA (multi-socket) systems

Humio fully utilizes the available IO channels (network and disk), physical memory, and CPU during query execution. Coordinating memory across cores will slow Humio down, leaving hardware resources underutilized. NUMA hardware is more sensitive to this.

Non-uniform memory access (NUMA) on multisocket hardware systems is challenging for multithreaded software with mutable shared state. Multithreaded object-oriented systems fit this description and so this is something to be aware of. Thankfully the JVM is our platform and has some support for NUMA since version 6 (aka 1.6), but limited support for those optimizations across garbage collectors.

There are two strategies for NUMA deployment

  1. Use the operating system to split each socket into a separate logical space, and
  2. Configure the JVM to be NUMA-aware in hopes that it will make good choices about thread and memory affinity.

Strategy 1: Run one JVM per NUMA node

The intent here is to pin a JVM process to each NUMA node (not CPU, not core, not thread) and that node’s associated memory (RAM) sockets only, and thereby avoid the overhead of cross-NUMA-node coordination (which is expensive) using tools provided by the operating system. We have successfully done this on Linux, but certainly other operating systems have similar primitives.

Using this strategy it is important that you do not enable any of the JVM flag related to NUMA.

On Linux you’ll use numactl in your startup script to confine a JVM process to a NUMA node and that node’s memory.

/usr/bin/numactl --cpunodebind=%i --membind=%i --localalloc -- command {arguments ...}

The command being java and arguments being those passed to the JVM at startup.

Pros:

  • You have more Humio nodes in your cluster, which looks cool.
  • You can use any GC algorithm you’d like.
  • We at Humio have deployed on NUMA hardware using this strategy.

Cons:

  • You have more Humio nodes in your cluster, which can be confusing.
  • You’ll use a multiple of the RAM used for the JVM (heap -Xms, stack -Xms) for operating each NUMA node, reducing the available RAM for file system buffers.

Strategy 2: Run one JVM per system with a NUMA-aware garbage collector and proper configuration

The intent here is to run one JVM per system across all NUMA nodes and let the JVM deal with the performance penalties/benefits of such a hardware layout. In our experience, the JVM does not reach the full potential of NUMA hardware when running in this manner but that is changing (as the JVM and the Scala language mature), and we expect someday that it will be a simpler, higher performance, and more efficient configuration.

For this strategy it is important to enable NUMA support in the JVM by specifying the -XX:+UseNUMA option.

Pros:

  • You’ll use less RAM per JVM, leaving more available for filesystem caching.
  • You’ll have less contention on the PCI bus and network hardware.

Cons:

  • You have to choose a NUMA-aware GC algorithm.
  • You have to remember to enable the NUMA-specific code in the JVM.
  • You can’t use any GC algorithm you’d like; you have to choose one that is NUMA-aware.
  • We at Humio have NUMA hardware in production running Humio; we don’t use this strategy for performance reasons.

NUMA-aware Garbage Collectors in the JVM

Collector Version Distribution
ParallelGC JDK8+ *
G1GC JDK14+ *
C4 JDK8+ Azul Zing

A NUMA-aware JVM will partition the heap with respect to the NUMA nodes, and when a thread creates a new object the memory allocated resides on RAM associated with the NUMA node of the core that is running the thread. Later, if the same thread uses that object it will be in cache or the NUMA node’s local memory (read: close by, so fast to access). Also when compacting the heap, the NUMA-aware JVM avoids moving large data chunks between nodes (and reduces the length of stop-the-world events).

The parallel collector (to enable use the -XX:+UseParallelGC flag) has been NUMA-aware for years and works well; it should be your first choice. Should you choose G1GC please also add ‑XX:‑G1UseAdaptiveIHOP as it will improve predictable performance under load and lower GC overhead.

Shenandoah GC does not include support specific to running on NUMA hardware at this time which means that it isn’t suitable for use on such systems.

The “Zero Garbage Collector” (ZGC) has only basic NUMA support which it enables by default on multi-socket systems unless pinned to a single NUMA node.

Reach out; we’re happy to help

Configuring the JVM for optimum performance is a black art, not a science, and Humio will perform vastly differently on the same hardware with different JVM configurations. Please reach out to us for help; this is an important and subtle topic. When deploying Humio, please read carefully and feel free to consult with us through our support channels if you have any concerns or simply want advice.

Helpful Java/JVM resources