Getting Metrics out of Humio

Humio outputs a number of metrics that can be used to monitor and operate Humio itself.

The metrics are available in different ways.

JMX

Humio can expose all metrics over JMX. To enable this, you need to set the standard JMX options to your JVM by adding them to the HUMIO_JVM_ARGS configuration.

For example

HUMIO_JVM_ARGS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=5000

Prometheus

Setting the PROMETHEUS_METRICS_PORT configuration will enable Prometheus to scrape metrics from Humio.

Humio Debug Logs

The Humio debug log also contains all the metrics. You can find them in the humio repository or the special humio-metrics repository

Metric Types

There are two types of metrics in Humio, node level metrics and objects level metrics. The last type is for objects such as repositories, ingest listeners, or storage partitions. An example of an object level metric is ingest-bytes/<repo>. Here <repo> is a placeholder for a concrete repository in a given Humio system.

Node Level Metrics

Metric Name Description
elastic-search-ingestion-request-errors Number of ingest errors in the elastic-search endpoint since the node started
elastic-search-ingestion-requests Time spent ingesting a bulk using the elasticsearch ingest protocol
event-collector-request-errors Number of ingest errors in the http-event-collector endpoint since the node started
event-latency Overall latency of ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files
global-publish-wait-for-value Time spent waiting to see the value being read back from Kafka when pushing an update to the global state
hashfilter-included-blocks Number of blocks included using hashfilters in queries and thus read from compressed blocks in segment files
hashfilter-skipped-blocks Number of blocks skipped using hashfilters in queries and thus not read from compressed blocks in segment files
http-requests Timing of all inbound http requests
humio-ingestion-request-errors Number of ingest errors in the humio ingestion endpoint since the node started
ingest-listener-tcp-available TCP ingest listener free slots for lines to be processed (High when idle, zero when over-loaded)
ingest-writer-bulksize Histogram of size in number of bytes compressed data of jobs that carry events. Some jobs are no-payload and are not included here
ingest-writer-jobs Number of jobs pushed to in-memory job queue for digest writers
kafka-chatter-bytes Number of bytes written to kafka on the chatter topic
kafka-chatter-put Time waiting for getting ack from Kafka when publishing to the chatter topic
kafka-ingestqueue-put Time waiting for getting ack when adding ingest events to the ingest queue
kafka-request-bytes Number of bytes written to kafka as compressed events for the ingest queue
kafka-request-events Number of events written to kafka as compressed events for the ingest queue
livequery-count Number of live- (real time-) queries active
load-segment-total Time spent reading (waiting for) blocks from segment files
logplex-ingestion-request-errors Number of ingest errors in the logplex endpoint since the node started
mapsegment Time spent on ‘map’ phase while searching non-real time segment files
proxied-query-polls Timing of internal requests due to polling of queries not hitting the server coordinating the query
queries Total number of queries started since this node started
query Measure how long it takes for queries to complete
read-compressed-bytes Number of bytes of read from compressed blocks in segment files
read-prefilter-bytes Number of bytes of read from pre-filter files
recompress-millis Number of milliseconds CPU time spent merging and re-compressing segment files
s3-archiving-bytes-per-second Bytes archived in S3 per second
s3-archiving-errors-per-second Errors per second archiving logs in S3
s3-archiving-writes-per-second Successful S3 archival writes per second
schedulesegments Time spent scheduling segment files for the ‘map’ phase while searching non-real time segment files
timestamp-parsing-failed Total number of timestamp strings that did not parse as a time stamp since start of the node

Object Level Metrics

Metric Name Description
current-live-events/<repo> Number of events being processed updating live quries
data-ingester-errors/<repo> Number of events that got an @error tag added to their fields during parsing
event-latency-partition/<repo> Per-partition latency of the ‘humio-ingest’ topic of the ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files
ingest-bytes/<repo> Number of bytes uncompressed in flushed blocks for segments being constructed
ingest-eventsize/<repo> Number of bytes uncompressed summed over individual events in blocks in progress
ingest-parsing/<repo> Time spent parsing incoming events
ingest-queue-consumer/<repo> Time spent constructing segment file blocks in memory and writing them to disk, including updating live queries if any
ingest-reader-partition-bytes/<repo> Number of bytes read from kafka as compressed events from the ingest queue
ingest-reader-partition-events/<repo> Number of events added to segment file blocks being constructed
ingest-reader-polltime/<repo> Time blocked waiting for next message from Kafka from ingest queue
ingest-writer-partition-bytes/<repo> Number of bytes written to kafka as compressed events into the ingest queue
notifications/<repo> Time spent shipping a notification from an alert
query-millis/<repo> Number of milliseconds spent processing historical queries
repo-queries/<repo> Number of queries started per repo
udpingestbytes/<repo> Number of bytes read by udp ingest listener
written-events-after-queue/<repo> Number of events added to segment file blocks being constructed
written-events/<repo> Number of events written to the ingest queue after being parsed