Humio Metrics

Humio outputs a number of metrics that can be used to monitor and operate Humio itself.

The metrics are available in different ways.

JMX

Humio can expose all metrics over JMX. To enable it, you need to set the standard JMX options to your JVM by adding them to the HUMIO_JVM_ARGS configuration configuration.

Example:

HUMIO_JVM_ARGS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=5000

Prometheus

Setting the PROMETHEUS_METRICS_PORT configuration will enable Prometheus to scrape metrics from Humio.

Humio debug logs

The debug log of Humio also contains all the metrics. You can find them in the humio repository or the special humio-metrics repository

Metric Types

There are two types of metrics in Humio, node level metrics and objects level metrics. The last type is for objects such as repositories, ingest listeners, or storage partitions. An example of an object level metric is ingest-bytes/<repo>. Here <repo> is a placeholder for a concrete repository in a given Humio system.

Node Level Metrics

elastic-search-ingestion-request-errors

Number of ingest errors in the elastic-search endpoint since the node started

elastic-search-ingestion-requests

Time spent ingesting a bulk using the elasticsearch ingest protocol

event-collector-request-errors

Number of ingest errors in the http-event-collector endpoint since the node started

event-latency

Overall latency of ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files

global-publish-wait-for-value

Time spent waiting to see the value being read back from Kafka when pushing an update to the global state

hashfilter-included-blocks

Number of blocks included using hashfilters in queries and thus read from compressed blocks in segment files

hashfilter-skipped-blocks

Number of blocks skipped using hashfilters in queries and thus not read from compressed blocks in segment files

http-requests

Timing of all inbound http requests

humio-ingestion-request-errors

Number of ingest errors in the humio ingestion endpoint since the node started

ingest-listener-tcp-available

TCP ingest listener free slots for lines to be processed (High when idle, zero when over-loaded)

ingest-writer-bulksize

Histogram of size in number of bytes compressed data of jobs that carry events. Some jobs are no-payload and are not included here

ingest-writer-jobs

Number of jobs pushed to in-memory job queue for digest writers

kafka-chatter-bytes

Number of bytes written to kafka on the chatter topic

kafka-chatter-put

Time waiting for getting ack from Kafka when publishing to the chatter topic

kafka-ingestqueue-put

Time waiting for getting ack when adding ingest events to the ingest queue

kafka-request-bytes

Number of bytes written to kafka as compressed events for the ingest queue

kafka-request-events

Number of events written to kafka as compressed events for the ingest queue

livequery-count

Number of live- (real time-) queries active

load-segment-total

Time spent reading (waiting for) blocks from segment files

logplex-ingestion-request-errors

Number of ingest errors in the logplex endpoint since the node started

mapsegment

Time spent on ‘map’ phase while searching non-real time segment files

proxied-query-polls

Timing of internal requests due to polling of queries not hitting the server coordinating the query

queries

Total number of queries started since this node started

query

Measure how long it takes for queries to complete

read-compressed-bytes

Number of bytes of read from compressed blocks in segment files

read-prefilter-bytes

Number of bytes of read from pre-filter files

s3-archiving-bytes-per-second

Bytes archived in S3 per second

s3-archiving-errors-per-second

Errors per second archiving logs in S3

s3-archiving-writes-per-second

Successful S3 archival writes per second

schedulesegments

Time spent scheduling segment files for the ‘map’ phase while searching non-real time segment files

timestamp-parsing-failed

Total number of timestamp strings that did not parse as a time stamp since start of the node

Object Level Metrics

current-live-events/<repo>

Number of events being processed updating live quries

data-ingester-errors/<repo>

Number of events that got an @error tag added to their fields during parsing

event-latency-partition/<partitions>

Per-partition latency of the ‘humio-ingest’ topic of the ingest+digest pipeline including time spent in parsers, updating live queries and adding events to blocks for segment files

ingest-bytes/<repo>

Number of bytes uncompressed in flushed blocks for segments being constructed

ingest-eventsize/<repo>

Number of bytes uncompressed summed over individual events in blocks in progress

ingest-parsing/<repo/parser>

Time spent parsing incoming events

ingest-queue-consumer/<repo>

Time spent constructing segment file blocks in memory and writing them to disk, including updating live queries if any

ingest-reader-partition-bytes/<partitions>

Number of bytes read from kafka as compressed events from the ingest queue

ingest-reader-partition-events/<partitions>

Number of events added to segment file blocks being constructed

ingest-reader-polltime/<partitions>

Time blocked waiting for next message from Kafka from ingest queue

ingest-writer-partition-bytes/<partitions>

Number of bytes written to kafka as compressed events into the ingest queue

notifications/<repo>

Time spent shipping a notification from an alert

query-millis/<repo>

Number of milliseconds spent processing historical queries

repo-queries/<repo>

Number of queries started per repo

udpingestbytes/<listener>

Number of bytes read by udp ingest listener

written-events-after-queue/<repo>

Number of events added to segment file blocks being constructed

written-events/<repo>

Number of events written to the ingest queue after being parsed