Linux System Logs & Metrics

Getting system logs and host metrics from a Linux systems enables you to monitor a large array of issue. Here as a small list of some of the things you could do:

  • Find servers that have too much load
  • Detect when your are running out of disk space
  • See when services reboot or crashes
  • See which user run which commands with sudo

Host Metrics

To get the standard host metrics, like cpu usage, load, memory, etc. use Metricbeat. Metricbeat can extract metrics from many different applications. For linux host metrics the system module is of interest.

Metricbeat can provide a lot of metrics (many per process for example). Experiment with finding the level you need.

Example Metricbeat Configuration

metricbeat.modules:
  - module: system
    enabled: true
    period: 10s
    metricsets:
      - cpu
      - load
      - filesystem
      - fsstat
      - memory
      - network

output.elasticsearch:
  hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
  username: $INGEST_TOKEN
Where:

  • $BASEURL - is the base URL of your Humio server (e.g. https://cloud.humio.com:443 or http://localhost:8080)
  • $INGEST_TOKEN - is the ingest token for your repository, (e.g. a string such as fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii).

See the page on Metricbeat for more information.

Example queries
check out these queries on host metrics

System Logs (syslog)

To ship the interesting system logs from /var/log/ to Humio use Filebeat.

Example Filebeat Configuration

filebeat.inputs:
- paths:
    - /var/log/syslog
    - /var/log/auth.log
  fields:
    "@type": syslog-utc

output.elasticsearch:
  hosts: ["https://$HOST:443/api/v1/dataspaces/$REPOSITORY_NAME/ingest/elasticsearch"]
  username: $INGEST_TOKEN
Where:

  • $BASEURL - is the base URL of your Humio server (e.g. https://cloud.humio.com:443 or http://localhost:8080)
  • $INGEST_TOKEN - is the ingest token for your repository, (e.g. a string such as fS6Kdlb0clqe0UwPcc4slvNFP3Qn1COzG9DEVLw7v0Ii).

Notice the type is syslog-utc, which points to the built in syslog parser, expecting the timestamp to be in UTC time. Often syslog timestamps are in local time. Go ahead and create a new parser with another timezone in Humio if necessary. You can copy the built in syslog-utc and just change the timezone. See Parsing for details.

Check out the Filebeat page for more information.

Custom Logs or Metrics

If you have custom logs or metrics you want to ship we suggest one of these strategies:

  1. Append the logs/metrics to a log file and use Filebeat to ship them similarly to the System logs above.

  2. Use cron to run a script that send data to Humio via it Ingest API.