Advanced log routing with Fluent Bit 1.4

This use case is a community submission from Evgeni Sladkovski Trifork Amsterdam

In Fluent Bit 1.4 there is a filter called rewrite_tag that allows you to change tags based on various conditions and enables message routing to different outputs. It can also act as a filter for which logs actually need to be forwarded to Humio.

Let’s say you have some container logs that you want in the Humio logs repository, and some logs that I consider metrics and want in the Humio metrics repository. With this new filter I can do

[FILTER]
   Name          rewrite_tag
   Match         kube.*
   Rule          $kubernetes['labels']['humio'] ^(logs)$ logs.container false
   Emitter_Name  logs_rewrite
[OUTPUT]
   Name es
   Match logs.container
   Index logs
   HTTP_User {{ .Values.logs.token }}
[FILTER]
  Name          rewrite_tag
  Match         kube.*
  Rule          $kubernetes['labels']['humio'] ^(metrics)$ metrics.container false
  Emitter_Name  metrics_rewrite
[OUTPUT]
  Name es
  Match metrics.*
  Index metrics
  HTTP_User {{ .Values.metrics.token }}

So if a pod has a label humio: logs it goes to the logs repo and if the label is humio: metrics then it goes to the metrics repo. If a pod doesn’t have any of those labels its output is ignored.

For more information