Log Metrics with Fluent Bit

Starter

Professional

Caution

This is the legacy approach for generating metrics from logs using Fluent Bit. For modern log monitoring with full log collection, processing, and browsing, use the OpenTelemetry-based log pipeline instead.

Overview

Glouton can read application logs and generate metrics from the number of lines matching a regular expression pattern per second. It uses Fluent Bit to collect the logs.

Installation

You can install Fluent Bit as a Linux package, a Docker container, or a Helm chart on Kubernetes.

Caution

If you installed Glouton as a Docker container, you also need to install Fluent Bit with Docker. Installing the Linux package is not supported in this case because Glouton won’t be able to reload Fluent Bit when the configuration changes.

Linux Package

Install the bleemeo-agent-logs package:

On Ubuntu/Debian:

sudo apt-get install bleemeo-agent-logs

On CentOS/Fedora:

sudo yum install bleemeo-agent-logs

Docker

First, create the Fluent Bit configuration:

sudo mkdir -p /etc/glouton
cat << EOF | sudo tee /etc/glouton/fluent-bit.conf
[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020

@INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
EOF

Then run the Fluent Bit container:

sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
docker run -d --restart=unless-stopped --name bleemeo-agent-logs \
  -v "/etc/glouton/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro" \
  -v "/:/hostroot:ro" -v "/var/lib/glouton:/var/lib/glouton" \
  -l "prometheus.io/scrape=true" -l "prometheus.io/port=2020" \
  -l "prometheus.io/path=/api/v1/metrics/prometheus" \
  kubesphere/fluent-bit:v2.0.6 --watch-path /var/lib/glouton/fluent-bit/config

Kubernetes

On Kubernetes, Fluent Bit can be installed with the official Helm Chart.

Create the file values.yaml with the following content:

image:
  repository: kubesphere/fluent-bit
  tag: v2.0.6

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "2020"
  prometheus.io/path: "/api/v1/metrics/prometheus"

config:
  service: |
    [SERVICE]
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_PORT    2020

    @INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
  inputs: ""
  filters: ""
  outputs: ""

args:
  - --watch-path
  - /var/lib/glouton/fluent-bit/config

daemonSetVolumes:
  - name: hostroot
    hostPath:
      path: /
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: lib
    hostPath:
      path: /var/lib/glouton/fluent-bit

daemonSetVolumeMounts:
  - name: hostroot
    mountPath: /hostroot
    readOnly: true
  - name: varlog
    mountPath: /var/log
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: lib
    mountPath: /var/lib/glouton/fluent-bit

Then add the Helm repository and install the chart:

# Create the initial empty configuration.
sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
# Install the Helm chart.
kubectl create namespace fluent-bit
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install -f values.yaml -n fluent-bit fluent-bit fluent/fluent-bit

Configuration

Glouton creates metrics by applying a regular expression to log lines. The metrics correspond to the number of times the expression matches per second.

Logs can be selected by path, container name, or container labels.

log:
  inputs:
    # Select the logs by path, container name or labels.
    # path:
    # container_name:
    # container_selectors:

    # List of metrics to generate.
    filters:
      # Generated metric name.
      - metric: apache_errors_rate
        # Ruby regular expression to apply on each log line.
        # For testing purposes you can use https://rubular.com/.
        regex: \[error\]

Select Logs by Path

If your application runs directly on the host:

log:
  inputs:
    - path: /var/log/apache/access.log
      filters:
        - metric: apache_errors_rate
          regex: \[error\]

Select Logs by Container Name

If your application runs in a container with a stable name:

log:
  inputs:
    - container_name: redis
      filters:
        - metric: redis_logs_rate
          regex: .*

Select Logs by Container Labels

Select container logs by labels or annotations:

log:
  inputs:
    - container_selectors:
        component: uwsgi
      filters:
        - metric: uwsgi_logs_rate
          regex: .*