Logs Monitoring

Glouton can read the logs from your applications and generate metrics from the number of lines matching a pattern.

Installation

Glouton uses Fluent Bit to collect the logs on your server.

You can install it as Linux package, a Docker container or a Helm chart on Kubernetes.

caution

If you installed Glouton as Docker container, you also need to install Fluent Bit with Docker. Installing the Linux package is not supported in this case because Glouton won't be able to reload Fluent Bit when the configuration changes.

Linux Package

To support logs monitoring, you need to install the package bleemeo-agent-logs. It will install Fluent Bit and configure it to talk with your agent.

On Ubuntu/Debian:

sudo apt-get install bleemeo-agent-logs

On CentOS/Fedora:

sudo yum install bleemeo-agent-logs

Docker

First, you need to create the Fluent Bit configuration.

sudo mkdir -p /etc/glouton
cat << EOF | sudo tee /etc/glouton/fluent-bit.conf
[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_PORT    2020

@INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
EOF

Then you can run the Fluent Bit container.

sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
docker run -d --restart=unless-stopped --name bleemeo-agent-logs \
  -v "/etc/glouton/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro" \
  -v "/:/hostroot:ro" -v "/var/lib/glouton:/var/lib/glouton" \
  -l "prometheus.io/scrape=true" -l "prometheus.io/port=2020" \
  -l "prometheus.io/path=/api/v1/metrics/prometheus" \
  kubesphere/fluent-bit:v2.0.6 --watch-path /var/lib/glouton/fluent-bit/config

Kubernetes

On Kubernetes, Fluent Bit can be installed with the official Helm Chart.

Create the file values.yaml with the following content:

image:
  repository: kubesphere/fluent-bit
  tag: v2.0.6

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "2020"
  prometheus.io/path: "/api/v1/metrics/prometheus"

config:
  service: |
    [SERVICE]
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_PORT    2020

    @INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
  inputs: ""
  filters: ""
  outputs: ""

args:
  - --watch-path
  - /var/lib/glouton/fluent-bit/config

daemonSetVolumes:
  - name: hostroot
    hostPath:
      path: /
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: lib
    hostPath:
      path: /var/lib/glouton/fluent-bit

daemonSetVolumeMounts:
  - name: hostroot
    mountPath: /hostroot
    readOnly: true
  - name: varlog
    mountPath: /var/log
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: lib
    mountPath: /var/lib/glouton/fluent-bit

Then add the Helm repository and install the chart.

# Create the initial empty configuration.
sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
# Install the Helm chart.
kubectl create namespace fluent-bit
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install -f values.yaml -n fluent-bit fluent-bit fluent/fluent-bit

Configuration

Glouton can create metrics by applying a regular expression to the log lines. The metrics created correspond to the number of times your regular expression matched a line per second.

The logs can be given by path, by a container name, or by container labels or annotations.

log:
  inputs:
    # Select the logs by path, container name or labels.
    # path:
    # container_name:
    # container_selectors:

    # List of metrics to generate.
    filters:
      # Generated metric name.
      - metric: apache_errors_rate
        # Ruby regular expression to apply on each log line.
        # For testing purposes you can use the following web editor
        # to test your expressions: https://rubular.com/.
        regex: \[error\]

Select Logs by Path

If your application is running directly on the host, you can select the logs directly by path.

log:
  inputs:
    # Apache errors per second.
    - path: /var/log/apache/access.log
      filters:
        - metric: apache_errors_rate
          regex: \[error\]

Select Logs by Container Name

If your application is running in a container with a stable name, you can select its logs by the container name.

log:
  inputs:
    # Redis logs per second by container name.
    - container_name: redis
      filters:
        - metric: redis_logs_rate
          regex: .*

Select Logs by Container Labels

You can select the log of a container by labels or annotations.

log:
  inputs:
    # UWSGI logs count by pod labels.
    - container_selectors:
        component: uwsgi
      filters:
        - metric: uwsgi_logs_rate
          regex: .*

Logs Monitoring

Installation​

Linux Package​

Docker​

Kubernetes​

Configuration​

Select Logs by Path​

Select Logs by Container Name​

Select Logs by Container Labels​