Logs Monitoring
Glouton can read the logs from your applications and generate metrics from the number of lines matching a pattern.
Installationβ
Glouton uses Fluent Bit to collect the logs on your server.
You can install it as Linux package, a Docker container or a Helm chart on Kubernetes.
If you installed Glouton as Docker container, you also need to install Fluent Bit with Docker. Installing the Linux package is not supported in this case because Glouton won't be able to reload Fluent Bit when the configuration changes.
Linux Packageβ
To support logs monitoring, you need to install the package
bleemeo-agent-logs
. It will install Fluent Bit and configure it to talk with
your agent.
On Ubuntu/Debian:
sudo apt-get install bleemeo-agent-logs
On CentOS/Fedora:
sudo yum install bleemeo-agent-logs
Dockerβ
First, you need to create the Fluent Bit configuration.
sudo mkdir -p /etc/glouton
cat << EOF | sudo tee /etc/glouton/fluent-bit.conf
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
@INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
EOF
Then you can run the Fluent Bit container.
sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
docker run -d --restart=unless-stopped --name bleemeo-agent-logs \
-v "/etc/glouton/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro" \
-v "/:/hostroot:ro" -v "/var/lib/glouton:/var/lib/glouton" \
-l "prometheus.io/scrape=true" -l "prometheus.io/port=2020" \
-l "prometheus.io/path=/api/v1/metrics/prometheus" \
kubesphere/fluent-bit:v2.0.6 --watch-path /var/lib/glouton/fluent-bit/config
Kubernetesβ
On Kubernetes, Fluent Bit can be installed with the official Helm Chart.
Create the file values.yaml
with the following content:
image:
repository: kubesphere/fluent-bit
tag: v2.0.6
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: "/api/v1/metrics/prometheus"
config:
service: |
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
@INCLUDE /var/lib/glouton/fluent-bit/config/fluent-bit.conf
inputs: ""
filters: ""
outputs: ""
args:
- --watch-path
- /var/lib/glouton/fluent-bit/config
daemonSetVolumes:
- name: hostroot
hostPath:
path: /
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: lib
hostPath:
path: /var/lib/glouton/fluent-bit
daemonSetVolumeMounts:
- name: hostroot
mountPath: /hostroot
readOnly: true
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: lib
mountPath: /var/lib/glouton/fluent-bit
Then add the Helm repository and install the chart.
# Create the initial empty configuration.
sudo mkdir -p /var/lib/glouton/fluent-bit/config
sudo touch /var/lib/glouton/fluent-bit/config/fluent-bit.conf
# Install the Helm chart.
kubectl create namespace fluent-bit
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install -f values.yaml -n fluent-bit fluent-bit fluent/fluent-bit
Configurationβ
Glouton can create metrics by applying a regular expression to the log lines. The metrics created correspond to the number of times your regular expression matched a line per second.
The logs can be given by path, by a container name, or by container labels or annotations.
log:
inputs:
# Select the logs by path, container name or labels.
# path:
# container_name:
# container_selectors:
# List of metrics to generate.
filters:
# Generated metric name.
- metric: apache_errors_rate
# Ruby regular expression to apply on each log line.
# For testing purposes you can use the following web editor
# to test your expressions: https://rubular.com/.
regex: \[error\]
Select Logs by Pathβ
If your application is running directly on the host, you can select the logs directly by path.
log:
inputs:
# Apache errors per second.
- path: /var/log/apache/access.log
filters:
- metric: apache_errors_rate
regex: \[error\]
Select Logs by Container Nameβ
If your application is running in a container with a stable name, you can select its logs by the container name.
log:
inputs:
# Redis logs per second by container name.
- container_name: redis
filters:
- metric: redis_logs_rate
regex: .*
Select Logs by Container Labelsβ
You can select the log of a container by labels or annotations.
log:
inputs:
# UWSGI logs count by pod labels.
- container_selectors:
component: uwsgi
filters:
- metric: uwsgi_logs_rate
regex: .*