Log Metrics with OpenTelemetry

Starter

Professional

Overview

Glouton can read application logs and generate metrics from the number of lines matching a regular expression (or a richer condition) per second. Log-to-metric counting is built directly into Glouton’s embedded OpenTelemetry Collector, alongside (and sharing configuration with) shipping logs.

A single receiver definition can:

• collect and ship logs to the Bleemeo Cloud Platform,
• select its logs by file path, by container name, by container labels/annotations, or by listening from the network (OTLP),
• feed one or more metrics, each with its own matching conditions, labels, and more.

Sharing configuration doesn’t mean the two features depend on each other: counting lines never requires shipping them. Log-to-metric runs independently of shipping_enable, so the metrics below keep being produced with log shipping switched off entirely.

Configuration

Everything lives under log.opentelemetry in glouton.conf, plus a top-level opentelemetry.listeners section for receivers that accept pushed logs from network.

log:
  opentelemetry:
    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        metrics:
          - metric: app_errors_count
            regex: '\[error\]'

Note

This receiver does two things: it produces the metric and ships the matched logs to the Bleemeo Cloud Platform. A receiver’s send_logs defaults to receivers_default_send_logs, which is true — so every example on this page ships logs as well, unless you say otherwise.

For metrics only, add send_logs: false to the receiver, set log.opentelemetry.receivers_default_send_logs: false to change the default for every receiver at once, or turn shipping off entirely with log.opentelemetry.shipping_enable: false. The metrics are unaffected in all three cases.

Each metric definition accepts:

Field	Type	Description
metric	string	Name of the generated metric.
regex	string	Shortcut for a single condition matching the raw log line body. See the official documentation for Go regular expressions.
conditions	list of strings	One or more OTTL conditions, OR’ed together, matched against the record body or a parsed attribute.
attributes	list of {key, default_value} objects	Dynamic group-by-value: one series per distinct value seen for the named attribute key. Comes with real cardinality risk.
labels	map of string to string	Fixed key: value pairs stamped on every sample – zero cardinality risk.
item	string	Identifies which source a metric’s series came from. Defaults to the receiver’s name; set explicitly to override.

Note

. and - in a label name (e.g. an attributes key like http.response.status_code) are converted to _ before the metric reaches the panel. Use the underscored form (http_response_status_code) in dashboard queries.

For example:

log.opentelemetry.receivers.example:
  operators:
    - type: regex_parser
      regex: '^\S+ \S+ \S+ \[.*\] ".*" (?P<status_code>\d+) .*$'
  metrics:
    - metric: error_count
      regex: "ERROR"
    - metric: http_requests_by_status
      conditions:
        - 'IsMatch(attributes["status_code"], "5..")'
        - 'IsMatch(attributes["status_code"], "4..")'
      attributes:
        - key: status_code
          default_value: "unknown"
      labels:
        service: checkout
        env: production
      item: checkout-api

conditions here are OR’ed together: a line matching either one counts. If a metric sets both regex and conditions, they aren’t mutually exclusive: regex is just a simpler way to declare a IsMatch(body, ...) condition, added to the same OR’ed list as everything under conditions – so a line counts if it matches regex or any of the conditions. attributes splits the resulting metric into one series per distinct value of status_code seen in matching lines (falling back to default_value when the attribute is missing) – unlike labels, which stamps the same fixed value on every sample regardless of the line’s content.

The operators block above extracts status_code from the raw line – without it the attribute would never exist, and http_requests_by_status would silently never count anything. See Matching Parsed Attributes below for more, including the http.response.status_code naming convention, built-in log formats that extract it for you, and why a second move step is sometimes needed.

Priority between item, labels, and attributes

These three all end up as labels on the same series, but they’re not independent, for two unrelated reasons:

• When each is known. item and labels come straight from the receiver’s config, so they’re already set the moment the series is declared – before any log line has even matched. attributes values are only known once a line actually matches: they’re extracted from that line’s own content.
• Who wins on a key collision. Nothing stops a labels entry, or a matched line’s attributes, from using a key that’s already spoken for – most notably item itself. Precedence is item > labels > attributes: item and labels are reserved-key-safe and always win; an attributes entry only fills in a label key nothing else has already claimed, and is dropped (never merged or promoted) on collision. One exception: labels: {item: ...} is treated as an equivalent, alternate spelling of the top-level item: field rather than a competing label – see the example below.

log.opentelemetry.receivers.example:
  metrics:
    - metric: requests_total
      labels:
        item: shipping-override   # honored: same as writing item: shipping-override
      attributes:
        - key: item               # ignored: attributes never override item
        - key: http.method        # kept: no collision

Here the series reports item="shipping-override" (from labels.item, since no top-level item: was set – had both been set, the top-level item: would win), the attributes entry named item is dropped, and http.method’s per-line value is added as an extra label.

Select Logs by Path

If your application writes logs to a file on the host:

log:
  opentelemetry:
    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        metrics:
          - metric: app_errors_count
            regex: '\[error\]'

Note

Glouton needs read access to each file. If it can’t open one directly (not world-readable and Glouton isn’t running as root), it automatically retries via sudo tail/sudo stat – which only works if the Glouton user already has passwordless sudo rights for those two commands, as set up by the standard Glouton package install. If that also fails, the file is silently skipped (only visible in debug-level logs). On Windows there’s no sudo fallback: an unreadable file is skipped outright.

See more about file permissions here.

Select Logs by Container Name

If your application runs in a container with a stable name:

log:
  opentelemetry:
    receivers:
      redis:
        container_name: redis
        metrics:
          - metric: redis_errors_count
            regex: ERROR

Select Logs by Container Labels

Select container logs by labels or annotations:

log:
  opentelemetry:
    receivers:
      web-workers:
        container_selectors:
          app: web-worker
        metrics:
          - metric: web_error_rate
            regex: '\[error\]'

Select Logs by Container Labels (No Receiver Needed)

A container doesn’t need a receiver entry at all: set the glouton.log_metrics label or annotation directly on it, naming a log.metrics_rules entry to count its logs against. The metric’s item is then that container’s own runtime name, so different containers sharing the same label value don’t merge into one series.

log:
  metrics_rules:
    known_web_errors:
      - metric: web_error_rate
        conditions:
          - 'IsMatch(body, "\\[error\\]")'

glouton.log_metrics=known_web_errors

Other glouton.* labels/annotations affect log shipping, not metrics, and are covered on the log shipping page: glouton.log_enable, glouton.send_logs, glouton.log_format, glouton.log_filter.

Whether this container’s logs are also shipped is decided by those labels — glouton.send_logs, else glouton.log_enable, else whether container auto-discovery is on. There is no receiver here, so receivers_default_send_logs does not apply: to count without shipping, set glouton.send_logs=false on the container.

A container already matched by an explicit receiver’s container_name/container_selectors ignores glouton.log_metrics – it’s counted exactly once, under that receiver’s own item, not twice.

Select Logs Pushed Over the Network

A receiver can also accept logs pushed to it (e.g. from another OpenTelemetry Collector) instead of reading files or watching containers. Declare a listener under the top-level opentelemetry.listeners, then reference it by name from one or more receivers with from_listeners.

Several receivers can share one listener by referencing a named entry under the top-level opentelemetry.listeners:

opentelemetry:
  listeners:
    otlp:
      protocols:
        grpc:
          endpoint: "localhost:4317"
        http:
          endpoint: "localhost:4318"

log:
  opentelemetry:
    receivers:
      billing_from_grpc:
        from_listeners: [otlp]
        metrics:
          - metric: billing_error_rate
            conditions:
              - 'resource["service.name"] == "billing"'

You can also declare a network listener without endpoints to use the default OpenTelemetry endpoints (localhost:4317 for grpc protocol, localhost:4318 http protocol):

opentelemetry:
  listeners:
    otlp:
      protocols:
        grpc:
        http:

Matching Parsed Attributes

By default, regex match against the raw log line (body when using conditions). To match against a structured field instead (e.g. an HTTP status code), the line must first be parsed into attributes with log_format.

If a built-in format already extracts the field you need (e.g. apache_access and nginx_access both produce http.response.status_code), just reference it by name:

log:
  opentelemetry:
    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        log_format: apache_access
        metrics:
          - metric: apache_server_error
            conditions:
              - 'IsMatch(attributes["http.response.status_code"], "5..")'

Otherwise, define your own format under log.opentelemetry.known_log_formats and reference it from the receiver’s own log_format:

log:
  opentelemetry:
    known_log_formats:
      my_custom_format:
        - type: regex_parser
          regex: '^\S+ \S+ \S+ \[.*\] ".*" (?P<status_code>\d+) .*$'
        - type: move # named capture groups can't contain dots, so rename into the dotted key used below
          from: attributes.status_code
          to: attributes['http.response.status_code']

    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        log_format: my_custom_format
        metrics:
          - metric: app_server_error
            conditions:
              - 'IsMatch(attributes["http.response.status_code"], "5..")'

For a one-off receiver, inline parsing stanzas can be used instead of a named format, with operators:

log:
  opentelemetry:
    receivers:
      app:
        include:
          - /var/log/myapp/*.log
        operators:
          - type: regex_parser
            regex: '^\S+ \S+ \S+ \[.*\] ".*" (?P<status_code>\d+) .*$'
          - type: move
            from: attributes.status_code
            to: attributes['http.response.status_code']
        metrics:
          - metric: app_server_error
            conditions:
              - 'IsMatch(attributes["http.response.status_code"], "5..")'

See more about log formats and operators here.

Metric Rules

Reusable metric definitions can be declared once under log.metrics_rules and pulled into any receiver with {include: <name>}:

log:
  metrics_rules:
    known_web_errors:
      - metric: web_error_rate
        conditions:
          - 'IsMatch(body, "\\[error\\]")'

  opentelemetry:
    receivers:
      web-workers:
        container_selectors:
          app: web-worker
        metrics:
          - include: known_web_errors

Migrating from Fluent Bit

Existing log.inputs configurations (the legacy Fluent Bit-era format) keeps working: Glouton automatically translates them into the equivalent log.opentelemetry.receivers / log.metrics_rules shape at startup with a deprecation warning. There’s nothing to migrate by hand, but new configurations should use log.opentelemetry.receivers / metrics directly.