File Log Collection

Starter

Professional

Basic Configuration

To collect logs from files, define receivers in the log.opentelemetry.receivers section. Each receiver watches one or more file paths using glob patterns.

log.opentelemetry.receivers:
  my-app:
    include:
      - '/var/log/my-app/*.log'

This makes the application’s logs appear in the log section of the panel.

Note

Write the path as it is on the host, even including when Glouton itself runs in a container. Glouton reaches the file through the host’s view of the filesystem (the mount behind df.host_mount_point), so do not prefix the path with /hostroot.

File Permissions

You may need to allow Glouton to read the log file. This can be done with sudo, by adding in /etc/sudoers.d/glouton-local:

glouton     ALL=(ALL) NOPASSWD: /usr/bin/tail --follow=name --bytes=* /var/log/my-app/access.log
glouton     ALL=(ALL) NOPASSWD: /usr/bin/stat --printf=%s /var/log/my-app/access.log

Operators

Operators enrich and transform each log line. They are based on Stanza’s operators, so any operator from an OpenTelemetry Collector-Contrib configuration can be reused directly.

log.opentelemetry.receivers:
  my-app:
    include:
      - '/var/log/my-app/access.log'
    operators:
      - type: add
        field: resource['service.name']
        value: 'My app'

The list of available operator types can be found in the Stanza operators documentation.

Using Known Log Formats as Operators

A known log format can be applied directly as the receiver’s format, or included within operators:

log.opentelemetry.receivers:
  web-app:
    include:
      - '/var/log/nginx/access.log'
    log_format: nginx_access # apply a known log format
  my-app:
    include:
      - '/var/log/my-app/access.log'
    operators:
      - type: add
        field: resource['service.name']
        value: 'My app'
      - include: my_app_parser # include a known log format within operators
      - type: remove
        field: attributes['some-attr']

The format must be either a built-in default or a custom format you define.

Filters

Filters control which log records are exported. They can be applied per-receiver or globally.

Drop Filter (OTTL)

Drop matching log records using OpenTelemetry Transformation Language expressions:

log.opentelemetry.receivers:
  my-app:
    include:
      - '/var/log/my-app/access.log'
    filters:
      log_record:
        - 'IsMatch(body, ".*password.*")'

Include / Exclude Filters

For more granular control, use include and exclude filters with attribute matching:

log.opentelemetry.receivers:
  my-app:
    include:
      - '/var/log/my-app/access.log'
    log_format: apache_access # record_attributes below need this to exist first
    filters:
      include:
        match_type: regexp
        resource_attributes:
          - key: 'some-attr'
            value: 'some value'
        record_attributes:
          - key: 'http.response.status_code'
            value: '5..'
          - key: 'http.request.method'
            value: 'POST|PATCH'
        severity_texts:
          - 'info'
        severity_number:
          min: 'INFO'
          match_undefined: true
        bodies:
          - 'log with status_code = 5\d\d'
      exclude: # same structure as include

Tip

When writing regexps, prefer single quotes to avoid escaping backslashes.

record_attributes matches against a log record’s own attributes, same as log-to-metrics’ attributes: – the field must already exist, either because a built-in log_format extracts it (as here) or because it was pushed pre-parsed over OTLP. Without that, http.response.status_code/http.request.method would never exist and this filter would never match anything.

Filter Behavior

• The match_type can be strict (exact match) or regexp
• Within include or exclude, all conditions (resource_attributes, record_attributes, severity_texts, bodies) use OR logic — a log matching any of them qualifies
• Entries inside each section are also OR conditions
• When both include and exclude are specified, include filtering occurs first
• Filters can also be applied globally — see Global Filters

Service Log Files

When Glouton discovers services, it automatically sets up log processing for them. Learn more about supported services on the services page.

You can specify formats and filters for a specific service type by referencing known log formats and known log filters:

service:
  - type: 'my-app'
    ...
    log_format: 'my_app_parser'
    log_filter: 'my_app_filter'

To handle specific log files for a service:

- type: "my-app"
  ...
  log_files:
    - file_path: '/var/log/my-app/access.log'
      log_format: 'my_app_access_parser'
      log_filter: 'my_app_access_filter'
    - file_path: '/var/log/my-app/error.*.log'
      log_format: 'my_app_error_parser'
      log_filter: 'my_app_error_filter'