Input docker behaviour with Openshift 3.10

[rbonvalot]

Filebeat Version : 6.4.3-1
Openshift version : 3.10

Since Openshift version 3.10, docker has been replaced by cri-o. Log format changed from json to cri format and we need to use input docker to parse file as describe in this issue #7418

Nethertheless, the path of docker logs in Openshift 3.10 is /var/log/containers/<pod_name><container_name>-<container_id>.log. Docker input consider this is this /var/lib/docker/containers/<container_id>/*.log.

I used this workaround for all docker

filebeat:
   inputs:
   - type: docker
     containers:
       path: "/var/log/containers"
       ids:
         - ''
     symlinks: true
     multiline.pattern: '^\['
     multiline.match: after
     multiline.negate: true
     fields:
       type: openshift-pods
     fields_under_root: true
     exclude_lines: ["DEBUG","TRACE"]
     exclude_files: ["rabbitmq","nginx","redis"]

It will considered this path -> /var/log/containers//*.log and it's ok if I wanted to process all log on the same way. In the other side, I need to process rabbitmq, nginx and redis pods on another ways but I'm blocked;

For example

filebeat:
   inputs:
   - type: docker
     containers:
       path: "/var/log/containers"
       ids:
         - ''
     symlinks: true
     multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
     multiline.match: after
     multiline.negate: true
     fields:
       type: rabbit-pods
     fields_under_root: true
     ## Negative pattern not supported
     exclude_lines: [".*(?!rabbitmq).*"]
     ##
     exclude_lines: ["DEBUG","TRACE"]

This example failed. I can't use exclude_line with negative pattern. Include line is not supported too. I think it should be good to be able to modify the "/*.log" automatically added by filebeat with input docker.

PS : directory /var/log/pods should be the good directory for input but it contain only ID of dockers and I'll lost all <pod_name><container_name>-<container_id> I mentionned before.

[rbonvalot]

Hello,

Exact same issue with autodiscover. I wanted to exclude rabbitmq with container name image condition Here is the config I tried with daemon set (without condition)

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: openshift-logging
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}/*"
                  fields:
                    type: openshift-pods
                    test: true
    processors:
      - add_cloud_metadata:
    output.logstash:
      hosts: ['${LOGSTASH_HOST:logstash}:${LOGSTASH_PORT:5044}']
    logging:
      level: debug
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: openshift-logging
  labels:
    app: filebeat
spec:
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      nodeSelector:
        region: air-dev
        role: appli
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.5.3
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: LOGSTASH_HOST
          value: "myserver"
        - name: LOGSTASH_PORT
          value: "5044"
        securityContext:
          runAsUser: 0
          privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlogpods
          mountPath: /var/lib/docker/container
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: data
        hostPath:
          path: /usr/share/filebeat/data

As you see, I mapped /var/log/pods to /var/lib/docker/container. The ID contained by /var/log/pods aren't the same as Kubernetes ID.

Here is some logs

2018-12-14T16:52:14.239Z        DEBUG   [autodiscover]  autodiscover/autodiscover.go:204        Got a stop event: map[host:10.129.4.188 kubernetes:{"annotations":{"openshift":{"io/scc":"privileged"}},"container":{"id":"71248c226578b994b2838b865567abdcd1e1dfd13477683501fa675fd76d97ff","image":"docker.elastic.co/beats/filebeat:6.5.3","name":"filebeat","runtime":"cri-o"},"labels":{"app":"filebeat","controller-revision-hash":"2634999952","pod-template-generation":"1"},"namespace":"openshift-logging","node":{"name":"air-dev-ocpnod-2"},"pod":{"name":"filebeat-zjql4"}} meta:{"kubernetes":{"container":{"name":"filebeat"},"labels":{"app":"filebeat","controller-revision-hash":"2634999952","pod-template-generation":"1"},"namespace":"openshift-logging","node":{"name":"air-dev-ocpnod-2"},"pod":{"name":"filebeat-zjql4"}}} config:[] stop:true], generated configs: []

The id is 71248c226578b994b2838b865567abdcd1e1dfd13477683501fa675fd76d97ff.

When I search the id on /var/log/containers/ folder on the server

[root@air-dev-ocpnod-2 ~]# ls -l /var/log/containers/ |grep 71248c226578b994b2838b865567abdcd1e1dfd13477683501fa675fd76d97ff
lrwxrwxrwx. 1 root root 65 Dec 14 17:52 filebeat-zjql4_openshift-logging_filebeat-71248c226578b994b2838b865567abdcd1e1dfd13477683501fa675fd76d97ff.log -> /var/log/pods/994b0909-ffc0-11e8-8321-005056ac2018/filebeat/0.log

You can see that the of /var/log/pods/ isn't the same that we get from Kubernetes API and I can't retrieve it.

[elasticmachine]

Pinging @elastic/infrastructure

[jp-paillet]

I made the following fixes to handle the docker case using the code branch 6.5. I'll see how to submit it in a better way later as I'm a complete beginner here.

jpa@debian-lab:~/go/src/github.com/elastic/beats/filebeat$ git diff input/docker/config.go
diff --git a/filebeat/input/docker/config.go b/filebeat/input/docker/config.go
index e33bb869a..9e46ff4d4 100644
--- a/filebeat/input/docker/config.go
+++ b/filebeat/input/docker/config.go
@@ -22,6 +22,7 @@ var defaultConfig = config{
        Containers: containers{
                IDs:    []string{},
                Path:   "/var/lib/docker/containers",
+               Suffix: "/*.log",
                Stream: "all",
        },
 }
@@ -39,6 +40,7 @@ type config struct {
 type containers struct {
        IDs  []string `config:"ids"`
        Path string   `config:"path"`
+       Suffix string   `config:"suffix"`

        // Stream can be all, stdout or stderr
        Stream string `config:"stream"`
jpa@debian-lab:~/go/src/github.com/elastic/beats/filebeat$ git diff input/docker/input.go
diff --git a/filebeat/input/docker/input.go b/filebeat/input/docker/input.go
index fef86c065..9c46da93c 100644
--- a/filebeat/input/docker/input.go
+++ b/filebeat/input/docker/input.go
@@ -20,6 +20,7 @@ package docker
 import (
        "fmt"
        "path"
+       "strings"

        "github.com/elastic/beats/filebeat/channel"
        "github.com/elastic/beats/filebeat/input"
@@ -52,8 +53,14 @@ func NewInput(
                return nil, errors.New("Docker input requires at least one entry under 'containers.ids'")
        }

-       for idx, containerID := range config.Containers.IDs {
-               cfg.SetString("paths", idx, path.Join(config.Containers.Path, containerID, "*.log"))
+       if len(config.Containers.Suffix) == 0 {
+          for idx, containerID := range config.Containers.IDs {
+               cfg.SetString("paths", idx, path.Join(config.Containers.Path, containerID))
+          }
+       } else {
+          for idx, containerID := range config.Containers.IDs {
+               cfg.SetString("paths", idx, path.Join(config.Containers.Path, strings.Join([]string{containerID, config.Containers.Suffix}, "")))
+          }
        }

Please find below some unit test evidencies :

1. Custom path and suffix
   Excerpt from filebeat.yml

- type: docker
  enabled: true
  containers:
    path: '/var/log/log_test'
    suffix: '*.log'
    ids:
      - '*redis'

Outputs

jpa@debian-lab:~/go/src/github.com/elastic/beats/filebeat$ ./filebeat -e -v -d "*" 2>&1 | grep "Configured paths"
2018-12-18T09:56:29.216+0100    INFO    log/input.go:138        Configured paths: [/var/log/*.log]
2018-12-18T09:56:29.220+0100    INFO    log/input.go:138        Configured paths: [/var/log/log_test/*redis*.log]

2. Default values
   Excerpt from filebeat.yml

- type: docker
  enabled: true
  containers:
    #path: '/var/log/log_test'
    #suffix: '*.log'
    ids:
      - '*redis'

Outputs

jpa@debian-lab:~/go/src/github.com/elastic/beats/filebeat$ ./filebeat -e -v -d "*" 2>&1 | grep "Configured paths"
2018-12-18T09:57:34.176+0100    INFO    log/input.go:138        Configured paths: [/var/log/*.log]
2018-12-18T09:57:34.180+0100    INFO    log/input.go:138        Configured paths: [/var/lib/docker/containers/*redis/*.log]

[ruflin]

@jp-paillet Perhaps you can open a PR with the above changes? That would make discussing the code changes easier.

[jp-paillet]

Hi Nicolas, yes, I'm learning how to use a PR. Sorry for the noise.

[noel-reddy]

Was this issue fixed

[jp-paillet]

Was this issue fixed

I'm sorry but I had no time to finalize the PR #9620.
We are using the initial changes on our servers, and we are happy with them, but indeed they are still not yet properly recorded at your side...
I'll spend time on the PR next week.