cadvisor metrics in master are inconsistent - not all calls return the correct metrics

[smarterclayton]

Running the metrics dump from master (in this case alpha.0 and latest) I get only a portion of metrics each time. The number of metrics should be consistent from run to run.

○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
     338
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
     338
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
       0
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
       0
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      50
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
     568
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
       0
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
       0
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
       0
○ oc get --raw /metrics/cadvisor --server https://10.1.2.2:10250 | grep container_name | wc -l
      82

[smarterclayton]

@sjenning reported by the CM ops team, I hadn't noticed this before but it's happening in in both alpha.0 and master latest for 3.7. Could be something related to a patch we carry.

[smarterclayton]

I see

E0824 23:09:06.952339    9754 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/38b26f8b877f3d280460b5da0110b62ba954808b44ad5fa92da5c936262945d8 with output stdout: , stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/38b26f8b877f3d280460b5da0110b62ba954808b44ad5fa92da5c936262945d8': No such file or directory
 - exit status 1, rootInodeErr: cmd [find /rootfs/var/lib/docker/overlay2/38b26f8b877f3d280460b5da0110b62ba954808b44ad5fa92da5c936262945d8 -xdev -printf .] failed. stderr: find: '/rootfs/var/lib/docker/overlay2/38b26f8b877f3d280460b5da0110b62ba954808b44ad5fa92da5c936262945d8': No such file or directory
; err: exit status 1, extraDiskErr: du command failed on /rootfs/var/lib/docker/containers/949106cc41597b695e2f9088adb22026efad3c2d9757c04a15b512c8de6ced04 with output stdout: , stderr: du: cannot access '/rootfs/var/lib/docker/containers/949106cc41597b695e2f9088adb22026efad3c2d9757c04a15b512c8de6ced04': No such file or directory
 - exit status 1

in the logs but may be unrelated.

[smarterclayton]

Also seeing

W0824 22:36:57.971384    9754 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available

[smarterclayton]

W0824 21:37:15.871049    9754 helpers.go:771] eviction manager: no observation found for eviction signal allocatableNodeFs.available
E0824 21:37:25.167629    9754 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db with output stdout: 1280412	/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db
, stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db/merged/proc/9754/fdinfo/317': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db/merged/proc/32673/task/32673/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db/merged/proc/32673/task/32673/fdinfo/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db/merged/proc/32673/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/640c4cfbd8bd91643972edeaa5bd97135e2edc29717cc7c3f42d9f70099cc1db/merged/proc/32673/fdinfo/4': No such file or directory
 - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>

[smarterclayton]

I don't see those log lines correlating with repeatedly calling cadvisor, so may be unrelated.

[derekwaynecarr]

we need google/cadvisor#1681

[smarterclayton]

Hrm, we have that already.

[derekwaynecarr]

looks like this: kubernetes/kubernetes#51473

[smarterclayton]

Cherrypigking