process walker perfs: optimize readLimits and readStats

[alban]

I am testing Scope with a load of 100 processes running my openfiles program, so that should generate 50.000 opened files.

I get the following pprof data:

For the proc walker, that's:

• 1.12s in syscall.Syscall. Hopefully, proc walker: optimize open file counter #2456 will help with that
• 0.79s in readLimits (string parsing)
• 0.61s in readStats (string parsing)

There is room for improving the string parsing. For readStats, I wrote an alternative implementation of the parser: https://gist.github.com/alban/7bdf6ced1352d8efbc6c22a64eabe7e3 and I get the following improvements:

$ sudo ./read-stats-limits 
time (old): 8.345651064s
time (new): 5.330234374s

I initially tried with fmt.Sscanf but unfortunately, the performance was bad. My new implementation follows the pattern explained on http://stackoverflow.com/questions/31333353/faster-input-scanning

[alban]

The /proc/<pid>/limits files are cached (by probe/process/cache.go, with a timeout of 30s) so they are not read from procfs each time. But only the unparsed version is cached, and it is parsed each time in readLimits(). Ditto for /proc/<pid>/cmdline and /proc/<pid>/comm.

We should try to cache the parsed version instead. The cache timeout could be increased too: processes should rarely change their "ulimit" and their "cmdline"/"comm" (if done at all, this is usually done only one time at startup).

The constant statsTimeout is declared but unused: the stats are not cached in master.

[alban]

I compared:

• Branch master
• My branch https://github.com/kinvolk/scope/commits/alban/perf-proc-walker containing:
  • optimizing ReadDirNames
  • optimizing readStats parsing
  • optimizing readLimits parsing
  • caching parsed values instead of full files

for 4 minutes with the following commands:

$ sudo ./scope launch -probe.ebpf.connections --probe.http.listen :4041
$ go tool pprof http://localhost:4041/debug/pprof/profile?seconds=240
web

The scenario is:

• 100 processes with 500 file descriptors each (50.000 opened files)
• 7 idle Docker containers (nginx & busybox)
• approx. 510 processes on my Fedora laptop

---

Branch master

---

Branch perf-proc-walker

---

The function github.com/weaveworks/scope/probe/process.(*walker).Walk takes respectively:

• 20s
• 10.24s (almost 2x faster)

Overall, the graphs' titles say respectively:

• 48.82s
• 28.60s (41% faster overall)