Collect system data for client monitoring #5182

nflaig · 2023-02-20T13:05:19Z

Motivation

Follow up PR for #5037 to implement collection of system data to support full feature set of beaconcha.in monitoring app.

Description

Replaces currently hard-coded values in system stats with proper provider functions. Most of the system data is collected using systeminformation which is a widely used and well maintained package with 0 dependencies. There are a few issues running in a containerized environment as some data might be inaccurate or missing but the important data is collected correctly even there. These issues might also be addressed in the future, see sebhildebrandt/systeminformation#777, although it is really not that important.

Performance

Dashboards are already in place to keep track of the performance. The functions to collect system data are not executed in the main thread and a lot of the time use child processes which execute native system calls depending on OS. The performance measurements from the last few days looks really promising and I did not see any issues as of now both validator efficiency and system load seem unaffected by enabling client monitoring.

Added some screenshots of prometheus metrics, just let me now if I should provide further metrics.

Prometheus metrics

Takes between 0.5 - 1 second to collect system data, depending on the environment (on windows ~3 seconds).

It should not cause any spikes in Lodestar CPU usage as it is executed in a different process

Overall container CPU usage looks got to me as well, it does not seem like collecting system data consumes too much CPU

Event Loop Lag seems unaffected as well

Process memory usage looks stable

Container memory usage as well (no rss memory leak)

Beaconcha.in mobile app

We support all dashboards in the beaconcha.in mobile app and also all notifications which is the most valuable feature of the app in my opinion. A node operator gets a lot of value from the app which you normally would only get if you run prometheus + grafana and invest additional time to configure proper alerting.

Mobile app screenshots