Epic: More OpenTelemetry tracing

[hlinnaka]

Motivation

Get a better picture of latency and where time is spent.

I've added tracing to a bunch of places in the console and compute_tools already, but there's more we could do.

Tasks

• Add tracing to outgoing requests from proxy
• Add tracing to incoming HTTP requests in proxy. Mostly, the proxy-kick.
• Add tracing to pageserver. Would be nice to trace individual GetPage requests. You can generate a flamegraph from the opentelemetry traces, which would be nice since commit e9583db removed the previous flamegraphing support we had.
• Add tracing to safekeeper
• Trace Kubernetes operations. There's some preliminary support for that at https://kubernetes.io/docs/concepts/cluster-administration/system-traces/. I'd love to get a trace of where the time is spent in compute pod startup; I don't know if Kubernetes' tracing is useful for that.
• Add tracing to neonvm. To trace qemu and VM startup.
• Add tracing to Postgres. Could be done as an open source extension, to trace query execution. Would be useful outside neon.
• Add tracing to the Postgres neon extension, to trace the GetPage requests from that side.

We can implement these things in any order, as the need arises. The more tracing we have in different components, the more fine-grained information we get, but there's no particular hurry on any one of these.