Observability Configuration Follow-Up

[eric-sap]

This issue is a placeholder to look further into the config-observability settings and determine if they are feasible to add to the eventing-kafka controller and dispatcher.

Background

The main function of observability seems to be the enabling of profiling content on individual pods in the knative-eventing namespace. More specifically, on any pod that uses the sharedmain.Main() implementation from knative.dev/pkg/injection/sharedmain/main.go

This is the (abbreviated) code that appears to enable the observability server in sharedmain.Main():

profilingHandler := profiling.NewHandler(logger, false)
profilingServer := profiling.NewServer(profilingHandler)
eg, egCtx := errgroup.WithContext(ctx)
eg.Go(profilingServer.ListenAndServe)
go func() {
    <-egCtx.Done()
    profilingServer.Shutdown(context.Background())
    eg.Wait();
}()
cmw := sharedmain.SetupConfigMapWatchOrDie(ctx, logger.Sugar())
sharedmain.WatchObservabilityConfigOrDie(ctx, cmw, profilingHandler, logger.Sugar(), environment.ServiceName)

However, putting only this block of code into the controller or dispatcher's non-shared main() function does not accomplish the intented effect. You can browse to a server on port 8008 after the controller/dispatcher executes the go function above, but it returns 404s for all paths.

Research Question: What else is necessary to start a profiling server that actually serves the /debug/pprof path?

Also, we start our own Prometheus server via prometheus.NewMetricsServer() in the channel and dispatcher non-shared main() functions. It is possible that the observability code requires the use of Prometheus in a different manner.

Research Question: Do we need to use Prometheus differently in order to leverage the knative observability machanism?

Instructions

Starting a profiling server on the controller pod:

• Modify the config-observability configmap in the knative-eventing namespace and add these lines to the "data" section:

  metrics.backend-destination: prometheus
  profiling.enable: "true"
  

• Wait for the watcher to update the configuration, or restart the eventing-kafka-channel-controller pod manually

• Run a port-forwarding command to the controller pod, for example

  kubectl -n knative-eventing port-forward eventing-kafka-channel-controller-xxxxxxxxxx-xxxxx 8008
  

• Browse to the local forwarded address

  http://localhost:8008/debug/pprof/
  

References

# Original Tracing and Observability issue
https://github.com/knative-sandbox/eventing-kafka/issues/22

# KNative information related to observability (mainly for knative-serving, but generic profiling seems common to eventing and service)
https://knative.dev/docs/serving/installing-logging-metrics-traces/

# Information about various profiling URLs
https://github.com/knative/serving/blob/master/test/performance/profiling.md

[travis-minke-sap]

/assign @eric-sap