Add Native Observability (Profiling and Metrics)

[eric-sap]

Background

In the eventing-kafka code (specifically in pkg/common/prometheus/metrics.go) we create a Prometheus server manually, and use the Observe function in channel/producer/producer.go and dispatcher/dispatcher.go to record metrics.

We want to take out this explicit Prometheus server and switch to using the knative-eventing built-in observability functionality (which will by default create a Prometheus server, but should be able to be changed to use anything that opencensus supports).

Steps

Code changes

• We need to add something very similar to the pkg/broker/ingress/stats_reporter.go to eventing-kafka (probably in common/metrics or similar)
  • Alter this new StatsReporter to use knative_kafka_produced_msg_count and knative_kafka_consumed_msg_count instead of the ingress broker metrics
  • Change the "metrics *prometheus.MetricsServer" field in the Producer struct to a StatsReporter instead
  • Add a call to NewStatsReporter() in the channel and dispatcher main(), pass the resulting structure into the producer.NewProducer() and dispatch.NewDispatcher() calls instead of the NewMetricsServer() result we currently use
  • Change Producer->processProducerEvents and Dispatcher->handleKafkaMessages() to create a ReportArgs structure and call the statsReporter.ReportEventCount() instead of the current metrics.Observe() calls
• Add an observability.go to common/k8s with an InitializeObservability function to be called from the channel and dispatcher's main() functions (see attached files for examples). The main purpose of this function is to start the profiling server and update the metrics exporter (which starts it, in the case of Prometheus).
• Need to add METRICS_DOMAIN (which should NOT have a slash in it, unlike the tests in pkg/reconciler/apiserversource/controller_test.go) that should probably just be "eventing-kafka"
• METRICS_DOMAIN needs to be set in the controller and passed as a variable to the channel and dispatcher, much like any of a number of other environment variables.
• We might need to pass some other pieces of information like the container name and pod name, if those are not already readily available, as the NewStatsReporter() function needs them.
• Remove the old code from pkg/common/prometheus/metrics.go (possibly the entire file or directory; needs to be determined if anything in there needs to be kept)

When this is all complete there should be (if possible) no references to Prometheus explicitly in the eventing-kafka codebase (barring vendor code).

Examples
There are a few examples of how to use the StatsReporter.ReportEventCount function in the eventing code, such as pkg/broker/ingress/ingress_handler.go

The github.com/census-ecosystem/opencensus-go-exporter-prometheus project has examples of how to use opencensus to create views and verify that metrics are being produced and exported properly.

InitializeObservability sample code:
observability.zip

References

Original Observability research issue: #29