[FEATURE] Support Health and Readiness Probes in flagd

[beeme1mr]

Overview

In production environments, feature flags may be considered a mission critical service that must be available. It's important for flagd to expose a means for Kubernetes to know when the container is ready to perform flag evaluation. Also, needs to be aware if flagd is in an unhealth state so that it can take corrective measures. These settings should be configurable so that a Kubernetes cluster administrator can decide various error conditions should be handled.

Requirements

• Conduct research on the pros and cons of adding health and readiness probes.
• Add health and readiness probes to flagd to monitor the state of the services.
• Update the OpenFeature Operator to automatically configure the health and readiness checks.
• Document the new behavior and configuration options in the OFO documentation.

Resources

• https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

[thisthat]

I had a look into this topic. We can re-use the HTTP server created for serving Prom metrics for the probes. This simplifies the implementation. Mind that K8s has a basic behavior regarding health probes: when it fails, it restarts the POD. Hence, we should not add any fancy configuration for this.
For the ready probe, I propose splitting the current ISync interface into two functions: Init(Context) and Sync(Context, chan). This would also make a cleaner implementation for providers such as gRPC.

[Kavindu-Dodan]

PR [1] completes all requirements, except for the configurability

Provide configuration options in the OpenFeature Operator to customize the probes.

Current reconfigurability is on enable/disable behavior (default is enabled). Further, probes will have a initialDelaySeconds set to 5 seconds. Other configurations will be set to default values [2].

In my view, current implementation is sufficient for the requirements we have right now.

[1] - open-feature/open-feature-operator#390
[2] - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes