feat(kds): introduce zone health checks

[michaelbeaumont]

Explanation

Zone:

• Send health check periodically

Global:

• When we get a health check, mark it in the corresponding ZoneInsight
• Introduce new component ZoneWatch that listens for ZoneOpenedStream and marks the (tenantID, zone) for watching
• Periodically check all watched (tenantID, zone) and if the time of last health check in ZoneInsight is too late, send ZoneWentOffline
• All opened streams also listen for ZoneWentOffline events and the handler returns if said event is received, ending the stream

We store the info in ZoneInsight because:
All instances need to potentially kill streams but not every instance will receive a health check from connected zones

Tests

The need for time.Sleep in the tests comes about because it happens asynchronously that:

1. ZoneWatch subscribes to ZoneOpenedStream events in Start
   in reality, this is guaranteed to happen before ZoneOpenedStream events are sent by the fact that we only send them in response to new gRPC streams being opened
2. ZoneOpenedStream is witnessed by ZoneWatch
   the test adds a time.Sleep because we only want to update the health check time once and in particular after the zone starts being watched and then check that it's disconnected
   in reality, a zone will continually send its health check ping so it eventually will be updated after the initial seen last time.

Todo

• configurability

Checklist prior to review

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --
• Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

[jakubdyszkiewicz]

I think we could introduce tests at least for ZoneWatch. Any ideas for E2E test?

[michaelbeaumont]

I think we could introduce tests at least for ZoneWatch. Any ideas for E2E test?

I think e2e will be somewhat difficult because of the nature of isolating failure to the app level health check. The always possible, but ugly option is to introduce an option like "don't send health check pings" to the CP but IMO we should avoid this artificial, public test interface

@slonka mentioned potentially sending SIGSTOP to make the process stop sending pings but keep TCP connections open. The problem that I see is that HTTP2 PING ACKs also stop, which will trigger the gRPC keep alive to fail.

[jakubdyszkiewicz]

but IMO we should avoid this artificial, public test interface

I agree.

We could put Kong in the middle (we have test infra to deploy KIC) which apparently does not proxy PING/GOAWAY and then do SIGKILL on Zone CP. I wonder if this would work.

[slonka]

We could put Kong in the middle (we have test infra to deploy KIC) which apparently does not proxy PING/GOAWAY and then do SIGKILL on Zone CP. I wonder if this would work.

It should work 👍

[michaelbeaumont]

OK, if we have some infrastructure for starting up kong gateway it becomes significantly less work