informer library for shared watching with retries and updated list

[grosser]

fixes #456

[cben]

Minor questions aside, this is great ❤️

[cben]

Will version "0" result in .metadata.resourceVersion we can watch from? It means "any version is OK" which allows server returning cached response, but I vaguely remember hearing that might allow a version so old it'll return 410 when watched (?)

[cben]

Additionally, when List+Watch restarts, this might lead to replaying a state + events that are older than we already observed?

https://kubernetes.io/docs/reference/using-api/api-concepts/#the-resourceversion-parameter doesn't exactly answer this, but does say about supplying resourceVersion="0" to watch directly:

Get State and Start at Any: Warning: Watches initialize this way may return arbitrarily stale data! Please review this semantic before using it, and favor the other semantics where possible. Start a watch at any resource version, the most recent resource version available is preferred, but not required; any starting resource version is allowed. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches. Clients that cannot tolerate this should not start a watch with this semantic. To establish initial state, the watch begins with synthetic "Added" events for all resources instances that exist at the starting resource version. All following watch events are for all changes that occurred after the resource version the watch started at.

[grosser]

• if the version was too old it would results in a restart of the list/watch loop, so we should be good (never saw this happening)
• watch does not receive the 0, but what the list returned ... but it could end up in a replay since the list might have gotten old data ... so ideally we'd keep track of the last resourceVersion we saw and then do something with that (with a "latest" approach we'd lose some events though)

[cben]

• for 410 (Gone / Expired), this will restart the fill_cache -> watch_to_update_cache loop ✔️
• can any other errors be interesting for user?

[grosser]

no idea, I never had trouble with just ignoring :D

[cben]

If this happens (shouldn't), it'd raise in the worker thread. Would ruby even print it?

[grosser]

might be most consistent to ignore it so the worker does not crash ...
signaling back is kinda tricky, that's what I'm planing to use a logger for

[grosser]

still any showstoppers ?
added logger to make errors more visible

[cben]

[we're awaiting a baby any time now, so my response times will be more erratic than they already were.]

One thought I had is this actually consists of several parts:

• an implementation of automatic watch restarts.
• pub/sub via queue.
• a cache of last seen state for all objects.

In theory they could be split, one might want restarts without keeping a cache in RAM (or only a subset of cache).
But it is a pretty convenient package together, and an established pattern from the Go client. And we could always split it later if there's demand.

LGTM 👍 Can you check the test failure?

  1) Failure:
RetryTest#test_timeout [/home/runner/work/kubeclient/kubeclient/test/test_informer.rb:122]:
not all expectations were satisfied
unsatisfied expectations:
- expected exactly once, invoked twice: #<AnyInstance:Kubeclient::Common::WatchStream>.finish(any_parameters)

[grosser]

I think I got the tests fixed now ... truffle-ruby had some timing issues, so had to disable 1 test there

[cben]

LGTM. I'm still uncertain about using resourceVersion "0" for list, but we can revise that later if needed.