Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[receiver/k8sobjects] Fix issue where resourceVersion was not being remembered, #24806

Conversation

TylerHelmuth
Copy link
Member

Description:
Fixes an issue where the latest resourceVersion that RetryWatcher is remembering is not honored.

It turns out that RetryWatcher also doesn't like being passed "" or "0" during startup, and if you pass it a default value of "1" it will most likely return with a 410 complaining that the resource is too old and is long gone.

To fix this, we not perform an initial List to get the latest values and a proper resourceVersion. This is recommended by https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes.

Link to tracking Issue:
Closes #24805

Testing:
I tested this a lot manually. I updated the unit tests, but it was a struggle - the fake client framework doesn't honor the resourceVersion concept like k8s actually does.

Documentation:
Updated the readme to call out that Watch mode needs list access if no resource_version is provided.

err := kr.consumer.ConsumeLogs(obsCtx, logs)
kr.obsrecv.EndLogsOp(obsCtx, metadata.Type, 1, err)
logs, err := watchObjectsToLogData(&data, time.Now(), config)
if err != nil {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're passing around resourceVersion values properly there is now a chance that the client returns a 410 and an status object complaining that the resource doesn't exist. This is a rare situation that would normally only happen if the user self-supplied a resourceVersion that was too old.

At the moment I am handling that as an error and allowing the processor to keep running, but the group that had the error will stop. The proper way to handle that situation is to do another List to get a proper resourceVersion again. I think we should do that, but as a followup PR.

@TylerHelmuth TylerHelmuth force-pushed the k8sobjects-fix-resourceversion-bug branch from 96f9882 to 7c48a78 Compare August 3, 2023 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[receiver/k8sobjects] resourceVersion is not remembered
3 participants