Retry is not happening after exception #308
Comments
|
Hello. Thanks for reporting. Yes, indeed, Kopf relies on patching the object to cause a cascaded reaction with the next steps (maybe not the wisest way, but it was done so originally, and I had no reasons to redesign that yet). As far as I know, Kopf does not handle this case at the moment, and only assumes that status is part of the object — which can be controlled with custom resources, but not with built-in resources. I think, it would be easy to change the function to patch either an object or its |
|
A note for self on how it is defined in a GKE cluster: So, The same happens with CRDs when status is declared as a sub-resource: A little experiment shows that it indeed reacts only when |
|
As I mention, it seems that added custom fields in /status are just ignored: |
|
Ouch. That is highly unfortunate. Sorry, I missed that statement. I couldn't quickly figure out how to create VolumeAttachments in GKE even with a pod+pvc pair — so I could not test it on this resource specifically (yesterday evening). Well, at least, it now works with other resources with status-as-a-subresource (#119). What I can suggest, is a per-resource-kind or per-operator config where the internal status is stored: with Actually, there is a similar task #23 (and #283) — but there, the main use-case is multiple operators handling the same resources, so that they do not collide with each other. It can be extended to arbitrary root field name too. Storing it in an arbitrary top-level field would be the easiest way. Otherwise, annotations can be used indeed; but labels & annotations are allowed to be strings only, so serialisation/deserialisation would be needed. |
|
I can reproduce in other built in resources like pod or pvc. I jsut tried patching a top-level field, again no luck. Maybe new versions of kubernetes are getting more strict.. |
|
Indeed. I now also have this reproduced with pods in Minikube with 1.16. Neither status nor root accepts arbitrary fields. The only place it accepts arbitrary content is annotations. So far, so good — let's abuse annotations! I have made a little dirty-hacking draft to store the operator's state locally in two formats: per-handler and per-handler-field. After the handling cycle, these annotations are wiped (as previously with the status fields). Per handler: apiVersion: zalando.org/v1
kind: KopfExample
metadata:
annotations:
my-op.zalando.org/create_fn_1: '{"started": "2020-02-14T16:58:25.396364", "stopped":
"2020-02-14T16:58:25.401844", "delayed": null, "retries": 1, "success": true,
"failure": false, "message": null}'
my-op.zalando.org/create_fn_2: '{"started": "2020-02-14T16:58:25.396421", "stopped":
null, "delayed": null, "retries": 0, "success": false, "failure": false, "message":
null}'
creationTimestamp: "2020-02-14T16:58:25Z"
finalizers:
- kopf.zalando.org/KopfFinalizerMarker
generation: 1Per handler-field: apiVersion: zalando.org/v1
kind: KopfExample
metadata:
annotations:
my-op.zalando.org/create_fn_1.delayed: "null"
my-op.zalando.org/create_fn_1.failure: "false"
my-op.zalando.org/create_fn_1.message: "null"
my-op.zalando.org/create_fn_1.retries: "1"
my-op.zalando.org/create_fn_1.started: '"2020-02-14T17:10:42.713227"'
my-op.zalando.org/create_fn_1.stopped: '"2020-02-14T17:10:42.718587"'
my-op.zalando.org/create_fn_1.success: "true"
my-op.zalando.org/create_fn_2.delayed: "null"
my-op.zalando.org/create_fn_2.failure: "false"
my-op.zalando.org/create_fn_2.message: "null"
my-op.zalando.org/create_fn_2.retries: "0"
my-op.zalando.org/create_fn_2.started: '"2020-02-14T17:10:42.713403"'
my-op.zalando.org/create_fn_2.stopped: "null"
my-op.zalando.org/create_fn_2.success: "false"
creationTimestamp: "2020-02-14T17:10:42Z"
finalizers:
- kopf.zalando.org/KopfFinalizerMarker
generation: 1Per handler-field, if serialised a little bit smarter: apiVersion: zalando.org/v1
kind: KopfExample
metadata:
annotations:
my-op.zalando.org/create_fn_1.retries: "1"
my-op.zalando.org/create_fn_1.started: "2020-02-14T17:10:42.713227"
my-op.zalando.org/create_fn_1.stopped: "2020-02-14T17:10:42.718587"
my-op.zalando.org/create_fn_1.success: "true"
my-op.zalando.org/create_fn_2.retries: "0"
my-op.zalando.org/create_fn_2.started: "2020-02-14T17:10:42.713403"
creationTimestamp: "2020-02-14T17:10:42Z"
finalizers:
- kopf.zalando.org/KopfFinalizerMarker
generation: 1Works like a charm! — Both for CRs and pods. In both cases, and as per the docs, the generation is not increased when metadata is changed (this happens only for spec). But it generates the watch-events — which makes sense since most of the printed columns of Beside this, such approach allows to customise the operator name: instead of hardcoded The first format is compact. The second format is quite verbose, but it allows the individual fields to be put as printable columns, or to be filtered by them. Another approach would be to put the whole state as one annotation, but it is complicated by merging of the existing state with its updates (unlike REST merge-patching, where different fields are merged inside of a dict server-side, one single field is always fully overwritten and requires client-side merging). I will try to do some magic on that to make it nice and simple. But this can be considered as a general direction for built-in resources in the future. For custom resources, I prefer to keep |
|
One annotation per handler looks already good to me 👍 |
|
Any update? |
|
Sorry, I was a bit busy with this coronavirus aftermath at work recently (well, not "a bit", actually). Nevertheless, I finally have some spare time now, and some of the framework's tasks are shifted from my private time to my work time — so, let's see how much faster it will go now and in the next few days. PS: For custom resources, there is a workaround described in #321 (with |
|
Tested kopf==0.27rc5 (that includes #331) and now if works as expected. Thanks! |
Long story short
If an exception occur inside a handler for
volumeattachmentsobjects, the handler will never get retried.Looking at the logs, it seems that kopf try to patch the object status with
{"status":{"kopf": {"processing": ...}}}This patch call succeed but the object status is not changed. I guess this is the reason no further attempts are made.
In theory the
/statusendpoint should be used to patch the status, but from my testing this have no effects as well. It seems like custom fields are not accepted inside"status".Possible fix: use an annotation to store kopf status?
Description
The code snippet to reproduce the issue
test_va.py
va.yaml
The exact command to reproduce the issue
kopf run test_va.py & kubectl create -f va.yamlThe full output of the command that failed
Environment
The text was updated successfully, but these errors were encountered: