Store handler progress data in an annotation, not the status subresource

[lack]

Problem

As of apiextensions.k8s.io/v1 it is much more difficult to create a CRD that allows unvalidated information, as an openapi schema is required for even the status subresource. Tracking the handler progress in an annotation instead of the status subresource means it's easier to use kopf to write operators for CRDs that we don't own.

To define a wide-open status subresource in a v1 CRD, you can do the following:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kopfexamples.zalando.org
spec:
  scope: Namespaced
  group: zalando.org
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                # Main spec validation schema...
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true

Or for something more structured you can allow only the kopf property of the status subresource to be unvalidated as follows:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kopfexamples.zalando.org
spec:
  scope: Namespaced
  group: zalando.org
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                # Main spec validation schema...
            status:
              type: object
              properties:
                result:
                  type: string
                  description: The result from the handler
                  enum:
                  - created
                  - updated
                # Other structured subresource schema data...
                kopf:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true

Proposal

Kopf already stores important metadata in annotations including kopf.zalando.org/last-handled-configuration and kubectl.kubernetes.io/last-applied-configuration. Setting up the state.py mechanism to similarly use an annotation instead of the status subresource for its progress data should, I hope, be relatively straight-forward.

For resources whose CRDs can be altered, the other mechanisms to allow status.kopf.progress is an okay work-around. But the v1 schema enforcement means that kopf can't store its progress data there any more if the CRD cannot be altered.

Checklist

• Many users can benefit from this feature, it is not a one-time case
• The proposal is related to the K8s operator framework, not to the K8s client libraries

[flo-02-mu]

I am running into this issue when watching a k8s service object. The patch command does not return any error, but no kopf status is added. Thus, no retry attempt is made in case of exceptions.

[mnarodovitch]

I also had the same issues, but in an other context.
Seems, that the CustomResourceDefinition API silently ignores fields, which are not explicitly preserved.

[gtsystem]

Related to bug #308 where the same issue is visible for standard resources

[asteven]

Workaround for persisting the handler return values:

from functools import wraps
import json


def annotate_results(function=None, key='kopf.zalando.org/handler-results'):
    """A decorator that persists handler results as annotations on the
    resource.

    Before calling a handler, load any existing results from
    annotations and pass them as the keyword argument 'results'
    to the handler.

    Store any outcome returned by the handler in the annotation.

    Note that this implementation does not (yet) work with async handlers.
    """
    def actual_decorator(f):
        @wraps(f)
        def wrapper(*args, **kwargs):
            meta = kwargs['meta']
            results_json = meta['annotations'].get(key, None)
            if results_json:
                results = json.loads(results_json)
            else:
                results = {}
            kwargs['results'] = results
            result = f(*args, **kwargs)
            if result:
                results[f.__name__] = result
                patch = kwargs['patch']
                patch.metadata.annotations[key] = json.dumps(results)

            # We don't return the result as we have handled it ourself.
            # Otherwise kopf tries to store it again in the objects status
            # which doesn't work anyway on k8s >= 1.16.
            #return result
        return wrapper
    if function:
        return actual_decorator(function)
    return actual_decorator

Example usage with default annotation key:

@kopf.on.create('', 'v1', 'persistentvolumeclaims')
@kopf.on.update('', 'v1', 'persistentvolumeclaims')
@annotate_results
def create_results_1(name, **_):
    return {'hello': 'from create_results_1'}

@kopf.on.update('', 'v1', 'persistentvolumeclaims')
@annotate_results
def show_results_1(name, results, **_):
    import pprint
    pprint.pprint(results)

Example usage with custom annotation key:

@kopf.on.create('', 'v1', 'persistentvolumeclaims')
@kopf.on.update('', 'v1', 'persistentvolumeclaims')
@annotate_results(key='zfs-provisioner/handler-results')
def create_results_2(name, **_):
    return {'hello': 'from create_results_2'}


@kopf.on.update('', 'v1', 'persistentvolumeclaims')
@annotate_results(key='zfs-provisioner/handler-results')
def show_results_2(name, results, **_):
    import pprint
    pprint.pprint(results)

[nolar]

This was solved in #331 and follow-ups, and is available since kopf>=0.27. Some fixes were done later for sub-handlers (nolar/kopf#517), and some improvements for readability of json (nolar/kopf#516) — available since kopf>=0.28rc2.