Support generating Kustomize friendly annotation for CRDs

[tamalsaha]

https://kubectl.docs.kubernetes.io/pages/examples/kustomize.html

# Each entry in this list should be a relative path to
# a file for custom resource definition(CRD) in openAPI definition.
#
# The presence of this field is to allow kustomize be
# aware of CRDs and apply proper
# transformation for any objects in those types.
#
# Typical use case: A CRD object refers to a ConfigMap object.
# In kustomization, the ConfigMap object name may change by adding namePrefix, nameSuffix, or hashing
# The name reference for this ConfigMap object in CRD object need to be
# updated with namePrefix, nameSuffix, or hashing in the same way.
#
# The annotations can be put into openAPI definitions are:
#   "x-kubernetes-annotation": ""
#   "x-kubernetes-label-selector": ""
#   "x-kubernetes-identity": ""
#   "x-kubernetes-object-ref-api-version": "v1",
#   "x-kubernetes-object-ref-kind": "Secret",
#   "x-kubernetes-object-ref-name-key": "name",
crds:
- crds/typeA.json
- crds/typeB.json

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MariusSchiffer]

I think this is still a useful thing.

Example: A CRD uses PodTemplate, which contains configMapRef. This reference will not be updated by kustomize for the configMapGenerator as it cannot see that this is a configMap reference.

[k8s-ci-robot]

@MariusSchiffer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

I think this is still a useful thing.

Example: A CRD uses PodTemplate, which contains configMapRef. This reference will not be updated by kustomize for the configMapGenerator as it cannot see that this is a configMap reference.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sbueringer]

I don't get it. What is the concrete ask here? Can you give an example what exactly controller-gen could support here?

[MariusSchiffer]

controller-gen should add x-kubernetes-object-ref-kind and x-kubernetes-object-ref-api-version to the generated CRD schema for core kubernetes objects (or all).
This allows kustomize (and maybe other tools) to know where to find the object definition.

An example why this is useful is kustomize's configMapGenerator feature.
When kustomize generates a configMap, it replaces all mentions of this configMap in configMapRef in the processed resources with the generated name.
But this only works if kustomize correctly recognizes it as a configMapRef. For this, during CRD loading, it checks x-kubernetes-object-ref-kind, see here: https://github.com/kubernetes-sigs/kustomize/blob/880a7a0187f0cb34132baa706d4a48d1ef5b8912/api/internal/accumulator/loadconfigfromcrds.go#L163

Here is an example: kubernetes-sigs/kustomize#681 (comment)

For another concrete example, elastic's cloud-on-k8s has CRDs which includes podTemplate (directly included from kubernetes definitions). Same problem that kustomize cannot recognize the podTemplate.

Definitions here: https://github.com/elastic/cloud-on-k8s/blob/main/pkg/apis/beat/v1beta1/beat_types.go#L96
Output here: https://github.com/elastic/cloud-on-k8s/blob/main/config/crds/v1/resources/beat.k8s.elastic.co_beats.yaml#L89

They actually remove everything below podTemplate as duplicating core kubernetes CRD parts becomes large quickly: https://github.com/elastic/cloud-on-k8s/blob/main/config/crds/v1/patches/beat-patches.yaml#L3

I'm not sure if this is the best approach (I feel like there could be something more native-y than using a x- property).
It also looks like this would require changing JSONSchemaProps.

[JoelSpeed]

If I've read this correctly, you would need to re-ignite kubernetes/kubernetes#82942 and get this support into Kube as part of the CRD apiextensions-apiserver before changing anything in controller-tools.

JSONSchemaProps relates to a Kubernetes API, it's not part of this repo, we cannot just change it, and expect it to work

[MariusSchiffer]

After some searching I found more on kustomize's side, so this has to be solved there: kubernetes-sigs/kustomize#3418
Thank you for the fast reply