Provide support for CRD upgrades natively in zarf

[mjnagel]

Is your feature request related to a problem? Please describe.

Today zarf's posture for CRD upgrades is identical to helm's - https://docs.zarf.dev/ref/deploy/#handling-customresourcedefinitions-crds. While this does make sense given zarf is using helm under the hood it means that the end user is left to figure out how to upgrade CRDs on their own. Typically this means either the chart needs to handle upgrades (I've seen this done with k8s jobs that apply the CRD manifests) or it has to be handled via the zarf package. Handling via the zarf package is certainly possible, but not ideal (I think it would have to be done with files + actions, since even manifests get wrapped in a helm chart and would rely on helm's behavior).

Describe the solution you'd like

I would like zarf to provide config options for CRDs per chart. By default this would align with the upstream helm default behavior (install only, don't touch upgrades), but could be configured based on package needs. This could mean something like a new field in the helm chart component spec to specify whether to do CRD upgrades or not:

components:
  - name: my-chart
    charts:
      - name: my-chart
        url: https://mychart.com
        version: 2.0.0
        namespace: myns
        crdUpgrade: true

When this field is set Zarf would handle CRD upgrades itself (this would have to be done via k8s api/sdk directly rather than helm given the helm posture on CRDs).

Describe alternatives you've considered

Continue down the current path, shifting the CRD upgrade responsibility to the end user to handle either via the helm chart or some zarf package changes. Perhaps to ease the burden on end users and example package could be provided to show how CRD upgrades would be expected to take place.

Additional context

Another project in a similar space of "managing helm installs" is Flux - they have build a similar spec around their HelmRelease which allows for configuring CRD upgrades. I'm not sure if their project/code would be a useful place to look for inspiration on how to approach this problem differently.

I do think CRD upgrades are likely to be a common problem/need in the future - many upstream charts leave this to the end user as a manual upgrade (ex: kube-prom-stack). It would be great if zarf could provide something to lessen this burden.

[phillebaba]

I think we should just follow a similar method to what Flux does. I was partially involved in the discussions at the time, and it became clear that managing CRDs was required. Implementing this should be possible for Zarf as we already have the chart so getting the CRDs from the crd directory in the chart should be simple. We should also keep to Flux choice to not delete CRDs and only create or update them.

[mjnagel]

@phillebaba yeah I think that would be a great path to go down. One thing I was always curious on was the decision to only "manage" CRDs if they are in the crd directory. In my experience with a number of the opensource charts we use for uds-core there's definitely a handful of charts that use templates for CRDs instead. I wonder if it would possible/reasonable to use the kind only (and not consider directory) to determine what is a CRD we want to manage?

[phillebaba]

Here is a user example that does not involve the Helm crd directory.

kind: ZarfPackageConfig
metadata:
  name: knative
  description: Deploys Knative using Manifest files
  version: 0.0.1
  authors: Justin Bailey
  documentation: https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/
  source: https://github.com/knative/serving

components:
  - name: knative-serving
    required: true
    description: "Knative Serving"
    manifests:
      - name: knative-crds
        files:
          - https://github.com/knative/serving/releases/download/knative-v1.14.1/serving-crds.yaml
      - name: knative-serving
        namespace: knative-serving
        kustomizations:
          - manifests/
    images:
      - gcr.io/knative-releases/knative.dev/serving/cmd/activator:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/controller:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/queue:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/webhook:v1.14.1
      - gcr.io/knative-releases/knative.dev/net-kourier/cmd/kourier:v1.14.0
      - docker.io/envoyproxy/envoy:v1.26-latest

[Miaoxiang-philips]

I also think it would be very meaningful if Zarf could support this feature.

If there isn't a good solution for now, could we consider having a dedicated document on Zarf CRDs upgrades to provide some suggestions or alternatives?

[mjnagel]

I did some digging and reading today and believe my initial understanding of Helm and CRDs was a bit flawed. It seems like "helm does not upgrade CRDs" only applies if using the special crds folder within a chart. Based on what I'm seeing. CRDs are actually upgraded as expected when they are in the templates folder. Depending on how a chart is architected, CRD upgrades may actually be possible with Zarf (for example I've successfully upgraded CRDs on the Istio base chart which uses the templates folder). I think these findings also line up with where the upgrade caveat is placed in the helm doc (note that it's nested under Method 1 and not mentioned under Method 2).

This still does not account for CRDs in the crds folder but I think this could still solve a number of scenarios. I'm unsure on the behavior if CRDs are included as zarf manifests. May be worth a doc update to explain this assuming my findings are correct here.

A few upstream helm code links that seem to backup that the special "skip upgrade" logic only applies to resources in the crds folder:

• https://github.com/helm/helm/blob/main/pkg/chart/chart.go#L154
• https://github.com/helm/helm/blob/a933bcc7d13733a2bef60a184c13fe3ed6703fb5/pkg/action/install.go#L264
• https://github.com/helm/helm/blob/a933bcc7d13733a2bef60a184c13fe3ed6703fb5/pkg/action/install.go#L171