Update cluster controller docs with more detail about turndown and breaking changes #246

michaelmdresser · 2022-05-27T21:03:18Z

No description provided.

No reason to repeat instructions that are already available in an OSS repo.

michaelmdresser · 2022-06-02T16:43:41Z

I just added a migration guide to this PR for copying turndownschedules.kubecost.k8s.io resources to turndownschedules.kubecost.com. These instructions were tested by running a series of experiments with both CRD definitions on GKE clusters running K8s v1.21.

I also observed behavior of upgrading a GKE cluster to K8s v1.22 with turndownschedules.kubecost.k8s.io already defined. (turndownschedules.kubecost.k8s.io cannot be deployed freshly on K8s v1.22+ clusters because of policy enforcement of the .k8s.io namespace). Based on that experiment, these instructions should also reliably work on K8s v1.22+.

michaelmdresser · 2022-06-02T19:30:52Z

Migration testing notes, for reference.

Create K8s v1.21 cluster

gcloud container clusters create \
    "michael-crd-test" \
    --project "guestbook-227502" \
    --region "us-central1-c" \
    --network "michael-vpc" \
    --subnetwork "us-central1" \
    --release-channel "stable" \
    --cluster-version "1.21.11-gke.1100" \
    --num-nodes 1

kubectl get node

NAME                                              STATUS   ROLES    AGE     VERSION
gke-michael-crd-test-default-pool-7ecd71f0-j675   Ready    <none>   2m20s   v1.21.11-gke.1100

kubectl version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"archive", BuildDate:"2022-05-04T15:25:22Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.11-gke.1100", GitCommit:"20da4c21b3a6b1a56ff6ad5ecb7dee013aaf1b83", GitTreeState:"clean", BuildDate:"2022-04-01T09:40:07Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

Install KC v1.93.2 with cluster controller enabled

helm repo update
helm install kubecost kubecost/cost-analyzer --create-namespace --namespace kubecost --version "1.93.2" --set "clusterController.enabled=true"

kubectl get deployment -n kubecost

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
kubecost-cluster-controller   0/1     1            0           2s
kubecost-cost-analyzer        0/1     1            0           2s
kubecost-grafana              0/1     1            0           2s
kubecost-kube-state-metrics   0/1     1            0           2s
kubecost-prometheus-server    0/1     1            0           2s

Check defined CRDs

kubectl get crd | grep -i 'kubecost'

turndownschedules.kubecost.k8s.io                2022-06-02T19:22:03Z

kubectl get crd turndownschedules.kubecost.k8s.io -o yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: kubecost
    meta.helm.sh/release-namespace: kubecost
  creationTimestamp: "2022-06-02T19:22:03Z"
  generation: 1
  labels:
    app: cost-analyzer
    app.kubernetes.io/instance: kubecost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cost-analyzer
    helm.sh/chart: cost-analyzer-1.93.2
  name: turndownschedules.kubecost.k8s.io
  resourceVersion: "2062"
  uid: e8a8fbca-12bb-48b5-97e7-d1baa33f2df8
spec:
  conversion:
    strategy: None
  group: kubecost.k8s.io
  names:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  preserveUnknownFields: true
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - description: The state of the turndownschedule
      jsonPath: .status.state
      name: State
      type: string
    - description: The next turndown date-time
      jsonPath: .status.nextScaleDownTime
      name: Next Turndown
      type: string
    - description: The next turn up date-time
      jsonPath: .status.nextScaleUpTime
      name: Next Turn Up
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              end:
                format: date-time
                type: string
              repeat:
                enum:
                - none
                - daily
                - weekly
                type: string
              start:
                format: date-time
                type: string
            type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  conditions:
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: '[spec.preserveUnknownFields: Invalid value: true: must be false, spec.versions[0].schema.openAPIV3Schema.type:
      Required value: must not be empty at the root]'
    reason: Violations
    status: "True"
    type: NonStructuralSchema
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: protected groups must have approval annotation "api-approved.kubernetes.io",
      see https://github.com/kubernetes/enhancements/pull/1111
    reason: MissingAnnotation
    status: "False"
    type: KubernetesAPIApprovalPolicyConformant
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1alpha1

Create a TurndownSchedule

read -r -d '' TDS << END
apiVersion: kubecost.k8s.io/v1alpha1
kind: TurndownSchedule
metadata:
  name: test-schedule
  finalizers:
  - "finalizer.kubecost.k8s.io"
spec:
  start: 2022-06-12T00:00:00Z
  end: 2022-06-12T06:00:00Z
  repeat: weekly
END

echo "${TDS}" | kubectl apply -f -

turndownschedule.kubecost.k8s.io/test-schedule created

Check CRD and version

kubectl get tds -o yaml

apiVersion: v1
items:
- apiVersion: kubecost.k8s.io/v1alpha1
  kind: TurndownSchedule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kubecost.k8s.io/v1alpha1","kind":"TurndownSchedule","metadata":{"annotations":{},"finalizers":["finalizer.kubecost.k8s.io"],"name":"test-schedule"},"spec":{"end":"2022-06-12T06:00:00Z","repeat":"weekly","start":"2022-06-12T00:00:00Z"}}
    creationTimestamp: "2022-06-02T19:22:16Z"
    finalizers:
    - finalizer.kubecost.k8s.io
    generation: 1
    name: test-schedule
    resourceVersion: "2288"
    uid: d8e487ad-17cb-49b8-90b6-60eaae1964ee
  spec:
    end: "2022-06-12T06:00:00Z"
    repeat: weekly
    start: "2022-06-12T00:00:00Z"
kind: List
metadata:
  resourceVersion: ""

Upgrade KC from dev branch

In Helm chart repo

git checkout mmd/update-cluster-controller-v0.1.0

helm upgrade kubecost ./cost-analyzer --namespace kubecost --set "clusterController.enabled=true"

Check defined CRDs
Should be .k8s.io AND .kubecost.com

kubectl get crd | grep -i 'kubecost'

turndownschedules.kubecost.com                   2022-06-02T19:22:31Z
turndownschedules.kubecost.k8s.io                2022-06-02T19:22:03Z

kubectl get crd turndownschedules.kubecost.k8s.io -o yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: kubecost
    meta.helm.sh/release-namespace: kubecost
  creationTimestamp: "2022-06-02T19:22:03Z"
  deletionTimestamp: "2022-06-02T19:22:36Z"
  finalizers:
  - customresourcecleanup.apiextensions.k8s.io
  generation: 1
  labels:
    app: cost-analyzer
    app.kubernetes.io/instance: kubecost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cost-analyzer
    helm.sh/chart: cost-analyzer-1.93.2
  name: turndownschedules.kubecost.k8s.io
  resourceVersion: "2504"
  uid: e8a8fbca-12bb-48b5-97e7-d1baa33f2df8
spec:
  conversion:
    strategy: None
  group: kubecost.k8s.io
  names:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  preserveUnknownFields: true
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - description: The state of the turndownschedule
      jsonPath: .status.state
      name: State
      type: string
    - description: The next turndown date-time
      jsonPath: .status.nextScaleDownTime
      name: Next Turndown
      type: string
    - description: The next turn up date-time
      jsonPath: .status.nextScaleUpTime
      name: Next Turn Up
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              end:
                format: date-time
                type: string
              repeat:
                enum:
                - none
                - daily
                - weekly
                type: string
              start:
                format: date-time
                type: string
            type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  conditions:
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: '[spec.preserveUnknownFields: Invalid value: true: must be false, spec.versions[0].schema.openAPIV3Schema.type:
      Required value: must not be empty at the root]'
    reason: Violations
    status: "True"
    type: NonStructuralSchema
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: protected groups must have approval annotation "api-approved.kubernetes.io",
      see https://github.com/kubernetes/enhancements/pull/1111
    reason: MissingAnnotation
    status: "False"
    type: KubernetesAPIApprovalPolicyConformant
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2022-06-02T19:22:03Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  - lastTransitionTime: "2022-06-02T19:22:36Z"
    message: CustomResource deletion is in progress
    reason: InstanceDeletionInProgress
    status: "True"
    type: Terminating
  storedVersions:
  - v1alpha1

kubectl get crd turndownschedules.kubecost.com -o yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: kubecost
    meta.helm.sh/release-namespace: kubecost
  creationTimestamp: "2022-06-02T19:22:31Z"
  generation: 1
  labels:
    app: cost-analyzer
    app.kubernetes.io/instance: kubecost
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cost-analyzer
    helm.sh/chart: cost-analyzer-1.92.0
  name: turndownschedules.kubecost.com
  resourceVersion: "2434"
  uid: cc8d8af6-577a-44ca-bfb4-f32c321cdebd
spec:
  conversion:
    strategy: None
  group: kubecost.com
  names:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - description: The state of the turndownschedule
      jsonPath: .status.state
      name: State
      type: string
    - description: The next turndown date-time
      jsonPath: .status.nextScaleDownTime
      name: Next Turndown
      type: string
    - description: The next turn up date-time
      jsonPath: .status.nextScaleUpTime
      name: Next Turn Up
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              end:
                format: date-time
                type: string
              repeat:
                enum:
                - none
                - daily
                - weekly
                type: string
              start:
                format: date-time
                type: string
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: TurndownSchedule
    listKind: TurndownScheduleList
    plural: turndownschedules
    shortNames:
    - td
    - tds
    singular: turndownschedule
  conditions:
  - lastTransitionTime: "2022-06-02T19:22:31Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2022-06-02T19:22:31Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  storedVersions:
  - v1alpha1

Check existing turndownschedule.kubecost.k8s.io resources
(should be one)

kubectl get turndownschedule.kubecost.k8s.io -o yaml

apiVersion: v1
items:
- apiVersion: kubecost.k8s.io/v1alpha1
  kind: TurndownSchedule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kubecost.k8s.io/v1alpha1","kind":"TurndownSchedule","metadata":{"annotations":{},"finalizers":["finalizer.kubecost.k8s.io"],"name":"test-schedule"},"spec":{"end":"2022-06-12T06:00:00Z","repeat":"weekly","start":"2022-06-12T00:00:00Z"}}
    creationTimestamp: "2022-06-02T19:22:16Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2022-06-02T19:22:36Z"
    finalizers:
    - finalizer.kubecost.k8s.io
    generation: 2
    name: test-schedule
    resourceVersion: "2511"
    uid: d8e487ad-17cb-49b8-90b6-60eaae1964ee
  spec:
    end: "2022-06-12T06:00:00Z"
    repeat: weekly
    start: "2022-06-12T00:00:00Z"
kind: List
metadata:
  resourceVersion: ""

Check if any turndownschedule.kubecost.com resources exist
(they shouldn’t)
```
kubectl get turndownschedule.kubecost.com
```

Follow migration steps

Copy resources

kubectl get turndownschedules.kubecost.k8s.io -o yaml \
    | sed 's|kubecost.k8s.io|kubecost.com|' \
    | kubectl apply -f -

turndownschedule.kubecost.com/test-schedule created

Check new resources

kubectl get turndownschedule.kubecost.com -o yaml

apiVersion: v1
items:
- apiVersion: kubecost.com/v1alpha1
  kind: TurndownSchedule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kubecost.com/v1alpha1","kind":"TurndownSchedule","metadata":{"annotations":{},"creationTimestamp":"2022-06-02T19:22:16Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2022-06-02T19:22:36Z","finalizers":["finalizer.kubecost.com"],"generation":2,"name":"test-schedule","resourceVersion":"2511","uid":"d8e487ad-17cb-49b8-90b6-60eaae1964ee"},"spec":{"end":"2022-06-12T06:00:00Z","repeat":"weekly","start":"2022-06-12T00:00:00Z"}}
    creationTimestamp: "2022-06-02T19:23:22Z"
    finalizers:
    - finalizer.kubecost.com
    generation: 1
    name: test-schedule
    resourceVersion: "2862"
    uid: 405fab31-84bf-4730-a94a-8596a4614937
  spec:
    end: "2022-06-12T06:00:00Z"
    repeat: weekly
    start: "2022-06-12T00:00:00Z"
kind: List
metadata:
  resourceVersion: ""

Check “default” resource type returned by kubectl get tds

kubectl get tds -o yaml

apiVersion: v1
items:
- apiVersion: kubecost.com/v1alpha1
  kind: TurndownSchedule
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"kubecost.com/v1alpha1","kind":"TurndownSchedule","metadata":{"annotations":{},"creationTimestamp":"2022-06-02T19:22:16Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2022-06-02T19:22:36Z","finalizers":["finalizer.kubecost.com"],"generation":2,"name":"test-schedule","resourceVersion":"2511","uid":"d8e487ad-17cb-49b8-90b6-60eaae1964ee"},"spec":{"end":"2022-06-12T06:00:00Z","repeat":"weekly","start":"2022-06-12T00:00:00Z"}}
    creationTimestamp: "2022-06-02T19:23:22Z"
    finalizers:
    - finalizer.kubecost.com
    generation: 1
    name: test-schedule
    resourceVersion: "2862"
    uid: 405fab31-84bf-4730-a94a-8596a4614937
  spec:
    end: "2022-06-12T06:00:00Z"
    repeat: weekly
    start: "2022-06-12T00:00:00Z"
kind: List
metadata:
  resourceVersion: ""

Check CRDs

kubectl get crd | grep -i 'kubecost'

turndownschedules.kubecost.com                   2022-06-02T19:22:31Z
turndownschedules.kubecost.k8s.io                2022-06-02T19:22:03Z

Remove finalizer from old resources

kubectl patch \
    crd/turndownschedules.kubecost.k8s.io \
    -p '{"metadata":{"finalizers":[]}}' \
    --type=merge

customresourcedefinition.apiextensions.k8s.io/turndownschedules.kubecost.k8s.io patched

Post-patch, existing turndownschedules.kubecost.k8s.io resources disappear because Helm tried to delete the old CRD during the upgrade but it remains pending because of the finalizer. Once patched, the old CRD is gone:

kubectl get crd | grep -i 'kubecost'

turndownschedules.kubecost.com                   2022-06-02T19:22:31Z

kubectl get tds

NAME            STATE   NEXT TURNDOWN   NEXT TURN UP
test-schedule

nikovacevic

A+ documentation!

AjayTripathy · 2022-06-03T04:53:01Z

@kirbsauce can we make sure the support team is up to date on this for any questions about cluster turndown migration?

michaelmdresser added 2 commits May 27, 2022 17:00

Update Cluster Controller docs: link to turndown

a148bd0

No reason to repeat instructions that are already available in an OSS repo.

Add breaking change note for turndown CRDs

105338b

michaelmdresser added the v1.94 label May 27, 2022

michaelmdresser mentioned this pull request May 27, 2022

Update cluster-controller to v0.1.0 kubecost/cost-analyzer-helm-chart#1444

Merged

Add TurndownSchedule migration guide for v1.94+

2fe5372

Note unnecessary delete command

23faa59

nikovacevic approved these changes Jun 2, 2022

View reviewed changes

michaelmdresser merged commit bd6096f into main Jun 2, 2022

michaelmdresser deleted the mmd/update-cluster-controller-scaledown branch June 2, 2022 20:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update cluster controller docs with more detail about turndown and breaking changes #246

Update cluster controller docs with more detail about turndown and breaking changes #246

michaelmdresser commented May 27, 2022

michaelmdresser commented Jun 2, 2022

michaelmdresser commented Jun 2, 2022

nikovacevic left a comment

AjayTripathy commented Jun 3, 2022

Update cluster controller docs with more detail about turndown and breaking changes #246

Update cluster controller docs with more detail about turndown and breaking changes #246

Conversation

michaelmdresser commented May 27, 2022

michaelmdresser commented Jun 2, 2022

michaelmdresser commented Jun 2, 2022

Migration testing notes, for reference.

nikovacevic left a comment

Choose a reason for hiding this comment

AjayTripathy commented Jun 3, 2022