Use pulumi.com labeling scheme

[EronWright]

Proposed changes

This PR switches over to a different labeling strategy for:

• the child resources of workspaces (statefulsets, services, pods, secrets), and
• the child resources of stacks (workspaces, updates)

The rationale here is to avoid using the app.kubernetes.io labels because they have various meanings, the user might also be using them, and may trigger unwanted behavior like object tracking in argocd. PKO uses labels for a functional purpose, e.g. in the selectors.

Regarding the specific labels, this PR seeks to relate the labels the apigroup that is using them. The workspace controller uses labels in the auto.pulumi.com API group. The stack controller uses labels in the pulumi.com API group.

Workspace

Before:

apiVersion: auto.pulumi.com/v1alpha1
kind: Workspace
metadata:
  labels:
    app.kubernetes.io/component: stack
    app.kubernetes.io/instance: random-yaml
    app.kubernetes.io/managed-by: pulumi-kubernetes-operator
    app.kubernetes.io/name: pulumi

After:

apiVersion: auto.pulumi.com/v1alpha1
kind: Workspace
metadata:
  labels:
    pulumi.com/component: stack             # <--- the workspace is a part of the stack
    pulumi.com/stack-name: random-yaml

Note that this change causes replacement of the statefulset, due to some recovery logic that we have in the workspace controller.

Update

Labels weren't previously applied.

After:

apiVersion: auto.pulumi.com/v1alpha1
kind: Update
metadata:
  labels:
    pulumi.com/component: stack       # <--- the update is a part of the stack
    pulumi.com/stack-name: random-yaml

Service

Before:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: workspace
    app.kubernetes.io/instance: random-yaml
    app.kubernetes.io/managed-by: pulumi-kubernetes-operator
    app.kubernetes.io/name: pulumi

After:

apiVersion: v1
kind: Service
metadata:
  labels:
    auto.pulumi.com/component: workspace               # <--- the service is a part of the workspace
    auto.pulumi.com/workspace-name: random-yaml

Similar changes for the StatefulSet, for the Pod, and for the Secret made by each Update.

Related issues (optional)

Closes #819

[rquitales]

Is there a rationale for removing the well known app.kubernetes.io/managed-by label? This makes it obvious to other tools what is managing these child objects.

[EronWright]

I was hesitant to use a subset of app labels, seemed a bit weird. The definition (ref) of the app.kubernetes.io/managed-by label is:

The tool being used to manage the operation of an application.

To me, it identifies the tool that is servicing the application at large. All these app labels seem to express a relationship to an application. The app concept may exist at a higher level, e.g. an ArgoCD Application. Since PKO doesn't know the application to which the workload is associated, it should not apply this label.

[rquitales]

Since PKO doesn't know the application to which the workload is associated, it should not apply this label.

I disagree. The objects we are creating here (Workspace, Statefulset) are all created by our own PKO and should also be managed by PKO. These objects are what enables PKO to work right now. What happens if another controller/tool takes over control of these objects? We'd likely get into a controller fight here. Though practically, these labels are most generally used for metrics, so having the app.kubernetes.io/managed-by would allow users to find metrics for PKO created/managed objects in their current dashboards.

[EronWright]

They could also use the pulumi.com labels to identity these pods.

If you were to deploy an EKS cluster with various addons, I would guess that you would not see app.kubernetes.io/managed-by on the majority of the objects. It just isn't as common as you're suggesting.

At least it could be considered a separate enhancement?

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 52.25%. Comparing base (14d4d3f) to head (97db54d).
Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
...r/internal/controller/auto/workspace_controller.go	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #829      +/-   ##
==========================================
+ Coverage   52.18%   52.25%   +0.06%     
==========================================
  Files          32       32              
  Lines        4501     4507       +6     
==========================================
+ Hits         2349     2355       +6     
  Misses       1959     1959              
  Partials      193      193              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rquitales]

While I disagree with removing app.kubernetes.io/managed-by, I won't block given the timeline for v2 release. We can discuss this offline.