Use a podTemplate inside the workspaceTemplate

[blampe]

We currently expect a fully-formed Pod spec, but we should use an apply configuration instead.

I have a branch where I started this work but I ran into a number of issues.

#713 (comment)

[alexstaeding]

I'll add some context for this from the slack conversation:

I was trying to add a pulumi plugin (for scala support) in my Stack custom resource for the operator. I found this issue/comment which suggests adding an init container to achieve this. (I assume Stack.spec.workspaceTemplate.spec.podTemplate.spec)

So far I've come across two issues with this approach, but I was just trying to get this to work relatively quickly so I may have missing something.

• The documentation mentions that containers "cannot be updated". The reality seems to be that a container must be specified, otherwise the stack spec is invalid e.g. The Stack "git" is invalid: spec.workspaceTemplate.spec.podTemplate.spec.containers: Required value. Specifying a dummy container works, and is added to the spec, although this should not be possible according to the docs - unless I misinterpreted "update". In any case, it certainly does not seem necessary.
• It seems that when the spec is merged, all entires that are merged put the elements of the patch spec first. This is a problem, because the initContainer for adding a plugin must run after the two default initContainers "bootstrap" and "fetch".

This is the quick & dirty spec I was trying to get to work:

workspaceTemplate:
    spec:
      podTemplate:
        spec:
          initContainers:
            - name: install-scala-cli-and-plugin
              image: curlimages/curl:latest
              command:
                - "/bin/sh"
                - "-c"
                - |
                  cd /share/workspace
                  curl -sSLf https://scala-cli.virtuslab.org/get | sh
                  pulumi plugin install language scala 0.3.2 --server github://api.github.com/VirtusLab/besom
          containers:
            - name: "dummy"
              image: busybox:latest

This is the resulting updated spec, which puts the custom initContainer before the others:

# ...
Pod Template:
  # ...
  Init Containers:
   install-scala-cli-and-plugin:
    Image:      curlimages/curl:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c
      cd /share/workspace
      curl -sSLf https://scala-cli.virtuslab.org/get | sh
      pulumi plugin install language scala 0.3.2 --server github://api.github.com/VirtusLab/besom
      
    Environment:  <none>
    Mounts:       <none>
   bootstrap:
    Image:      pulumi/pulumi-kubernetes-operator:v2.0.0-beta.3
    Port:       <none>
    Host Port:  <none>
    Args:
      cp
      /agent
      /tini
      /share/
    Environment:  <none>
    Mounts:
      /share from share (rw)
   fetch:
    Image:      pulumi/pulumi-kubernetes-operator:v2.0.0-beta.3
    Port:       <none>
    Host Port:  <none>
    Args:
      sh
      -c
      
      /share/agent init -t /share/source --git-url $GIT_URL --git-revision $GIT_REVISION &&
      ln -s /share/source/$GIT_DIR /share/workspace
          
    Environment:
      GIT_URL:       <...> # removed
      GIT_REVISION:  <...> # removed
      GIT_DIR:       
    Mounts:
      /share from share (rw)
  Containers:
   pulumi:
    Image:      docker.io/pulumi/pulumi:latest-nonroot
    Port:       50051/TCP
    Host Port:  0/TCP
    Command:
      /share/tini
      /share/agent
      --
      serve
      --workspace
      /share/workspace
      --skip-install
      --auth-mode
      kube
      --kube-workspace-namespace
      besom-start
      --kube-workspace-name
      git
    Requests:
      cpu:     100m
      memory:  64Mi
    Environment:
      POD_NAMESPACE:    (v1:metadata.namespace)
      POD_SA_NAME:      (v1:spec.serviceAccountName)
      AGENT_MEMLIMIT:  67108864 (requests.memory)
    Mounts:
      /share from share (rw)
   dummy:
    Image:        busybox:latest
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>

[project0]

The approach worked for me pretty well. Maybe there is just an misunderstanding how the template works?

The podTemplate is applied as a strategic merge patch (as described in the docs).

The documentation mentions that containers "cannot be updated". The reality seems to be that a container must be specified, otherwise the stack spec is invalid e.g. The Stack "git" is invalid: spec.workspaceTemplate.spec.podTemplate.spec.containers: Required value. Specifying a dummy container works, and is added to the spec, although this should not be possible according to the docs - unless I misinterpreted "update". In any case, it certainly does not seem necessary.

In your example you use a init container with just the curl image, there is no pulumi cli installed nor any volume mounted. So how should the plugin get into the main container executing pulumi? You just miss a bunch of things, maybe this example helps you #430 (comment) ?

[EronWright]

Thanks @alexstaeding and @project0 for the discussion, is there a specific change you think we should make, e.g. to the merging logic? How should we ensure that the init containers are correctly ordered?

[zackse]

@alexstaeding (documentation confusion aside) -- note that you can avoid adding a dummy container if you specify an empty list (due to the merge):

workspaceTemplate:
    spec:
      podTemplate:
        spec:
          initContainers:
            ...
          containers: []