Image architecture mismatch

[EronWright]

A follow-up to #430.

When deploying to a Kubernetes cluster that has arm64 nodes, the workspace pod may enter a crashloop due to an architecture mismatch between the image and the node. This can happen when a pod is scheduled to a node for which the correct architecture isn't available for any one of container images. Kubernetes doesn't have complete information about the images during scheduling, and not all of the Pulumi images are multi-architecture. The error may resemble:

Failed to pull image "pulumi/pulumi:latest-nonroot": rpc error: code = NotFound desc = 
  failed to pull and unpack image "docker.io/pulumi/pulumi:latest-nonroot": no match for platform in manifest: not found

There's at least three solutions:

1. Don't publish an arm64 build of pulumi-kubernetes-operator image.
2. Use a node affinity rule when a non-multi-arch image is being used. Maybe introduce an easier way to do that.
3. Publish multi-arch builds for all pulumi images: ARM and ARM64 docker containers pulumi-docker-containers#63

When one is using an architecture-specific pulumi image, perhaps one should also use a node affinity rule on the workspace pod spec to target that architecture. Some pulumi images are multi-arch, some aren't, so it isn't clear how to automate this.

For example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
          - key: "kubernetes.io/arch"
            operator: In
            values: ["amd64"]