[Feature] Run build pods on dedicated worker node

[metacpp]

In some cases, we want the build pod to run on specific nodes which satisfy some pre-conditions (os, role, etc.). Can we expose an interface in kpack configuration to support this ?

[matthewmcnew]

To accomplish this we will need to allow users to specify the node affinity, taints & tolerations, nodeSelector on Image and Build resources.

If we took the same structure from the pod spec that would look something like this:

apiVersion: kpack.io/v1alpha1
kind: Image
metadata:
  name: sample
spec:
  tag: sample/image
  builder:
    kind: ClusterBuilder
    name: some-builder
  serviceAccount: service-account
  source:
    git:
      url: https://github.com/buildpack/sample-java-app.git
      revision: master
  nodeSelector:
    disktype: ssd
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/e2e-az-name
            operator: In
            values:
            - e2e-az1
            - e2e-az2

[matthewmcnew]

Tagging this as 0.4.0 to target releasing alongside the api changes in our 0.4.0 release.

[tomkennedy513]

After talking about this we think it makes sense to add the ability to specify tolerations that will propagate down to the build pods as well

[skar07]

Any updates on this (an ETA would be nice)? This is essential for deployments based on roles for us

[KennethGomez]

Yea, it's needed for me too, an ETA would be nice

[loewenstein]

Would it make sense to allow full Pod templates in Image and Build, similar to Deployment and StatefulSet work?

[thomasDelaporte]

It would be nice to have a release for this before 0.4.0, I think this is an essential feature.

[vrishtrix]

I +1 @thomasDelaporte. Would be great if the feature's rolled out at its earliest.

[badlanguage]

Would it make sense to allow full Pod templates in Image and Build, similar to Deployment and StatefulSet work?

I think having the full pod spec might be too much (although I don't know the quirks of this), if we can have the taints & affinity added to the Builder then that should suffice.

My understanding so far is this:

• One would create a Builder with a nodeSelector or with a similar taint
• Once an image is created and the Builder is referenced
• The builder would build the image in the designated nodes
• The image would then also be stored on a designated node
• The image can then be used to create Pods as one would with Kind: Pod

Correct me if I went wrong somewhere.

WRT to the ETA, I think it would be better if we can ship this before the 0.4 as well, since it is a very important feature for most use cases. Would love to hear your thoughts @matthewmcnew

[Blidnight]

I would also love this feature for my organisation, for Builder, ClusterBuilder and Image (for storage)

[loewenstein]

I think having the full pod spec might be too much

It would immediately open up for anything one can configure on Pods, including things coming with future k8s versions.

In the end both image and build end up running a Pod, so the comparison with other workload creating resources doesn't seem to be too far off.

[matthewmcnew]

@skar07 @KennethGomez @thomasDelaporte I am tentatively planning on drafting an RFC for this that will allow users to configure affinity/taints/tolerations. Assuming the RFC is accepted we would likely be able to release this as part of 0.4 with the new v1alpha2 api.

Could you better elaborate on how/why you would like this functionality?

[matthewmcnew]

Would it make sense to allow full Pod templates in Image and Build, similar to Deployment and StatefulSet work?

At the moment, kpack build pods are fairly complicated with a set of init containers and multiple configured volumes. These volumes are intentionally mounted in certain steps to avoid exposing credentials to the buildpacks/application source code. So, I am not sure how we would expose a configurable and mergeable full pod spec on build.

[loewenstein]

Would it make sense to allow full Pod templates in Image and Build, similar to Deployment and StatefulSet work?

At the moment, kpack build pods are fairly complicated with a set of init containers and multiple configured volumes. These volumes are intentionally mounted in certain steps to avoid exposing credentials to the buildpacks/application source code. So, I am not sure how we would expose a configurable and mergeable full pod spec on build.

I think that Kubernetes strategic merge could be a solution worth investigating.

I am not sure this is the right direction, just seeing a risk of making more and more of the Pod configuration space available in the kpack API explicitly.

[skar07]

@skar07 @KennethGomez @thomasDelaporte I am tentatively planning on drafting an RFC for this that will allow users to configure affinity/taints/tolerations. Assuming the RFC is accepted we would likely be able to release this as part of 0.4 with the new v1alpha2 api.

Could you better elaborate on how/why you would like this functionality?

We have some projects that are designed for the Windows platforms and other cases where we provide SaaS services for end users, which requires building in certain locations (on site), and having access to the nodeSelector, affinity etc. would allow us to achieve this and decide which nodes handle the builds (and indirectly the host OS, geographic location etc.).
And the format provided here: #621 (comment) will be ideal for our use case.
Additionally, it would be great if this functionality is extended on to where the images are stored for the Builder & ClusterBuilder

[matthewmcnew]

I created an (currently WIP) RFC directly related to this issue here: #812