Add in runner and starter possibility to configure allowPrivilegeEscalation and capabilities in container SecurityContext

[pasi-romo-idealo]

Feature Description

I tried to start following kind of test in our cluster:

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
  namespace: test
spec:
  parallelism: 4
  script:
    configMap:
      name: stress-test
      file: test.js

but this fails to work because it gets rejected by our policy engine, which requires the workloads to e.g. run as non-root and set seccompProfile. I noticed that it is possible to configure security contexts for the runner and starter so I tried something like this:

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
  namespace: test
spec:
  parallelism: 4
  script:
    configMap:
      name: stress-test
      file: test.js
  runner:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  starter:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

otherwise it works, but unfortunately the operator does not seem to support "allowPrivilegeEscalation" and "capabilities"-configurations i.e. I get following error, already while applying the manifest:

error: error validating "k6.yml": error validating data: [ValidationError(K6.spec.runner.securityContext): unknown field "allowPrivilegeEscalation" in io.k6.v1alpha1.K6.spec.runner.securityContext, ValidationError(K6.spec.runner.securityContext): unknown field "capabilities" in io.k6.v1alpha1.K6.spec.runner.securityContext, ValidationError(K6.spec.starter.securityContext): unknown field "allowPrivilegeEscalation" in io.k6.v1alpha1.K6.spec.starter.securityContext, ValidationError(K6.spec.starter.securityContext): unknown field "capabilities" in io.k6.v1alpha1.K6.spec.starter.securityContext]; if you choose to ignore these errors, turn validation off with --validate=false

I tried to find out if this could be updated in the operator-code, and it seemed that the issue is related to this:
https://github.com/grafana/k6-operator/blob/main/api/v1alpha1/k6_types.go#L40

i.e. corev1.PodSecurityContext is being used, which does not support these configurations since they should get configured actually at the container level.

For now I can get around this issue by e.g. defining policy exceptions for this specific case, but of course it would be much nicer if you could configure this already in the k6-kind.

If you have an idea how this could be achieved, I would be more than happy to test this out in our environment.

Suggested Solution (optional)

No response

Already existing or connected issues / PRs (optional)

No response

[yorugac]

Hi @pasi-romo-idealo, thanks for opening the issue! This is surely an interesting one.

Do I understand correctly, that in your setup both allowPrivilegeEscalation and capabilities are a requirement? I.e. you must pass them to container, otherwise you have to make policy exceptions?

Thanks for the offer to help testing! Could you also please share specifics of your setup? (i.e. how to repeat this situation)

AFAIS, your diagnosis is correct: these two options, allowPrivilegeEscalation and capabilities, require SecurityContext struct while we're passing PodSecurityContext. The worst part is that those two are not subsets of each other 😞 I.e. each struct has fields that are not present in the 2nd struct. So we can't seamlessly switch to SecurityContext as it might break existing setups. At the moment, I think we could make an internal wrapper that would contain fields from both structs and then construct the Pod specs accordingly: it won't be "ideal" but it could work. But this needs a bit of grokking and double-checking. If you could share how best to repeat such a case, I'd appreciate that!

[pasi-romo-idealo]

Hi, thank you for having a look at this issue :)

That is true i.e. we use kyverno to enforce certain policies. The related policies in this case would be similar like these:

https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/disallow-capabilities-strict/disallow-capabilities-strict.yaml

and

https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/disallow-privilege-escalation/disallow-privilege-escalation.yaml

To go around this issue for now I can configure the kyverno-rule directly to ignore the namespace, where the tests are running or create a separate policy exception, which cause kyverno to skip the checking for this namespace.

I understand this is a bit edge case and maybe not that easy to handle somehow elegantly in the code :)

To reproduce this issue one would basically need a test cluster with kyverno installed + with mentioned rules enabled and then try to run a test. I can also try if I can reproduce this issue e.g. in a kind-cluster and get back to you when I have a working setup, which you could also try.

[pasi-romo-idealo]

So here is a one way how to reproduce the issue.

1. create e.g. a kind-cluster

kind create cluster

2. install kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/

helm install kyverno kyverno/kyverno -n kyverno --create-namespace

3. create policies

https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/disallow-capabilities-strict/disallow-capabilities-strict.yaml

and

https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/disallow-privilege-escalation/disallow-privilege-escalation.yaml

Make sure to change the "validationFailureAction" to "enforce".

5. create a test, which violates the policy

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
  namespace: k6-test-example
spec:
  parallelism: 4
  script:
    configMap:
      name: stress-test
      file: test.js
  runner:
    nodeSelector:
        kubernetes.io/arch: amd64
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
  starter:
    nodeSelector:
        kubernetes.io/arch: amd64
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault

6. test does not start and you can see the violation from kyverno logs

k logs -n kyverno kyverno-admission-controller-<pod id> | grep 'PolicyViolation'

shows

I1013 13:09:06.970936       1 event.go:307] "Event occurred" object="disallow-capabilities-strict" fieldPath="" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" message="Job k6-test-example/k6-sample-initializer: [autogen-require-drop-all] fail (blocked); validation failure: Containers must drop `ALL` capabilities."
I1013 13:09:06.970960       1 event.go:307] "Event occurred" object="disallow-privilege-escalation" fieldPath="" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" message="Job k6-test-example/k6-sample-initializer: [autogen-privilege-escalation] fail (blocked); validation error: Privilege escalation is disallowed. The fields spec.containers[*].securityContext.allowPrivilegeEscalation, spec.initContainers[*].securityContext.allowPrivilegeEscalation, and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation must be set to `false`. rule autogen-privilege-escalation failed at path /spec/template/spec/containers/0/securityContext/"

7. try to set the missing security context in test

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
  namespace: k6-test-example
spec:
  parallelism: 4
  script:
    configMap:
      name: stress-test
      file: test.js
  runner:
    nodeSelector:
        kubernetes.io/arch: amd64
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
  starter:
    nodeSelector:
        kubernetes.io/arch: amd64
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      seccompProfile:
        type: RuntimeDefault
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL

but this fails due to

error: error validating "k6-test-example.yml": error validating data: [ValidationError(K6.spec.runner.securityContext): unknown field "allowPrivilegeEscalation" in io.k6.v1alpha1.K6.spec.runner.securityContext, ValidationError(K6.spec.runner.securityContext): unknown field "capabilities" in io.k6.v1alpha1.K6.spec.runner.securityContext, ValidationError(K6.spec.starter.securityContext): unknown field "allowPrivilegeEscalation" in io.k6.v1alpha1.K6.spec.starter.securityContext, ValidationError(K6.spec.starter.securityContext): unknown field "capabilities" in io.k6.v1alpha1.K6.spec.starter.securityContext];

Hopefully this is helpful. Please let me know if you need any more information.

[pandrez]

+1 to this issue!

We also have the exact same issue as @pasi-romo-idealo.
We have kyverno enforcing some security policies in our clusters and it requires us to disable privilegeEscalation and drop unnecessary capabilities.

Let me know if you need more information!

[gonmmarques]

Hey,

+1 as well! I lack any proficiency in helping to contribute to this, but it would be really nice to have this possibility.

[yorugac]

@pasi-romo-idealo I've just tried your instructions: I can repeat it with this 👍 Thank you greatly for the details!

And thanks for the votes here, @pandrez @gonmmarques. It'd help us prioritize this, hopefully!

[FloGro3]

Hi,
+1 also from my side. We use it across several teams and with the latest Kubernetes cluster update (and update of security policies) we lack the possibility to do performance test with K6 on our normal clusters.