Support constraints on PodExecOptions

[aryann]

Describe the solution you'd like

I would like to be able to specify a constraint on a subresource. In particular, I'd like to prevent kubectl exec requests where either the tty or stdin fields of PodExecOptions is true.

I do not believe this is possible with Gatekeeper today.

Anything else you would like to add:

Some context on why this is useful: Kubernetes Audit Logging only logs the initial command used to initiate pods.exec (the command appears as cmd query parameters on the subresource URI). This means that any interactive sessions (e.g., through /bin/bash with kubectl exec -it) only log the initial command but no subsequent commands since the Kube API is no longer involved in the session. This makes completely auditing all interactions with the cluster challenging.

One alternative is to use auditd, but that does not work since you no longer have the identity of the actor.

Environment:

• Gatekeeper version: N/A
• Kubernetes version: (use kubectl version): N/A

[maxsmythe]

Digging into this a bit, specifically WRT looking at validating kubectl exec requests:

• We will need to listen to the CONNECT request type in order to get admission calls for the kubectl exec resource

• It looks like ConnectRequest is the resource that will be validated:

https://github.com/kubernetes/kubernetes/blob/3d43944bee17e7147c2ae59aa97a63fb5111d6b0/pkg/api/rest/rest.go#L250-L260

ConnectRequest's contents appear to be fairly opaque (lots of strings). I haven't dug deeply enough to see how these strings are formed.

[maxsmythe]

Looks like the ConnectRequest bit was a red herring.

Manually configuring Gatekeeper to reject CONNECT requests out-of-the-box and then using a "deny all" constraint to dump the request shows that we are given the kind PodExecOptions to match against:

{
  "_unstable": {
    "namespace": {
      "apiVersion": "v1",
      "kind": "Namespace",
      "metadata": {
        "annotations": {
          "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"owner\":\"me.agilebank.demo\"},\"name\":\"production\"}}\n"
        },
        "creationTimestamp": "2021-01-13T01:32:16Z",
        "labels": {
          "owner": "me.agilebank.demo"
        },
        "managedFields": [
          {
            "apiVersion": "v1",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:kubectl.kubernetes.io/last-applied-configuration": {}
                },
                "f:labels": {
                  ".": {},
                  "f:owner": {}
                }
              },
              "f:status": {
                "f:phase": {}
              }
            },
            "manager": "kubectl-client-side-apply",
            "operation": "Update",
            "time": "2021-01-13T01:32:16Z"
          }
        ],
        "name": "production",
        "resourceVersion": "834",
        "selfLink": "/api/v1/namespaces/production",
        "uid": "6149358d-fc9e-44ce-9302-e2bc63ec2e59"
      },
      "spec": {
        "finalizers": [
          "kubernetes"
        ]
      },
      "status": {
        "phase": "Active"
      }
    }
  },
  "dryRun": false,
  "kind": {
    "group": "",
    "kind": "PodExecOptions",
    "version": "v1"
  },
  "name": "opa",
  "namespace": "production",
  "object": {
    "apiVersion": "v1",
    "command": [
      "ls"
    ],
    "container": "opa",
    "kind": "PodExecOptions",
    "stderr": true,
    "stdout": true
  },
  "oldObject": null,
  "operation": "CONNECT",
  "options": null,
  "requestKind": {
    "group": "",
    "kind": "PodExecOptions",
    "version": "v1"
  },
  "requestResource": {
    "group": "",
    "resource": "pods",
    "version": "v1"
  },
  "requestSubResource": "exec",
  "resource": {
    "group": "",
    "resource": "pods",
    "version": "v1"
  },
  "subResource": "exec",
  "uid": "f5d8c9d3-eb47-42ab-ab1c-8f47b4c215ff",
  "userInfo": {
    "groups": [
      "system:masters",
      "system:authenticated"
    ],
    "username": "kubernetes-admin"
  }
}

[maxsmythe]

Note that I'm missing a step here:

for the ValidatingWebhookConfiguration adjustment, we want to validate a CONNECT operation against the pods/exec and pods/attach subresources.

[soorena776]

I don't seem to be able to repro getting gatekeeper to detect kubectl exec nginx -it -- /bin/sh requests. Here is what I did:

• create a kind cluster v1.19.1
• create a pod k run nginx --image=k8s.gcr.io/nginx:test-cmd -- /bin/sleep infinity
• install gatekeeper kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.3/deploy/gatekeeper.yaml
• modify gk's validating webhook to allow all operations (this should include the CONNECT operation mentioned above)

  apiVersion: admissionregistration.k8s.io/v1
  kind: ValidatingWebhookConfiguration
  metadata:
    labels:
      gatekeeper.sh/system: "yes"
    name: gatekeeper-validating-webhook-configuration
  webhooks:
  - admissionReviewVersions:
    - v1beta1
    failurePolicy: Ignore
    matchPolicy: Exact
    name: validation.gatekeeper.sh
    namespaceSelector:
      matchExpressions:
      - key: admission.gatekeeper.sh/ignore
        operator: DoesNotExist
    objectSelector: {}
    rules:
    - apiGroups:
      - '*'
      apiVersions:
      - '*'
      operations:
      - '*'
      resources:
      - '*'
      scope: '*'
    sideEffects: None
    timeoutSeconds: 3
  - admissionReviewVersions:
    - v1beta1
    failurePolicy: Fail
    matchPolicy: Exact
    name: check-ignore-label.gatekeeper.sh
    namespaceSelector: {}
    objectSelector: {}
    rules:
    - apiGroups:
      - ""
      apiVersions:
      - '*'
      operations:
      - '*'
      resources:
      - '*'
      scope: '*'
    sideEffects: None
    timeoutSeconds: 3
  

  • add denyAll template and constraint

  apiVersion: templates.gatekeeper.sh/v1beta1
  kind: ConstraintTemplate
  metadata:
    name: k8sdenyall
  spec:
    crd:
      spec:
        names:
          kind: K8sDenyAll
    targets:
      - target: admission.k8s.gatekeeper.sh
        rego: |
          package k8sdenyall
  
          violation[{"msg": msg}] {
            msg := sprintf("REVIEW OBJECT: %v", [input.review])
          }
  ---
  apiVersion: constraints.gatekeeper.sh/v1beta1
  kind: K8sDenyAll
  metadata:
    name: deny-all
  

• observe that it's possible to exec into the pod

  > kubectl exec nginx -it -- /bin/sh
  # hostname
  nginx
  #
  

[maxsmythe]

You need to explicitly match on subresources.

I think this is resources: '*/*' if you want to match on everything or

*/exec and */attach if you want to match those subresources for all resources.

[soorena776]

Thank you!
I updated the resource to '*/*' but still not seeing kubectl exec ... mentioned above being blocked.
For analogy, kubectl scale deployment nginx-deployment --replicas 1 gets blocked with the following dump:

{
    "_unstable": {
        "namespace": {
            "apiVersion": "v1",
            "kind": "Namespace",
            "metadata": {
                "creationTimestamp": "2021-03-12T16:01:33Z",
                "managedFields": [
                    {
                        "apiVersion": "v1",
                        "fieldsType": "FieldsV1",
                        "fieldsV1": {
                            "f:status": {
                                "f:phase": {}
                            }
                        },
                        "manager": "kube-apiserver",
                        "operation": "Update",
                        "time": "2021-03-12T16:01:33Z"
                    }
                ],
                "name": "default",
                "resourceVersion": "159",
                "selfLink": "/api/v1/namespaces/default",
                "uid": "cebe399b-6674-4bf3-b63b-69abf6dca4b3"
            },
            "spec": {
                "finalizers": [
                    "kubernetes"
                ]
            },
            "status": {
                "phase": "Active"
            }
        }
    },
    "dryRun": false,
    "kind": {
        "group": "autoscaling",
        "kind": "Scale",
        "version": "v1"
    },
    "name": "nginx-deployment",
    "namespace": "default",
    "object": {
        "apiVersion": "autoscaling/v1",
        "kind": "Scale",
        "metadata": {
            "creationTimestamp": "2021-03-12T18:18:37Z",
            "name": "nginx-deployment",
            "namespace": "default",
            "resourceVersion": "2883",
            "uid": "d709ce3c-8b1d-46b7-87f8-962e045bffe3"
        },
        "spec": {
            "replicas": 1
        },
        "status": {
            "replicas": 1,
            "selector": "app=nginx"
        }
    },
    "oldObject": {
        "apiVersion": "autoscaling/v1",
        "kind": "Scale",
        "metadata": {
            "creationTimestamp": "2021-03-12T18:18:37Z",
            "name": "nginx-deployment",
            "namespace": "default",
            "resourceVersion": "2883",
            "uid": "d709ce3c-8b1d-46b7-87f8-962e045bffe3"
        },
        "spec": {
            "replicas": 1
        },
        "status": {
            "replicas": 1,
            "selector": "app=nginx"
        }
    },
    "operation": "UPDATE",
    "options": {
        "apiVersion": "meta.k8s.io/v1",
        "kind": "UpdateOptions"
    },
    "requestKind": {
        "group": "autoscaling",
        "kind": "Scale",
        "version": "v1"
    },
    "requestResource": {
        "group": "apps",
        "resource": "deployments",
        "version": "v1"
    },
    "requestSubResource": "scale",
    "resource": {
        "group": "apps",
        "resource": "deployments",
        "version": "v1"
    },
    "subResource": "scale",
    "uid": "d9cefbf2-d8a6-4cf9-87e8-010169e90717",
    "userInfo": {
        "groups": [
            "system:masters",
            "system:authenticated"
        ],
        "username": "kubernetes-admin"
    }
}

[soorena776]

Thanks to @maxsmythe, I had to specifically call out pods/exec in gatekeeper validating configuration for the resources:

  rules:
  - apiGroups:
    - ""
    apiVersions:
    - '*'
    operations:
    - '*'
    resources:
    - 'pods/exec'

and that makes it work!

Created a gist here.

[HenriWilliams]

Hi,

@soorena776, your solution works for me as well, thanks for providing a gist! @maxsmythe Is this something that we could add support for in the gatekeeper helm chart? I'm happy to raise a PR for this.

[maxsmythe]

I don't see a reason why not. @sozercan @ritazh @shomron ?

[ritazh]

+1 I think this would be a great enhancement!

[ritazh]

@HenriWilliams Any luck with the PR? Let us know if you need any help!

[HenriWilliams]

You are able to do this in the current helm chart using:

validatingWebhookCustomRules:
- apiGroups:
  - '*'
  apiVersions:
  - '*'
  operations: 
  - CREATE
  - UPDATE
  - CONNECT
  resources:
  - '*'
  - 'pods/exec'
  - 'pods/attach'

which was added in this PR. I think because of this we can probably close this issue now.

[ritazh]

Closed via #1806

[gigi-at-zymergen]

@aryann I have exactly the use case you describe, and I'm a bit new to writing OPA gatekeeper constraints and constraint templates.

I'm wondering if you had examples of referencing the PodExecOptions to validate that tty and stdin are false? I'm reading up on rego and looking through all the examples and the library at https://github.com/open-policy-agent/gatekeeper-library/tree/master . If you had a working constraint template, I would encourage you to contribute it to the gatekeeper-library as I doubt I'm the only one who would find it useful and would be grateful not to have to climb that learning curve.

[gigi-at-zymergen]

For anyone that happens on this issue in the future. I got this to work in my org and have not yet contributed to the gateway-library, but can easilygive some pointers for others here:

1. Make sure your gatekeeper webhook config includes the CONNECT operation. The default install we were using included pods/exec sub-resource in resources but only had CREATE and UPDATE in the operations.

2. Example ConstraintTemplate:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdenyinteractiveexec
spec:
  crd:
    spec:
      names:
        kind: K8sDenyInteractiveExec
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdenyinteractiveexec

        violation[{"msg": msg}] {
          input.review.object.stdin == true
          msg := sprintf("Interactive exec are not permitted in production environment. REVIEW OBJECT: %v", [input.review])
        }

3. Example constraint matching on PodExecOptions

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDenyInteractiveExec
metadata:
  name: k8sdenyinteractiveexec
spec:
  match:
    kinds:
      - apiGroups: [ "" ]
        kinds: [ "PodExecOptions" ]

[nataraj24]

• Make sure your gatekeeper webhook config includes the CONNECT operation. The default install we were using included pods/exec sub-resource in resources but only had CREATE and UPDATE in the operations.

With above settings in validatingwebhookconfiguration its still allowing exec option. Information of any other setting which needs to be done would be helpful.