Transparent Proxy CNI Plug-in: escalated privileges required on Consul containers

[sankaranarayanan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

The current implementation of Transparent Proxy on Consul K8s requires the init-container we run before Envoy and application containers start to have CAP_NET_ADMIN privileges. This may not be acceptable for more security-conscious customers.

Writing our own CNI plugin for applying traffic redirection rules via iptables would enable us to delegate the actual running of the commands to kubelet instead of executing them in the init container.

---

Helm chart failed to install on aks, if azure-policy is enabled

Steps to reproduce

1. az aks enable-addons --addons azure-policy --name sampleCluster --resource-group samplegroups
2. helm install consul hashicorp/consul -f config.yaml

Consul config.yaml

global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.10.0
  metrics:
    enabled: true
    enableGatewayMetrics: true
controller:
  enabled: true
syncCatalog:
  enabled: true
  toConsul: true
  toK8S: false
  default: false
connectInject:
  enabled: true
  default: false
  envoyExtraArgs: "-l debug"
  metrics:
    defaultEnabled: true
    defaultEnableMerging: true
  transparentProxy:
    defaultEnabled: true
ui:
  enabled: true
  service:
    enabled: true
    type: LoadBalancer
ingressGateways:
  enabled: true
  defaults:
    replicas: 1
    service:
      type: LoadBalancer
      ports:
        - port: 80
meshGateway:
  enabled: true
  replicas: 1

Error Log

  Warning  FailedCreate      2s (x14 over 44s)  replicaset-controller  Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: ingress-gateway
[azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: consul-sidecar
[azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: copy-consul-bin
[azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: service-init

is there option to deploy consul with azure-policy enable in aks

[lkysow]

Hi! Is it possible to customize the policies? Our containers require certain permissions.

[sankaranarayanan]

Could you please help which kubernetes policies need to customize or can you list consul requirement of container permission to run on azure policy

we tried to modify on following policies but not able to deploy consul

policies

1. Kubernetes clusters should not allow container privilege escalation
2. Kubernetes cluster should not allow privileged containers

[andriktr]

In azure policies you can set namespace exceptions, so just put consul namespace to the exception list

Make sure you are modifying policy assignment which is applied to you cluster.
Another option to set policy effect from deny to audit.

[scoof]

Hi! Is it possible to customize the policies? Our containers require certain permissions.

That does not seem like a secure approach. It is common to disallow assuming capabilities and running privileged for containers, and you should not depend on it being available in all namespaces that wish to run transparent proxy.
I don't know if it possible to have the redirect-traffic functionality in a privileged container as part of the existing DaemonSet, and then have the sidecars signal that they want to have a transparent proxy injected?

[andriktr]

As azure policy uses OPA Gatekeeper as engine I believe it would be quite tricky to allow only consul connect inject init container
with its securityContext settings. Keeping in mind all possible azure limitations. Only recently MS announces posibilities to write custom policies. Consul connect inject init container requires privileged: true, runAsRoot: true, etc. and as it normally will present in all meshed apps, yes it will be quite insecure and difficult to control and allow these privileges explicitly only to consul connect init container across all namespaces. Currently trying to do something similar with 'kyverno' which is another k8s policy manager. Idea was to set the policy which will allow initContainers with privileged: true and other settings only if this init container image is from my-private-registry/my-image:tag . Kyverno is bit more flexible and do not requires to know REGO language which is used by OPA.

[scoof]

As azure policy uses OPA Gatekeeper as engine I believe it would be quite tricky to allow only consul connect inject init container
with its securityContext settings. Keeping in mind all possible azure limitations. Only recently MS announces posibilities to write custom policies. Consul connect inject init container requires privileged: true, runAsRoot: true, etc. and as it normally will present in all meshed apps, yes it will be quite insecure and difficult to control and allow these privileges explicitly only to consul connect init container across all namespaces. Currently trying to do something similar with 'kyverno' which is another k8s policy manager. Idea was to set the policy which will allow initContainers with privileged: true and other settings only if this init container image is from my-private-registry/my-image:tag . Kyverno is bit more flexible and do not requires to know REGO language which is used by OPA.

That would only work if there's no way for others to modify the command line arguments

[andriktr]

Yep, it is so. It also posible to filter out by service account which sends request to the api servers. In case if consul is the last sa which sends request to the api after injection then it possible to write a policy with condition like if request comes from consul service account which is sits in consul namespace then initcontainer with privileges is allowed. Let's say I'm creating a deployment with enabled consul who will be the final api requester me or consul service account which injects the init containers and envoy side car? For me it's clear that my initial request will be first sent to consul mutating webhook, altered and sent back to the k8s api but under which account this altered request will come to the api i'm not sure logically this should be a consul 'consul-connect-injector-webhook-svc-account'. Maybe somebody from consul-k8s can comment on this more.

[scoof]

I would say that this is a more general issue than AKS with Policy enabled. It is not possible to run transparent proxy without allowing Privileged containers in general for that namespace, which is a big no-no in my book. This is a deal breaker. Quoting from the Kubernetes docs, https://kubernetes.io/docs/concepts/security/pod-security-standards/#privileged

The Privileged policy is purposely-open, and entirely unrestricted. This type of policy is typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users.

Istio seems to solve this by providing a chained CNI: https://istio.io/latest/docs/setup/additional-setup/cni/

[andriktr]

I agree with you it should be changed on consul-k8s side to do not require the high privileged containers, keeping in mind that this container will present in all meshed aps. IMO it would acceptable to have such privileged container in consul system ns itself as then access to it might easily restricted with RBAC and limited to admins only.

[david-yu]

Hi @scoof and @andriktr we are aware that those containers require escalated privileges due to our transparent proxy implementation to apply traffic redirection rules. We acknowledge that for security conscious users that may pose a problem. On the roadmap we currently have CNI Plugin for Transparent Proxy slotted to address these concerns but will be not be delivered until next year. I've updated the title and description to reflect the ask. Please upvote so we can gather feedback on how to prioritize this feature.

[david-yu]

Thanks for your patience, we're excited to see this feature released next week with #1456 now merged.