[iptables] Possibility to exclude port from iptables at control-plane level

[lukidzi]

Description

It would be nice if you could exclude some ports from routing at the global level. Lets say your whole cluster needs to report metrics to service at port 8126 but it cannot go through sidecar. You could globally exclude this for each init container.

[lahabana]

@lukidzi there's already --exclude-ports that is getting improved with: #7588

Is this something that doesn't work for you?

[lukidzi]

Isn't it just for one sidecar by annotation? I thought about - what if I want to disable one port for all pods in the cluster/zone

[jijiechen]

Let me take this one

[jijiechen]

Isn't it just for one sidecar by annotation? I thought about - what if I want to disable one port for all pods in the cluster/zone

So the initial idea is to enlarge the scope that a port-excluding rule can apply to. Is it better to apply this scope at the Mesh level instead of the global control plane level?

One other question would be, should we also support excluding ports for inbound traffic?

The ports are taken into account by the kuma-init container which is auto injected. So the implementation will introduce a change to the injector:

1. Add fields into API protocol of 'Mesh.Networking' object:
   1. Outbound.ExcludedPorts repeated uint32
   2. Inbound.ExcludedPorts repeated uint32
2. Change behaviour of the injector:
   1. Fetch the Mesh object that the Pod/Namespace belongs to
   2. Read Mesh level settings and merge it with pod level settings (by annotations)
   3. Apply the result into init container generating

Any ideas?
@lahabana @lukidzi

[lahabana]

There's multiple issues in adding fields to the dataplane object:

• One assumption is that this works on both k8s and universal, we currently don't have a way to configure install tp from the cp.
• Dataplane object is supposed to be dynamically updated

I feel like either we should use a k8s only primitive or let users do this themselves with a webhook.

One option could be to extend further ContainerPatch ? I know that there was another use case for adding volumes to pods systematically too.

[jijiechen]

And by the way, it seems we currently do support a global configuration for Kubernetes based intallation:

• yaml configuration: runtime.kubernetes.injector.sidecarContainer.waitForDataplaneReady
• environment variable: kuma_runtime_kubernetes_sidecar_traffic_exclude_inbound_ports

See code in injector.go

Does this match the requirement? @lukidzi

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[lahabana]

@lukidzi can you answer @jijiechen question?

[lukidzi]

@jijiechen You are right it seems to be supported with kuma_runtime_kubernetes_sidecar_traffic_exclude_inbound_ports and kuma_runtime_kubernetes_sidecar_traffic_exclude_outbound_ports