No route to ClusterIP services when using IPVLAN

[rickmoonex]

I'm running Multus next to flannel on my K3s cluster. I've installed it with the host-local ipam as per documentation:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  valuesContent: |-
    config:
      fullnameOverride: multus
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        binDir: /var/lib/rancher/k3s/data/cni/
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig

I then created the network definition and a test pod for connecting to that vlan2003 network:

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: multus-iot
  namespace: iot
spec:
  config: |-
    {
      "cniVersion": "0.3.1",
      "name": "multus-iot",
      "plugins": [
        {
          "type": "ipvlan",
          "master": "vlan2003",
          "ipam": {
            "type": "static"
          }
        }
      ]
    }

apiVersion: v1
kind: Pod
metadata:
  name: sample-pod
  namespace: iot
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [{
        "name": "multus-iot",
        "namespace": "iot",
        "mac": "c6:5e:a4:8e:7a:58",
        "ips": ["10.129.203.9/24"]
      }]
spec:
  containers:
    - name: sample-pod
      command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
      image: alpine

This deploys like a charm and I'm able to ping the pod from the 10.129.203.0/24 network.

But I am not able to resolve any DNS request. When I check my resolvconf I see it points to the ClusterIP of the kube-dns service: 10.43.0.10.

Checking the routing table I see the following:

default via 10.42.0.1 dev eth0
10.42.0.0/24 dev eth0 scope link  src 10.42.0.41
10.42.0.0/16 via 10.42.0.1 dev eth0
10.129.203.0/24 dev net1 scope link  src 10.129.203.9

There is no route to 10.43.0.0/16 present.

But when I create a pod without a multus annotation, this issue is resolved.

Thanks for any help