SR-IOV interface added by Multus cannot reach external networks when used as primary/secondary interface

[dufanrong]

What happend:

• When using Multus CNI to add an SR-IOV interface to a Pod (alongside Calico as the default CNI), SR-IOV interfaces between Pods can ping each other, but the host cannot reach the SR-IOV IP of the Pod.
• If the SR-IOV interface is used as the primary interface (replacing Calico), the Pod loses all external connectivity.
• Example: An Nginx Pod with SR-IOV cannot receive external requests via its SR-IOV IP.

What you expected to happen:

• The SR-IOV interface should allow Pod-to-external-network communication when configured properly, regardless of being primary or secondary.

How to reproduce it (as minimally and precisely as possible):

1. Setup a K8s cluster with Calico as the default CNI.
2. Install Multus CNI, SR-IOV Device Plugin, and SR-IOV CNI.
3. Define a NetworkAttachmentDefinition for SR-IOV.
4. Deploy two Pods with the SR-IOV interface attached via Multus.
5. Observe connectivity issues:
   -- Host cannot ping the Pod's SR-IOV IP.
   -- Pods with SR-IOV as primary interface have no internet.

Anything else we need to know?:

• When using SR-IOV as a secondary interface, the Pod ends up with two default routes (one from Calico and one from SR-IOV NIC).
• Example Pod routing table (ip route output):

default via 10.244.104.1 dev eth0 
default via 10.56.217.1 dev net1 metric 100 
10.56.217.0/24 dev net1 proto kernel scope link src 10.56.217.170
10.244.104.1 dev eth0 scope link

Environment:

• Multus version image path and image ID (from 'docker images')
  ghcr.io/k8snetworkplumbingwg/multus-cni snapshot-thick a5db355310df
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:03:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

• Primary CNI for Kubernetes cluster:calico v3.20.6
• OS (e.g. from /etc/os-release):

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• File of '/etc/cni/net.d/'

cat 00-multus.conf 
{"capabilities":{"bandwidth":true,"portMappings":true},"cniVersion":"0.3.1","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","clusterNetwork":"/host/etc/cni/net.d/10-calico.conflist","type":"multus-shim"}

cat 10-calico.conflist 
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "log_file_path": "/var/log/calico/cni/cni.log",
      "datastore_type": "kubernetes",
      "nodename": "node2",
      "mtu": 0,
      "ipam": {
          "type": "calico-ipam"
      },
      "policy": {
          "type": "k8s"
      },
      "kubernetes": {
          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {"portMappings": true}
    },
    {
      "type": "bandwidth",
      "capabilities": {"bandwidth": true}
    }
  ]
cat calico-kubeconfig 
# Kubeconfig file for Calico CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[10.96.0.1]:443
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJMU1ETXdNekE0TVRNeU0xb1hEVE0xTURNd01UQTRNVE15TTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTDJHClVTT2VGZXoyK1ZtcTg0c3B4VTQ5MjFFRXZFKy9xM2krV0RjTnV0enNHRXdoaGJvaDJCbUdQREV0a3d1cDRKZkwKQWVzM2hvZHd4V1pYVlNWckdhdWdmeHowaHhBNEg1ekRWQTE4NEcrZVhKVDlRTVFaUGFWaEVZOWEyUU9maUhSUAo4MHJtK21qVk1xSm1FRWxDdGc3U0V4RzV0eExXMEloNDZqUkRPNldJWWxVaG1vRDA2a29YVFVROTZ1RGY2WnVUCnFzdDlObERrYkR3QWU0T1JIQnVacmVocE1UUjVYdno5NGRNS0RCbFo2dTliT1JXVXBmN1BXV3lqVE5MK1VBbzUKNUNOSHVSWTV3NzZvWm9LZmprNWZlcGdYcGdrUU9JRjAxYWhRSnJBbk45TTB6cFo0MlAxZVU4UHdJV3kySFFaegpNUE5IQUx4OTVVaDRia0JHSHBVQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZBY2hsZFFSSWxxdG1HTWloNVphczZuYWxyb2NNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFDdzF6MDdTZlQxWTFJRkFocUJ4TjRwQjN6RmVRQTFYVmtzTTQ1ajNwa0l3L1lpK2RGSgpjZjgzT01WNXoyTjc2QUlnMHhTOXQ2ZGJoVHBuYXoxdGU1QnJvNzhtbFhmUkpIbENZMXNCSDlDVkRHZzRGcnQvClFnTDFTYXRKellVOUVaTVlyU1BXS0Q3Q0E1QjFSekNrVmZFVU5lSmsrTWNQSEZNOXFkVmNQRGl2REZITFBiR2YKaE9Qb3gyMmVFU0xKc3FaTi80eW1EbmNralJPazVmUktwMDV6QjdOcy9tSm1kOStxZnpnM2lGSE02ZjJTWkd4UApYM3lsK1R1eFRFTzZzSkFhdFVTOVJqbVROK0pwZGwvRHcwWXFPcEMyZHo1clR5U1d2Kzkwb09NYWU1ZDRFZHRiClNxQmx5aEsvZFJibGlNbThGWlRJZ1J4eFdMZk1jL3B6QzZVawotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
users:
- name: calico
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjcyYkhaT2g3NmpKSDJrNGxRb2hpWjYwQm0weHVKejBpMEZVcGFOUEZ1dU0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tbm9kZS10b2tlbi1mOTdyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tbm9kZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImJiMmZiOTE2LTcxYTEtNGM1Yy1iYTFlLTNhNTFkNDZlYWU4MCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tbm9kZSJ9.grc9ci4bfjezFnS6EL0nQUQZiFjTCkcRL4sNRTa3ZCnXM6G0ucJVJhdRkGTpnvGTq2CJEoKogZF0_vkIKK86yCnUaUw19-5FVNIfx9bD5G9MF8cDkZbCzgaBKfdZLUfaoDsOy-c4ASA_553aTat8GO4UPns2OWTwSs40UCXmuXVDhDdLugaxh0k2Xa1Q4bVHVQIpZSfxXq_Pa_hdaPyM0QchT0iiF2ahTMRlwTYyfLRiTduI22wgf9A5ubeS5ugGvLIwnSHH_hwMsQLNvFh23vLTSn9ymAiry6d_J75MMOi7hp_0kUBwV6GwajR2UcBuL159AcSfi2YmNdUuP8BXDg
contexts:
- name: calico-context
  context:
    cluster: local
    user: calico

• File of '/etc/cni/multus/net.d'
• NetworkAttachment info (use kubectl get net-attach-def -o yaml)

apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    annotations:
      k8s.v1.cni.cncf.io/resourceName: mellanox.com/mlnx_sriov
    creationTimestamp: "2025-03-05T12:22:55Z"
    generation: 1
    managedFields:
    - apiVersion: k8s.cni.cncf.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:k8s.v1.cni.cncf.io/resourceName: {}
        f:spec:
          .: {}
          f:config: {}
      manager: kubectl-create
      operation: Update
      time: "2025-03-05T12:22:55Z"
    name: sriov-net1
    namespace: default
    resourceVersion: "274822"
    uid: 51452c69-8d12-4bf2-b540-d050e991c65c
  spec:
    config: '{ "type": "sriov", "cniVersion": "0.3.1", "name": "sriov-network", "ipam":
      { "type": "host-local", "subnet": "10.56.217.0/24", "routes": [{ "dst": "0.0.0.0/0"
      }], "gateway": "10.56.217.1" } }'
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
• Other log outputs (if you use multus logging)