[FEATURE] Support advanced scheduling of VMs with affinity rules(Harvester & RKE2 only)

[janeczku]

Is your feature request related to a problem? Please describe.

Advanced scheduling is necessary so that users may:

• configure workloads for improved availability, e.g. by running instances in in multiple zone
• ensure that workloads are scheduled to specific hardware flavors

Describe the solution you'd like

Users can define advanced scheduling rules for individual VMs and guest cluster nodes.

Expected behaviour

vmAntiAffinity, vmAffinity, nodeAffinity and nodeAntiAffinity rules can be specified when:

• Provisioning individual VMs
• Configuring machine pools for Harvester guest clusters.

References

• https://kubevirt.io/user-guide/operations/node_assignment/#affinity-and-anti-affinity
• https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity

[futuretea]

Currently, we supported nodeAffinity and nodeSelector for individual VMs
and supported nodeAffinity for guest cluster nodes

For harvester backend and node driver backend, vmAntiAffinity, vmAffinity, nodeAffinity and nodeAntiAffinity are all supported, need UI support, cc @guangbochen @DaiYuzeng

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

* [ ] If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: Harvester: Add pod affinity to VM config page dashboard#385 (comment)

* [ ] Is there a workaround for the issue? If so, where is it documented?
The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?~
  The PR is at: Do not delete pod affinity in patchAffinity #4026

* [ ] Does the PR include the explanation for the fix or the feature?

* [ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
The PR for the YAML change is at:
The PR for the chart change is at:

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:
  Harvester: Add pod affinity to VM config page dashboard#385
  HARVESTER: vm scheduling polish dashboard#780
  HARVESTER: Implement pod affinity in harvester manager rancher/dashboard#8331
  HARVESTER: terraform compatible vm scheduling rancher/dashboard#9061
  HARVESTER: fix podAffinity can not update match expressions dashboard#820
  HARVESTER: fix podAffinity can not update match expressions rancher/dashboard#9238
  HARVESTER: Implement pod affinity in RKE1 node driver rancher/ui#4979
  [RKE1]HARVESTER: polish vm scheduling for harvester driver rancher/ui#5036

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:
  Add VM Scheduling docs#385

* [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
- The automation skeleton PR is at:
- The automation test case PR is at:

* [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#511

[guangbochen]

Due to timeline constraints, we have tentatively moved the vmAntiAffinity UI future to v1.1.2, thanks.

[TachunLin]

@DaiYuzeng , @futuretea
After discussion, we plan to discuss the test plan and validate feature after rc1 to ensure the upstream pr merged.
Move the issue to Review stage.

[futuretea]

Test Plan Reference

Env Setup

1. Prepare a two nodes Harvester cluster. e.g. node1 and node2 (master version)
2. Add label topology.kubernetes.io/zone=zone1 to node1 and label topology.kubernetes.io/zone=zone2 to node2
3. Create VM test, specify node affinity: topology.kubernetes.io/zone in [zone1]
   
4. The VM test is successfully scheduled and is scheduled to the node of zone1
5. Import the Harvester cluster to Rancher v2.7-head

Cases

Case 1

1. create a vm with podAffinity

{
  "podAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "harvesterhci.io/vmName",
              "operator": "In",
              "values": [
                "test"
              ]
            }
          ]
        },
        "topologyKey": "topology.kubernetes.io/zone"
      }
    ]
  }
}

2. The VM is successfully scheduled to the same zone as VM test

Case 2

1. create a vm with podAntiAffinity

{
  "podAntiAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "harvesterhci.io/vmName",
              "operator": "In",
              "values": [
                "test"
              ]
            }
          ]
        },
        "topologyKey": "topology.kubernetes.io/zone"
      }
    ]
  }
}

2. The VM is successfully scheduled and will not be scheduled to the same zone as VM test

Case 3

1. create a vm with conflict podAffinity and nodeAffinity

{
  "podAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "harvesterhci.io/vmName",
              "operator": "In",
              "values": [
                "test"
              ]
            }
          ]
        },
        "topologyKey": "topology.kubernetes.io/zone"
      }
    ]
  },
  "nodeAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": {
      "nodeSelectorTerms": [
        {
          "matchExpressions": [
            {
              "key": "topology.kubernetes.io/zone",
              "operator": "In",
              "values": [
                "zone2"
              ]
            }
          ]
        }
      ]
    }
  }
}

2. The VM scheduling failed

Case 4

1. create a vm with conflict podAntiAffinity and nodeAffinity

{
  "podAntiAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "harvesterhci.io/vmName",
              "operator": "In",
              "values": [
                "test"
              ]
            }
          ]
        },
        "topologyKey": "topology.kubernetes.io/zone"
      }
    ]
  },
  "nodeAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": {
      "nodeSelectorTerms": [
        {
          "matchExpressions": [
            {
              "key": "topology.kubernetes.io/zone",
              "operator": "In",
              "values": [
                "zone1"
              ]
            }
          ]
        }
      ]
    }
  }
}

2. The VM scheduling failed

Case 5

1. create a vm with a no rule pod affinity

{
  "podAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "topologyKey": "topology.kubernetes.io/zone"
      }
    ]
  }
}

2. The VM scheduling failed

Terraform Template

terraform {
  required_providers {
    rancher2 = {
      source  = "rancher/rancher2"
      version = "3.0.1"
    }
  }
}


provider "rancher2" {
  api_url    = "<>"
  access_key = "<>"
  secret_key = "<>"
  insecure   = true
}


data "rancher2_cluster_v2" "foo-harvester" {
  name = "<>"
}

# Create a new Cloud Credential for an imported Harvester cluster
resource "rancher2_cloud_credential" "foo-harvester" {
  name = "foo-harvester"
  harvester_credential_config {
    cluster_id         = data.rancher2_cluster_v2.foo-harvester.cluster_v1_id
    cluster_type       = "imported"
    kubeconfig_content = data.rancher2_cluster_v2.foo-harvester.kube_config
  }
}

data "template_file" "affinity" {
  template = <<EOF
  {
    "podAffinity": {
      "requiredDuringSchedulingIgnoredDuringExecution": [
        {
          "labelSelector": {
            "matchExpressions": [
              {
                "key": "harvesterhci.io/vmName",
                "operator": "In",
                "values": [
                  "test"
                ]
              }
            ]
          },
          "topologyKey": "topology.kubernetes.io/zone"
        }
      ]
    }
  }
  EOF
}

data "template_file" "userdata" {
  template = <<EOF
#cloud-config
package_update: true
packages:
  - qemu-guest-agent
  - iptables
runcmd:
  - - systemctl
    - enable
    - '--now'
    - qemu-guest-agent.service
EOF
}

# Create a new rancher2 machine config v2 using harvester node_driver
resource "rancher2_machine_config_v2" "foo-harvester-v2" {
  generate_name = "foo-harvester-v2"
  harvester_config {
    vm_namespace = "default"
    cpu_count    = "2"
    memory_size  = "4"
    vm_affinity  = base64encode(data.template_file.affinity.rendered)
    #vm_affinity  = data.template_file.affinity.rendered
    disk_info    = <<EOF
    {
        "disks": [{
            "imageName": "<>",
            "size": 40,
            "bootOrder": 1
        }]
    }
    EOF
    network_info = <<EOF
    {
        "interfaces": [{
            "networkName": "<>"
        }]
    }
    EOF
    ssh_user     = "ubuntu"
    user_data    = base64encode(data.template_file.userdata.rendered)
  }
}

resource "rancher2_cluster_v2" "foo-harvester-v2" {
  name               = "foo-harvester-v2"
  kubernetes_version = "v1.26.5+rke2r1"
  rke_config {
    machine_pools {
      name                         = "pool1"
      cloud_credential_secret_name = rancher2_cloud_credential.foo-harvester.id
      control_plane_role           = true
      etcd_role                    = true
      worker_role                  = true
      quantity                     = 1
      machine_config {
        kind = rancher2_machine_config_v2.foo-harvester-v2.kind
        name = rancher2_machine_config_v2.foo-harvester-v2.name
      }
    }
    machine_selector_config {
      config = {
        cloud-provider-name = ""
      }
    }
    machine_global_config = <<EOF
cni: "calico"
disable-kube-proxy: false
etcd-expose-metrics: false
EOF
    upgrade_strategy {
      control_plane_concurrency = "10%"
      worker_concurrency        = "10%"
    }
    etcd {
      snapshot_schedule_cron = "0 */5 * * *"
      snapshot_retention     = 5
    }
    chart_values = ""
  }
}

[TachunLin]

Test Plan matrix

	podAffinity	podAntiAffinity	conflict podAffinity <-> nodeAffinity	conflict podAntiAffinity <-> nodeAffinity	Negative - Empty podAffinity
Multi-cluster RKE2 (Ready)	PASS	PASS	PASS	PASS	PASS
Terraform rancher 2.0 - RKE2 (Ready)	PASS	PASS	PASS	PASS	PASS
Single-cluster Harvester (Ready)	PASS	PASS	PASS	PASS	PASS
Multi-cluster RKE1
Terraform rancher 2.0 - RKE1

Expected Result

1. PodAffinity:

   • The VM is successfully scheduled to the same zone as VM test
     

2. podAntiAffinity:

   • The VM is successfully scheduled and will not be scheduled to the same zone as VM test
     

3. conflict podAffinity <-> nodeAffinity

   • The VM scheduling failed with message 3 Preemption is not helpful for scheduling.
     

4. conflict podAntiAffinity <-> nodeAffinity

   • The VM scheduling failed with message 3 Preemption is not helpful for scheduling.
     

5. Negative Empty podAffinity

   • The VM scheduling failed
     

Additional terraform-provider-rancher2 test plan (Check abnormal)

1. If users didn't edify anything, clusters should not be recreated.
   • no edit
   • add then remove new attribute
   • remove then add new attribute
2. terraform-provider-rancher2 can pass the json format to guest cluster without pending in the loading

#vm_affinity  = data.template_file.affinity.rendered

Test Information

• Test Environment: 3 nodes Harvester bare machines
• Harvester version: latest v1.2-head , v1.2.0-rc2
• Rancher version: v2.7-head
• terraform-provider-rancher2: v3.0.1

[TachunLin]

Tested on the following test plan path:

1. Multi-cluster vm provisioning on RKE2 instance from Rancher UI
2. Multi-cluster vm provisioning on RKE2 instance from terraform-rancher2.0
3. Single-cluster vm provisioning on Harvester

Move to ready for testing and plan to continue after the RKE1 content is ready.

[DaiYuzeng]

Hi! @TachunLin, RKE1 related changes won't be merged at this moment. I filed an issue to track #4121. Can we close this issue if there is no question and track RKE1 changes at the new issue?

[TachunLin]

Thanks @DaiYuzeng for the comment, we would track the support status of RKE1 podaffinity feature in issue #4121.
The Multi-cluster RKE1 and Terraform rancher 2.0 - RKE1 test scope would not be implement in current issue.

[TachunLin]

Verified failed on v1.2.0-rc2. Suggest moving to implement for further check.

While validating the advanced test plan, we found an potential issue.
When we edit a running VM already have podaffinity rule set, change the Operator from in list to not in list.
Save then restart the VM, we found the operator did not change accordingly on UI nor on the yaml content.

vokoscreenNG-2023-06-26_19-03-51.mp4

[DaiYuzeng]

Also occurred in Rancher Pod configurations. I am going to work on it. Thanks for your validation @TachunLin !

[TachunLin]

• Verified the in list and not in list operator for vm affinity work as expected
  

  podAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: harvesterhci.io/vmName
                      operator: NotIn
                      values:
                        - vm1
                topologyKey: topology.kubernetes.io/zone
  

• Verified the is set and not in set operator for vm affinity and anti-affinity work as expected

• Verified for node affinity the Required rule have higher priority to be applied than Preferred

• Verified we can't have two Required affinity rule set on the same VM

• Verified we can have two Required affinity rule set on the same VM, it will apply in sequence

[TachunLin]

Verified failed on v1.2-f4968e49-head (6/30) since found a potential issue
Move the issue to implement

Result

When we have a running vm contain two vm affinity set on it.
Then we edit the Weight field with backspace key, the weight field will be deleted and no longer display again.

vokoscreenNG-2023-07-03_18-47-52.mp4

• And we can save this setting and found the weight have no longer exists
  

Suggest we can keep the weight value in empty while clicking the backspace key to the end.
To provide a consistent user experience on the vm affinity page.

Test Information

• Test Environment: 3 nodes harvester on provo bare machine
• Harvester version: v1.2-f4968e49-head (6/30)

[TachunLin]

Verified fixed on v1.2-bd26e791-head (07/06) with Rancher v2.7-head (3319bfb). Close this issue.

Result

• We can use backspace key to edit the weight field in vm affinity page for both single and multiple cluster scenarios
  

• Also tested the priority, operator and weight works on multiple cluster environment just as the single cluster test scenarios in:
  [FEATURE] Support advanced scheduling of VMs with affinity rules(Harvester & RKE2 only) #2317 (comment)

Test Information

• Test Environment: 3 nodes harvester clusters on bare machine
• Harvester version: v1.2-bd26e791-head (07/06)
• Rancher: v2.7-head (3319bfb)

Verify Steps

#2317 (comment)
#2317 (comment)
#2317 (comment)