feat: Enable HA by default for Modelmesh Controller (kserve#272)

The pod anti-affinity will tell the Kubernetes schedule to schedule pods on different nodes if possible. This will allow users to enable HA by setting the number of replicas to a number greater than 1.

Signed-off-by: Anish Asthana <[email protected]>

config/manager/manager.yaml

@@ -21,7 +21,7 @@ spec:
   selector:
     matchLabels:
       control-plane: modelmesh-controller
-  replicas: 1
+  replicas: 1 # This can be increased safely to enable HA. A good value to set is 3.
   template:
     metadata:
       labels:
@@ -36,6 +36,17 @@ spec:
                     operator: In
                     values:
                       - amd64
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                labelSelector:
+                  matchExpressions:
+                    - key: control-plane
+                      operator: In
+                      values:
+                        - modelmesh-controller
+                topologyKey: topology.kubernetes.io/zone
       containers:
         - command:
             - /manager