feat(controller): multiple TrafficRoutingReconciler (#1472)

Signed-off-by: fblgit <[email protected]>

docs/architecture.md

@@ -34,7 +34,7 @@ Note also that the replica sets that take part in a Rollout are fully managed by
 This is the mechanism that traffic from live users enters your cluster and is redirected to the appropriate version. Argo Rollouts use the [standard Kubernetes service resource](https://kubernetes.io/docs/concepts/services-networking/service/), but with some extra metadata needed for management.
 
 Argo Rollouts is very flexible on networking options. First of all you can have different services during a Rollout, that go only to the new version, only to the old version or both.
-Specifically for Canary deployments, Argo Rollouts supports several [service mesh and ingress solutions](../features/traffic-management/) for splitting traffic with specific percentages instead of simple balancing based on pod counts.
+Specifically for Canary deployments, Argo Rollouts supports several [service mesh and ingress solutions](../features/traffic-management/) for splitting traffic with specific percentages instead of simple balancing based on pod counts and it is possible to use multiple routing providers simultaneously.
 
 ## AnalysisTemplate and AnalysisRun
 

docs/features/traffic-management/index.md

@@ -23,6 +23,7 @@ Argo Rollouts enables traffic management by manipulating the Service Mesh resour
 - [Istio](istio.md)
 - [Nginx Ingress Controller](nginx.md)
 - [Service Mesh Interface (SMI)](smi.md)
+- [Multiple Providers](mixed.md)
 - File a ticket [here](https://github.com/argoproj/argo-rollouts/issues) if you would like another implementation (or thumbs up it if that issue already exists)
 
 Regardless of the Service Mesh used, the Rollout object has to set a canary Service and a stable Service in its spec. Here is an example with those fields set:

docs/features/traffic-management/mixed.md

@@ -0,0 +1,60 @@
+# Multiple Providers
+!!! note
+
+    Multiple trafficRouting is available since Argo Rollouts v1.2
+
+The usage of multiple providers tries to cover scenarios where, for some reason, we have to use
+different providers on North-South and West-East traffic routing or any other hybrid architecture that
+requires the use of multiple providers.
+
+## Examples of when you can use multiple providers
+
+### Avoid injecting sidecars on your Ingress controller
+
+This is a common requirement of the service mesh and with multiple trafficRoutings you can leverage North-South traffic shifting to NGiNX 
+and West-East traffic shifting to SMI, avoiding the need of adding the Ingress controller inside the mesh.
+
+### Avoid manipulation of the host header at the Ingress
+
+Another common side effect of adding some of the Ingress controllers into the mesh, and is caused by the usage of those 
+mesh host headers to be pointing into a mesh hostname in order to be routed.
+
+### Avoid Big-Bang
+
+This takes place on existing fleets where downtime is very reduced or nearly impossible.
+To avoid [big-bang-adoption](https://en.wikipedia.org/wiki/Big_bang_adoption) the use of multiple providers can ease
+how teams can implement gradually new technologies. An example, where an existing fleet that is using a provider
+such as Ambassador and is already performing canary in a North-South fashion as part of their rollouts can gradually 
+implement more providers such as Istio, SMI, etc.
+
+### Hybrid Scenarios
+
+In this case, its very similar to avoiding the Big-Bang, either if it is part of the platform roadmap or a new redesign
+of the architecture, there are multiple scenarios where having the capacity of using multiple trafficRoutings is very 
+much in need: gradual implementation, eased rollback of architecture or even for a fallback.
+
+## Requirements
+
+The use of multiple providers requires that both providers comply with its minimum requirements independently.
+By example, if you want to use NGiNX and SMI you would need to have both SMI and NGiNX in place and produce the rollout configuration
+for both.
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: rollouts-demo
+spec:
+  strategy:
+    canary:
+      # Reference to a Service which the controller will update to point to the canary ReplicaSet
+      canaryService: rollouts-demo-canary
+      # Reference to a Service which the controller will update to point to the stable ReplicaSet
+      stableService: rollouts-demo-stable
+      trafficRouting:
+        nginx:
+          # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)
+          # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.
+          stableIngress: rollouts-demo-stable
+        smi: {}
+```

docs/getting-started.md

@@ -178,5 +178,6 @@ provider to achieve more advanced traffic shaping.
 * [ALB Guide](getting-started/alb/index.md)
 * [Ambassador Guide](getting-started/ambassador/index.md)
 * [Istio Guide](getting-started/istio/index.md)
+* [Multiple Providers Guide](getting-started/mixed/index.md)
 * [NGINX Guide](getting-started/nginx/index.md)
 * [SMI Guide](getting-started/smi/index.md)

docs/getting-started/mixed/index.md

@@ -0,0 +1,233 @@
+# Getting Started - Multiple Providers (Service Mesh Interface and NGiNX Ingress)
+
+!!! important
+    Available since v1.2
+
+This guide covers how Argo Rollouts integrates with multiple TrafficRoutings, using
+[Linkerd](https://linkerd.io) and 
+[NGINX Ingress Controller](https://github.com/kubernetes/ingress-nginx) for traffic shaping, but you
+should be able to produce any other combination between the existing trafficRouting options.
+
+This guide builds upon the concepts of the [basic getting started guide](../../getting-started.md), 
+[NGINX Guide](getting-started/nginx/index.md), and [SMI Guide](getting-started/smi/index.md).
+
+## Requirements
+- Kubernetes cluster with Linkerd installed
+- Kubernetes cluster with NGINX ingress controller installed and part of the mesh
+
+!!! tip
+    See the [environment setup guide for linkerd](../setup/index.md#linkerd-setup)
+    on how to setup a local minikube environment with linkerd and nginx.
+
+## 1. Deploy the Rollout, Services, and Ingress
+
+When SMI is used as one the traffic routers, the Rollout canary strategy must define
+the following mandatory fields:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: rollouts-demo
+spec:
+  strategy:
+    canary:
+      # Reference to a Service which the controller will update to point to the canary ReplicaSet
+      canaryService: rollouts-demo-canary
+      # Reference to a Service which the controller will update to point to the stable ReplicaSet
+      stableService: rollouts-demo-stable
+      trafficRouting:
+        smi: {}
+```
+When NGINX Ingress is used as the traffic router, the Rollout canary strategy must define
+the following mandatory fields:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: rollouts-demo
+spec:
+  strategy:
+    canary:
+      # Reference to a Service which the controller will update to point to the canary ReplicaSet
+      canaryService: rollouts-demo-canary
+      # Reference to a Service which the controller will update to point to the stable ReplicaSet
+      stableService: rollouts-demo-stable
+      trafficRouting:
+        nginx:
+          # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)
+          # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.
+          stableIngress: rollouts-demo-stable
+...
+```
+
+A combination of both should have comply with each TrafficRouting requirements, in this case:
+
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: rollouts-demo
+spec:
+  strategy:
+    canary:
+      # Reference to a Service which the controller will update to point to the canary ReplicaSet
+      canaryService: rollouts-demo-canary
+      # Reference to a Service which the controller will update to point to the stable ReplicaSet
+      stableService: rollouts-demo-stable
+      trafficRouting:
+        nginx:
+          # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)
+          # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.
+          stableIngress: rollouts-demo-stable
+        smi: {}
+```
+
+The Ingress referenced in `canary.trafficRouting.nginx.stableIngress` is required to have a host
+rule which has a backend targeting the Service referenced under `canary.stableService`.
+In our example, that stable Service is named: `rollouts-demo-stable`:
+
+```yaml
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: rollouts-demo-stable
+  annotations:
+    kubernetes.io/ingress.class: nginx
+spec:
+  rules:
+  - host: rollouts-demo.local
+    http:
+      paths:
+      - path: /
+        backend:
+          # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field
+          serviceName: rollouts-demo-stable
+          servicePort: 80
+```
+
+Run the following commands to deploy:
+
+* A Rollout with the Linkerd `linkerd.io/inject: enabled` annotation
+* Two Services (stable and canary)
+* An Ingress
+
+```shell
+kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/rollout.yaml
+kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/services.yaml
+kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-rollouts/master/docs/getting-started/mixed/ingress.yaml
+```
+
+After applying the manifests you should see the following rollout, services, and ingress resources
+in the cluster:
+
+```shell
+$ kubectl get ro
+NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE
+rollouts-demo   1         2         1            2
+
+$ kubectl get svc
+NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
+rollouts-demo-canary   ClusterIP   10.111.69.188    <none>        80/TCP    23m
+rollouts-demo-stable   ClusterIP   10.109.175.248   <none>        80/TCP    23m
+
+$ kubectl get ing
+NAME                   CLASS    HOSTS                 ADDRESS        PORTS   AGE
+rollouts-demo-stable   <none>   rollouts-demo.local   192.168.64.2   80      23m
+```
+
+You should also see a TrafficSplit resource which is created automatically and owned by the rollout:
+
+```
+$ kubectl get trafficsplit
+NAME            SERVICE
+rollouts-demo   rollouts-demo-stable
+```
+
+When inspecting the generated TrafficSplit resource, the weights are automatically configured to
+send 100% traffic to the `rollouts-demo-stable` service, and 0% traffic to the `rollouts-demo-canary`.
+These values will be updated during an update.
+
+```yaml
+apiVersion: split.smi-spec.io/v1alpha1
+kind: TrafficSplit
+metadata:
+  name: rollouts-demo
+  namespace: default
+spec:
+  backends:
+  - service: rollouts-demo-canary
+    weight: "0"
+  - service: rollouts-demo-stable
+    weight: "100"
+  service: rollouts-demo-stable
+```
+
+You should also notice a second ingress created by the rollouts controller,
+`rollouts-demo-rollouts-demo-stable-canary`. This ingress is the "canary ingress", which is a
+clone of the user-managed Ingress referenced under `nginx.stableIngress`. It is used by nginx
+ingress controller to achieve canary traffic splitting. The name of the generated ingress is
+formulated using `<ROLLOUT-NAME>-<INGRESS-NAME>-canary`. More details on the second Ingress are
+discussed in the following section.
+
+## 2. Perform an update
+
+Now perform an update the rollout by changing the image, and wait for it to reached the paused state.
+
+```shell
+kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow
+kubectl argo rollouts get rollout rollouts-demo
+```
+
+![Rollout Paused](../nginx/paused-rollout-nginx.png)
+
+At this point, both the canary and stable version of the Rollout are running, with 5% of the
+traffic directed to the canary and 95% to the stable. When inspecting the TrafficSplit generated by 
+the controller, we see that the weight has been updated to reflect the current `setWeight: 5` step of 
+the canary deploy.
+
+```yaml
+apiVersion: split.smi-spec.io/v1alpha1
+kind: TrafficSplit
+metadata:
+  name: rollouts-demo
+  namespace: default
+spec:
+  backends:
+  - service: rollouts-demo-canary
+    weight: "5"
+  - service: rollouts-demo-stable
+    weight: "95"
+  service: rollouts-demo-stable
+```
+When inspecting the rollout controller generated Ingress copy, we see that it has the following
+changes over the original ingress:
+
+1. Two additional
+[NGINX specific canary annotations](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary)
+are added to the annotations.
+2. The Ingress rules will have an rule which points the backend to the *canary* service.
+
+
+```yaml
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: rollouts-demo-rollouts-demo-stable-canary
+  annotations:
+    kubernetes.io/ingress.class: nginx
+    nginx.ingress.kubernetes.io/canary: "true"
+    nginx.ingress.kubernetes.io/canary-weight: "5"
+spec:
+  rules:
+  - host: rollouts-demo.local
+    http:
+      paths:
+      - backend:
+          serviceName: rollouts-demo-canary
+          servicePort: 80
+```
+
+As the Rollout progresses through steps, the weights in the TrafficSplit and Ingress resource will be adjusted
+to match the current setWeight of the steps.

docs/getting-started/mixed/ingress.yaml

@@ -0,0 +1,16 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: rollouts-demo-stable
+  annotations:
+    kubernetes.io/ingress.class: nginx
+spec:
+  rules:
+  - host: rollouts-demo.local
+    http:
+      paths:
+      - path: /
+        backend:
+          # Reference to a Service name, also specified in the Rollout spec.strategy.canary.stableService field
+          serviceName: rollouts-demo-stable
+          servicePort: 80

docs/getting-started/mixed/rollout.yaml

@@ -0,0 +1,41 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Rollout
+metadata:
+  name: rollouts-demo
+spec:
+  replicas: 1
+  strategy:
+    canary:
+      canaryService: rollouts-demo-canary
+      stableService: rollouts-demo-stable
+      trafficRouting:
+        nginx:
+          # Reference to an Ingress which has a rule pointing to the stable service (e.g. rollouts-demo-stable)
+          # This ingress will be cloned with a new name, in order to achieve NGINX traffic splitting.
+          stableIngress: rollouts-demo-stable
+        smi: {}
+      steps:
+      - setWeight: 5
+      - pause: {}
+  revisionHistoryLimit: 2
+  selector:
+    matchLabels:
+      app: rollouts-demo
+  template:
+    metadata:
+      annotations:
+        linkerd.io/inject: enabled
+      labels:
+        app: rollouts-demo
+    spec:
+      containers:
+      - name: rollouts-demo
+        image: argoproj/rollouts-demo:blue
+        ports:
+        - name: http
+          containerPort: 8080
+          protocol: TCP
+        resources:
+          requests:
+            memory: 32Mi
+            cpu: 5m