Skip to content

Commit

Permalink
docs: Controller sharding and horizontal scaling
Browse files Browse the repository at this point in the history 
Signed-off-by: Stefan Prodan <[email protected]>
  • Loading branch information
@stefanprodan
stefanprodan committed Apr 4, 2023
1 parent eba4edf commit f2d26e9
Show file tree
Hide file tree
Showing 2 changed files with 329 additions and 1 deletion.
7 changes: 6 additions & 1 deletion content/en/flux/cheatsheets/bootstrap.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "Bootstrap cheatsheet"
linkTitle: "Bootstrap"
description: "Showcase various configurations of Flux controllers at bootstrap time."
weight: 30
weight: 29
---

## How to customize Flux
Expand Down Expand Up @@ -193,6 +193,11 @@ patches:
name: "(kustomize-controller|helm-controller|source-controller)"
```
{{% alert color="info" title="Horizontal scaling" %}}
When vertical scaling is not an option, you can use sharding to horizontally scale
the Flux controllers. For more details please see the [sharding guide](sharding.md).
{{% /alert %}}
### Persistent storage for Flux internal artifacts
Flux maintains a local cache of artifacts acquired from external sources.
Expand Down
323 changes: 323 additions & 0 deletions content/en/flux/cheatsheets/sharding.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,323 @@
---
title: "Controller sharding and horizontal scaling"
linkTitle: "Sharding"
description: "Showcase how to configure sharding of Flux controllers at bootstrap time."
weight: 30
---

When Flux is managing tens of thousands of applications, it is advised to adopt
a sharding strategy to spread the load between multiple instances of Flux controllers.
To enable horizontal scaling, each controller can be deployed multiple times
with a unique label selector which is used as the sharding key.

What follows is a guide on how to bootstrap multiple controller instances and how to
shard the reconciliation of Flux resources using the `sharding.fluxcd.io/key` label.

## Bootstrap with sharding

At [bootstrap time](bootstrap.md), you can define the number of shards
and spin up a Flux controller instance for each shard.

First you'll need to create a Git repository and clone it locally, then
create the file structure required by bootstrap with:

```sh
mkdir -p clusters/my-cluster/flux-system
touch clusters/my-cluster/flux-system/gotk-components.yaml \
clusters/my-cluster/flux-system/gotk-sync.yaml \
clusters/my-cluster/flux-system/kustomization.yaml
```

Then you'll create a dedicated directory inside `flux-system` for each shard:

```sh
mkdir -p clusters/my-cluster/flux-system/shard1
touch clusters/my-cluster/flux-system/shard1/kustomization.yaml
```

### Configure controller sharding

In the `shard1` directory you can generate a set of controller deployments that
will reconcile the Flux resources labels with `sharding.fluxcd.io/key: shard1`.

To spin up a dedicated source-controller, kustomize-controller and helm-controller instance,
use the following patches in `clusters/my-cluster/flux-system/shard1/kustomization.yaml`:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
- ../gotk-components.yaml
nameSuffix: "-shard1"
commonAnnotations:
sharding.fluxcd.io/role: "shard"
patches:
- target:
kind: (Namespace|CustomResourceDefinition|ClusterRole|ClusterRoleBinding|ServiceAccount|NetworkPolicy)
labelSelector: "app.kubernetes.io/part-of=flux"
patch: |
apiVersion: v1
kind: all
metadata:
name: all
$patch: delete
- target:
labelSelector: "app.kubernetes.io/component=notification-controller"
patch: |
apiVersion: v1
kind: all
metadata:
name: all
$patch: delete
- target:
kind: Deployment
name: (image-reflector-controller|image-automation-controller)
patch: |
apiVersion: v1
kind: Deployment
metadata:
name: all
$patch: delete
- target:
kind: Service
name: source-controller
patch: |
- op: replace
path: /spec/selector/app
value: source-controller-shard1
- target:
kind: Deployment
name: source-controller
patch: |
- op: replace
path: /spec/selector/matchLabels/app
value: source-controller-shard1
- op: replace
path: /spec/template/metadata/labels/app
value: source-controller-shard1
- op: replace
path: /spec/template/spec/containers/0/args/6
value: --storage-adv-addr=source-controller-shard1.$(RUNTIME_NAMESPACE).svc.cluster.local.
- target:
kind: Deployment
name: kustomize-controller
patch: |
- op: replace
path: /spec/selector/matchLabels/app
value: kustomize-controller-shard1
- op: replace
path: /spec/template/metadata/labels/app
value: kustomize-controller-shard1
- target:
kind: Deployment
name: helm-controller
patch: |
- op: replace
path: /spec/selector/matchLabels/app
value: helm-controller-shard1
- op: replace
path: /spec/template/metadata/labels/app
value: helm-controller-shard1
- target:
kind: Deployment
name: (source-controller|kustomize-controller|helm-controller)
patch: |
- op: add
path: /spec/template/spec/containers/0/args/-
value: --watch-label-selector=sharding.fluxcd.io/key=shard1
```
The above configuration will generate three deployments `source-controller-shard1`,
`kustomize-controller-shard1` and `helm-controller-shard1` all configured
with `--watch-label-selector=sharding.fluxcd.io/key=shard1`.

To enable these deployments at bootstrap, you need to add the `shard1` directory to
the `clusters/my-cluster/flux-system/kustomization.yaml` resources:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml
- shard1
patches:
- target:
kind: Deployment
name: "(source-controller|kustomize-controller|helm-controller)"
annotationSelector: "sharding.fluxcd.io/role notin (shard)"
patch: |
- op: add
path: /spec/template/spec/containers/0/args/0
value: --watch-label-selector=sharding.fluxcd.io/key notin (shard1, shard2)
```

Note that you must exclude the sharding keys from the main controllers watch with
`--watch-label-selector=sharding.fluxcd.io/key notin (shard1, shard2)`. This ensures
that the main controllers will not reconcile any Flux resources labels with the sharding keys.

### Install and Upgrade shards

Push the changes to main branch:

```sh
git add -A && git commit -m "init flux" && git push
```

And run the bootstrap for `clusters/my-cluster`:

```sh
flux bootstrap git \
--url=ssh://git@<host>/<org>/<repository> \
--branch=main \
--path=clusters/my-cluster
```

Verify that the main controllers and the ones assigned to shard1 are running:

```console
$ kubectl -n flux-system get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
helm-controller 1/1 1 1 1m
helm-controller-shard1 1/1 1 1 1m
kustomize-controller 1/1 1 1 1m
kustomize-controller-shard1 1/1 1 1 1m
notification-controller 1/1 1 1 1m
source-controller 1/1 1 1 1m
source-controller-shard1 1/1 1 1 1m
```

When upgrading Flux, either by rerunning bootstrap with a newer version or
by using the [Flux GitHub Actions](https://github.com/fluxcd/flux2/tree/main/action),
the sharded controllers will be automatically upgraded along with the main ones.

## Assign resources to shards

To assign a group of Flux resources to a particular shard, you need to label
them with `sharding.fluxcd.io/key`.

For example, assuming you want to assign the reconciliation of an application
to the `shard1` controllers, you can label both the Flux source and its Kustomization
with `sharding.fluxcd.io/key: shard1`:

```yaml
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: podinfo
namespace: default
labels:
sharding.fluxcd.io/shard: shard1
spec:
interval: 10m
url: https://github.com/stefanprodan/podinfo
ref:
semver: 6.x
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: podinfo
namespace: default
labels:
sharding.fluxcd.io/shard: shard1
spec:
interval: 10m
targetNamespace: default
sourceRef:
kind: GitRepository
name: podinfo
path: ./kustomize
```

Note that Source object kinds which have a dependency on another kind
(i.e. `HelmChart` on a `HelmRepository`) need to have the same labels
applied to work as expected.

For example, assuming you want to assign the reconciliation of a Helm release
to the `shard1` controllers, you can label the HelmRelease, its chart and its repository
with `sharding.fluxcd.io/key: shard1`:

```yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: podinfo
namespace: default
labels:
sharding.fluxcd.io/shard: shard1
spec:
interval: 10m
type: oci
url: oci://ghcr.io/stefanprodan/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: podinfo
namespace: default
labels:
sharding.fluxcd.io/shard: shard1
spec:
interval: 10m
chart:
metadata:
labels:
sharding.fluxcd.io/shard: shard1
spec:
chart: podinfo
sourceRef:
kind: HelmRepository
name: podinfo
```

Note that with `.spec.chart.metadata.labels` we set the shading key on the
generated Flux `HelmChart` object, so that both the Helm repository and charts
are managed by source-controller-shard1 instance.

### Bulk assign shards

Instead of manual labeling each Flux resource with a shard key, you can use a top level
Flux Kustomization and automatically label all resources.

For example, assuming you want to assign a tenant to a particular shard, in the
root Flux Kustomization that reconcile the tenant's Flux sources, kustomizations and
Helm releases you can label these resources as follows:

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: tenant1
namespace: tenant1
spec:
interval: 10m0s
path: ./apps
sourceRef:
kind: GitRepository
name: tenant1
targetNamespace: tenant1
commonMetadata:
labels:
sharding.fluxcd.io/key: shard1
patches:
- target:
kind: HelmRelease
patch: |
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: all
spec:
chart:
metadata:
labels:
sharding.fluxcd.io/key: shard1
```

With `.spec.commonMetadata.labels` we set the shading key on all
the Flux resources and with the `.spec.patches` we set the same shading key
for all generated `HelmCharts`.

0 comments on commit f2d26e9

Please sign in to comment.