Update docs for three data hall and move something to implemented

[johscheuer]

Description

Please include a summary of the change and which issue is addressed. If this change resolves an issue, please include the issue number in the description.

Type of change

• Documentation

Discussion

Testing

Documentation

Updated.

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 2cc7eac
• Duration 3:19:37
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 2cc7eac
• Duration 2:50:36
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

LGTM

[simenl]

I like the simplicity of this approach!

I do have some concerns about running with using topologySpreadConstraints in production, as the foundationdb operator does not control the placement of the pods.

1. A pod recreation may fail, as the pod could be attached to a disk (PVC) in a zone which has a positive skew. This could happen for several reasons. E.g. a zone being unavailable, or lacking resources.

2. Replacement of pods can cause an uneven distribution of pods across zones. The new pods are added while old pods are removed, and they work against the same topologySpreadConstraint.

[johscheuer]

I like the simplicity of this approach!

I do have some concerns about running with using topologySpreadConstraints in production, as the foundationdb operator does not control the placement of the pods.

1. A pod recreation may fail, as the pod could be attached to a disk (PVC) in a zone which has a positive skew. This could happen for several reasons. E.g. a zone being unavailable, or lacking resources.
2. Replacement of pods can cause an uneven distribution of pods across zones. The new pods are added while old pods are removed, and they work against the same topologySpreadConstraint.

Both points are two valid concerns. We have a plan to improve the "placement" of pods inside the operator with some we called "logical fault domains": https://github.com/FoundationDB/fdb-kubernetes-operator/blob/main/docs/design/bin_pack_fault_domains.md. We target to work on this sometime this year.