Skip to content

Commit

Permalink
Add new "mlab/managed" label to physical k8s nodes (#261)
Browse files Browse the repository at this point in the history
* Adds new mlab/managed label key to physical nodes

Minimal sites do not have switches, yet today the Disco DaemonSet deploys to
every physical machine. On minimal sites Disco crashloops because there is no
switch. This commit adds a new node label named "mlab/managed", which has the
same meaning and possible values as the "managed" metadata field. Indeed, the
value is taken directly from /var/local/metadata/managed.

This script gets run by epoxy_client as part of the setup-after-boot.service.
This service is configured to run _after_ the write-metadata.service, which
should ensure that the file is properly populated with the appropriate value.

* Converts comma in managed value to a dash

For "full" physical sites the content of /var/local/metadata/managed is
"switch,machine". The plan was to use the value of this file as the value of a
new "mlab/managed" node label. However, kubernetes node label values cannot
contain commas. As a workaround, this commit changes any commas to dashes,
which are valid.

* Adds a new MANAGED variable to setup_k8s.sh

This may make the NODE_LABELS variable assignment a bit more readable.
  • Loading branch information
nkinkade authored Apr 16, 2024
1 parent 05caa82 commit 3a98990
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion configs/stage3_ubuntu/opt/mlab/bin/setup_k8s.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,28 @@ MACHINE=${HOSTNAME:0:5}
SITE=${HOSTNAME:6:5}
METRO="${SITE/[0-9]*/}"

# This value will be used to populate the node label "mlab/managed", which will
# allow operators to differentiate betweeen "full", "minimal" and "BYOS"
# physical sites. Since k8s does not support commas in label values, any commas
# are converted to dashes.
MANAGED=$(cat /var/local/metadata/managed | tr ',' '-')

# Adds /opt/bin (k8s binaries) and /opt/mlab/bin (mlab binaries/scripts) to PATH.
# Also, be 100% sure /sbin and /usr/sbin are in PATH.
export PATH=$PATH:/sbin:/usr/sbin:/opt/bin:/opt/mlab/bin

# Capture K8S version for later usage.
RELEASE=$(kubelet --version | awk '{print $2}')

NODE_LABELS="mlab/machine=${MACHINE},mlab/site=${SITE},mlab/metro=${METRO},mlab/type=physical,mlab/project=${GCP_PROJECT},mlab/ndt-version=production"
# Create a list of node labels
NODE_LABELS="mlab/machine=${MACHINE},"
NODE_LABELS+="mlab/site=${SITE},"
NODE_LABELS+="mlab/metro=${METRO},"
NODE_LABELS+="mlab/type=physical,"
NODE_LABELS+="mlab/project=${GCP_PROJECT},"
NODE_LABELS+="mlab/ndt-version=production,"
NODE_LABELS+="mlab/managed=${MANAGED}"

sed -ie "s|KUBELET_KUBECONFIG_ARGS=|KUBELET_KUBECONFIG_ARGS=--node-labels=$NODE_LABELS |g" \
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Expand Down

0 comments on commit 3a98990

Please sign in to comment.