Kubernetes

Kubernetes

• Kubernetes / AWS Architecture Platform - https://github.com/ObrienlabsDev/kubernetes-aws-platform-infra

My Kubernetes Issues on the CNCF

• https://github.com/kubernetes/kubernetes/issues/128460

Best Practices

Kubernetes Landing Zone - KLZ

Base Services

Karpenter

• vertical and horizonatl autoscaling - https://karpenter.sh/

Prometheus

Grafana

Consul

ELK

Gitlab

ArgoCD

cert-manager

• see https://github.com/ObrienlabsDev/pipeline/issues/3

Kubernetes Commands

Pods

Deployments

Services

Volumes

Volume Types

emptyDir - r/w pod etherial

hostPath

configMap.secret

nfs

persistentVolumeClaim

ConfigMaps

Secrets

Ingress Controllers

Kubernetes Installation

CAPI Management cluster

• https://github.com/ObrienlabsDev/pipeline/issues/13
• https://github.com/ObrienlabsDev/blog/issues/112

Kubernetes via Kubeadm on 3 Mac Mini M4 pro machines running Ubuntu 24.04 8Gb VMs on VMWare Fusion 13

• Kubernetes on Mac Mini M4 Pro Machines using Thunderbolt 5 networking on VMWare Fusion 13 Ubuntu 24.04 VMs Tracking: https://github.com/ObrienlabsDev/blog/issues/110

Kubernetes via Kubeadm on 2 Local Ubuntu 24.04 8Gb VMs on VMWare Fusion 13 on Macbook Pro M4 Max

Tracking: https://github.com/ObrienlabsDev/blog/issues/75

Provision two Ubuntu VMs in VMWare

Use 60g HD, 10 of 16 cores, 8G ram.

Don't add anything except Open SSH server

install net-tools for ifconfig

sudo apt install net-tools

Test remote login and add ssh-key

Follow Kubernetes Docs - Install Kubeadm

On both machines...

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

start at

https://kubernetes.io/docs/setup/

we will come back to the learning section

https://kubernetes.io/docs/tasks/tools/

We go directly to "installing kubeadm" in the production section

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Check prerequisites

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#verify-mac-address

MAC addresses are different - if VMs were independently created.

Disable swap

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#swap-configuration

Add a container runtime

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

IPv4 Packet filtering

ubuntu@ubuntu01:~$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF
net.ipv4.ip_forward = 1
ubuntu@ubuntu01:~$ cat /etc/sysctl.d/k8s.conf 
net.ipv4.ip_forward = 1

ubuntu@ubuntu01:~$ sudo sysctl --system
sysctl net.ipv4.ip_forward

Containerd install

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd via https://github.com/containerd/containerd/blob/main/docs/getting-started.md

Via Kubeadm on 7 Local Ubuntu 24.04 on Raspberry PI 5 8Gb

Kubeadm

• https://github.com/ObrienlabsDev/blog/issues/50
• https://github.com/ObrienlabsDev/blog/issues/54

Helm

• https://github.com/ObrienlabsDev/biometric-backend/issues/6#issuecomment-2275686040

RKE / Rancher Installations

Rancher 1.6 to 2.0 was the goto bare metal or VM supported kubernetes distribution in the 2017-2019 timeframe around Kubernetes 1.7. The rancher product was integrated into SUSE and continues to support RKE 1 (original) and RKE 2 (FedRAMP compliant). K3s is the edge version of the RKE distribution as a CNCF project.

https://docs.rke2.io/

see - https://github.com/ObrienlabsDev/biometric-backend/issues/11

RKE supported versions

Limit installations to Ubuntu 22.04+ on IA64 (ARM64 is experimental). As of 202408 the following kubernetes components are supported for v1.30.2

• Kubernetes: v1.30.2
• RKE2 Version: v1.30.2+rke2r1
• Etcd: v3.5.13-k3s1
• Containerd: v1.7.17-k3s1
• Runc: v1.1.12
• Metrics-server: v0.7.1
• CoreDNS: v1.11.1
• Ingress-Nginx: v1.10.1-hardened1
• Helm-controller: v0.16.1
• CNI: Canal (Flannel: v0.25.4, Calico: v3.28.0)
• CNI: Calico v3.27.3
• CNI: Cilium v1.15.5
• CNI: Multus v4.0.2

Minikube

• https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download

Minikube on Ubuntu 22 on MAC IA64 Hardware

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64
minikube start

Docker Desktop

Enable kubernetes 1.29

Triage kubectl not working

For

ichaelobrien@mbp7 kubernetes % kubectl get nodes
E0802 19:52:50.751864   60075 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:64151/api?timeout=32s": dial tcp 127.0.0.1:64151: connect: connection refused
The connection to the server 127.0.0.1:64151 was refused - did you specify the right host or port?

If previously running "kind" switch to docker desktop

michaelobrien@mbp7 kubernetes % kubectl get nodes    
NAME             STATUS   ROLES           AGE     VERSION
docker-desktop   Ready    control-plane   4m18s   v1.29.2

Google Kubernetes Engine - GKE

AWS EKS

• setup eksctl https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html#eksctl-install-update
• 20241025: follow https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html

Create VPC

Create a VPC with 2 public, 2 private subnets - https://docs.aws.amazon.com/eks/latest/userguide/creating-a-vpc.html#_public_and_private_subnets

Select Kubernetes V1.31 - as V1.30 will autocharge support fees in 8 months "End of standard support for Kubernetes version 1.30 is July 28, 2025. On that date, your cluster will enter the extended support period with additional fees. For more information"

Connect via kubectl context

aws eks update-kubeconfig --region us-east-1 --name prod
Added new context arn:aws:eks:us-east-1:45..:cluster/prod to /Users/..n/.kube/config

kubectl get nodes
NAME                            STATUS   ROLES    AGE   VERSION
ip-172-31-42-248.ec2.internal   Ready    <none>   21h   v1.30.4-eks-a737599
ip-172-31-80-165.ec2.internal   Ready    <none>   21h   v1.30.4-eks-a737599

kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   aws-node-8bwzk                 2/2     Running   0          21h
kube-system   aws-node-f9mz4                 2/2     Running   0          21h
kube-system   coredns-586b798467-fdvwr       1/1     Running   0          11d
kube-system   coredns-586b798467-gff7r       1/1     Running   0          11d
kube-system   eks-pod-identity-agent-cq8nn   1/1     Running   0          21h
kube-system   eks-pod-identity-agent-g4wxv   1/1     Running   0          21h
kube-system   kube-proxy-b8v4z               1/1     Running   0          21h
kube-system   kube-proxy-h2ttb               1/1     Running   0          21h

Azure

IBM Redhat Openshift Container Platform

• Latest 4.12

OCP Training

• https://www.redhat.com/en/services/training/do080-deploying-containerized-applications-technical-overview
• • https://rhtapps.redhat.com/promo/course/do080?segment=0
• https://www.redhat.com/en/services/training/red-hat-openshift-administration-i-operating-a-production-cluster
• https://www.redhat.com/en/services/training/do281-red-hat-openshift-administration-ii-configuring-a-production-cluster-with-exam

OCP on RHEL on prem

• https://www.redhat.com/en/technologies/cloud-computing/openshift/pricing
• https://www.redhat.com/en/technologies/cloud-computing/openshift/kubernetes-engine#buy

OCP on AWS

• https://www.redhat.com/en/technologies/cloud-computing/openshift/pricing

OCP on Azure

OCP on GCP

Kubernetes Projects

• config-syncer (was kubed) - similar to GCP config-sync - https://github.com/config-syncer/config-syncer