Homelab-K8s

This repository contains scripts and mostly yaml files to spin up and deploy services in a Kubernetes cluster running in Proxmox using TalOS as the foundation.

How To

1. Create you secret values: cp secrets/values_template.yaml secrets/values.yaml
2. Add all secrets you need. Especially start with the scriptConfigs section
   • ⚠️ ATTENTION ⚠️ As the secrets are stored in plain text it is highly recommended to use the scripts in scripts/secrets/ to push and pull all secrets (stored in secrets/ directory) to/from a vaultwarden or bitwarden instance. After you have successfully deployed everything delete the contents of secrets directory. Use at your own risk!
3. Generate secrets talosctl gen secrets -o secrets/secret.yaml
4. Generate the TalOS config with scripts/talos/config_generate.sh. The bash script is implemented to create 2 worker configs, each using the configured static IP from secrets/values.yaml
5. If you have not setup your VMs for TalOS follow this guide. Stop at generating/applying any config and come back to this README
6. Deploy the config with the help of nodes_setup.sh
7. Once the cluster is ready to use, set the kubeconfig export KUBECONFIG=$(pwd)/secrets/kubeconfig and deploy all services using scripts/cluster/setup.sh
8. Nodes can be reset using scripts/talos/nodes_reset.sh
9. Config of nodes can be updated using scripts/talos/nodes_update.sh

Future Setup

While working in the cluster and completing my Todos I found out way more elegant processes to bootstrap, deploy, and constantly update the cluster. It will include the projects in below ToDos:

• Sync Secrets with external-secrets
• Switch to Flux
• Use SOPS
• Switch from Kube-Vip to Cilium for Loadbalancer IPs Howto on TalOS and HowTo Migration and Helm Deployment
• Decide about new VM Setup: 1 'small' Controplane VM and 1 'beefy' Worker VM or a single Controlplane that allows scheduling workload as well (config)
• Decide about local-path-provisioner running on TalOS vs. Longhorn. Backups could be realized with a Restic CronJob (tutorial)

Todos

• Successfully create K8s cluster in Proxmox VMs using TalOS Linux
• Multi-Controlplane Cluster using 3 Controlplane TalOS VMs Maybe overkill for a Homelab
• Successfully deploy Cert-Manager which creates valid certificates
• Successfully deploy Kube-Vip
• Successfully deploy Traefik
• A Traefik pod runs on every node to have some kind of HA setup. --> Configure as Daemsonset
• Deploy an app and it is reachable in local network via external IP --> whoami-external
• Deploy an app and it is reachable in local network via some dummy dns + modification of /etc/hosts on a client machine whoami-internal
• Deploy an app end to end with dedicated certificates --> whoami-external
• Decide about storage provider. Requirements:
  • The node that hosts Proxmox has a dedicated 2TB SSD where data will be stored. This must be available to many pods
  • A multi VM multi Node setup must be possible
  • Some PVs that will be used for database and backed up on a daily basis.
  • Some PVs that just store some 'not so important data'. Can be used from the VMs storage itself, but must be checked against how TalOS behaves on reset/upgrade etc.
  • Decision: Longhorn --> Use PV+PVCs backed by longhorn volumes for everything except of media files (movies, music, books etc.), those will be stored and accessed to/from a NFS
• Testing of Longhorn + TalOS
  • Add a second disk to VM
  • Configure it in TalOS (adapt config scripts)
  • Verify talosctl reset does not delete contents of the disk
  • Create PV + PVC; write some data to it using a Deployment; configure and do backup of disk to another disk; check for files on disk; do a disaster recovery once
• The 2 TB Data SSD must be exposed in the cluster to read/write data as well as via SMB share to be able to access the files via Windows and Mac. According to official K8s documentation it is possible to deploy a Samba Deployments. Current idea: Mount Data SSD in a worker VM, make the Samba Server Deployment stick to that Node. But how will other Pods be able to write to that Disk? --> New Idea: Longhorn use RWX for the data disk. This exposes the disk via NFS according to their documentation ---> Final implementation: VM which only acts as NFS share
• Expose USB HDDs in network as backup drives. Currently the Fritzbox exposes them as Samba Shares. Requirements:
  • Longhorn can use the disks; according to Longhorn docs it should be possible to use a Samba Share
• Final Setup for backups: Use RPI to host Openmediavault and connect USB HDDs to this machine.
• successfully deploy storage provider
• TalOS extract secrets and use templating mechanism
• Proper scripts for setup of cluster and cluster installation
• Script to uninstall everything from the cluster
• Minimal README
• Write a script that fills in secrets/values.yaml and secrets/secrets.yaml from defined vaultwarden
• Script that stores current secrets/values.yaml and secrets/secrets.yaml in vaultwarden
• Use traefik ingress objects instead of standard ingress
• Use a single wildcard certificate instead of a certificate for each service
• Successfully deploy Crowdsec
  • Traefik access logs are created and parsed by crowdsec
• Successfully deploy Keel
• Configure Renovate
• Successfully deploy Vaultwarden incl. postgres communication
• Successfully deploy Authentik incl. postgres communication
• Successfully deploy Uptime-Kuma
• Successfully deploy Signoz
• Successfully deploy Portainer
• Successfully deploy Mediastack incl. postgres communication
• Successfully deploy Immich
• Successfully deploy Spoolman
• Successfully deploy Homepage
• Successfully deploy Wallabag
• Default Postgres deployment to be used by many services
• Security Postgres deployment to be used by important services
• Migrate stack from docker-compose to K8s
• Once migrated, test End2End and configure:
  • Crowdsec
  • Vaultwarden
  • Authentik
  • Uptime-Kuma
  • Signoz
  • Mediastack
  • Immich
  • Spoolman
  • Homepage
• Configure backups for Longhorn volumes
• Enhanced security using RBAC
• (Fun project) Write a golang cli to be able to remove the bash scripts. Especially for secrets pushing/pulling
• Networkpolicies for security postgres