Skip to content

Latest commit

 

History

History
98 lines (90 loc) · 8.46 KB

README.md

File metadata and controls

98 lines (90 loc) · 8.46 KB

Homelab-K8s

This repository contains scripts and mostly yaml files to spin up and deploy services in a Kubernetes cluster running in Proxmox using TalOS as the foundation.

How To

  1. Create you secret values: cp secrets/values_template.yaml secrets/values.yaml
  2. Add all secrets you need. Especially start with the scriptConfigs section
    • ⚠️ ATTENTION ⚠️ As the secrets are stored in plain text it is highly recommended to use the scripts in scripts/secrets/ to push and pull all secrets (stored in secrets/ directory) to/from a vaultwarden or bitwarden instance. After you have successfully deployed everything delete the contents of secrets directory. Use at your own risk!
  3. Generate secrets talosctl gen secrets -o secrets/secret.yaml
  4. Generate the TalOS config with scripts/talos/config_generate.sh. The bash script is implemented to create 2 worker configs, each using the configured static IP from secrets/values.yaml
  5. If you have not setup your VMs for TalOS follow this guide. Stop at generating/applying any config and come back to this README
  6. Deploy the config with the help of nodes_setup.sh
  7. Once the cluster is ready to use, set the kubeconfig export KUBECONFIG=$(pwd)/secrets/kubeconfig and deploy all services using scripts/cluster/setup.sh
  8. Nodes can be reset using scripts/talos/nodes_reset.sh
  9. Config of nodes can be updated using scripts/talos/nodes_update.sh

Future Setup

While working in the cluster and completing my Todos I found out way more elegant processes to bootstrap, deploy, and constantly update the cluster. It will include the projects in below ToDos:

Todos

  • Successfully create K8s cluster in Proxmox VMs using TalOS Linux
  • Multi-Controlplane Cluster using 3 Controlplane TalOS VMs Maybe overkill for a Homelab
  • Successfully deploy Cert-Manager which creates valid certificates
  • Successfully deploy Kube-Vip
  • Successfully deploy Traefik
  • A Traefik pod runs on every node to have some kind of HA setup. --> Configure as Daemsonset
  • Deploy an app and it is reachable in local network via external IP --> whoami-external
  • Deploy an app and it is reachable in local network via some dummy dns + modification of /etc/hosts on a client machine whoami-internal
  • Deploy an app end to end with dedicated certificates --> whoami-external
  • Decide about storage provider. Requirements:
    • The node that hosts Proxmox has a dedicated 2TB SSD where data will be stored. This must be available to many pods
    • A multi VM multi Node setup must be possible
    • Some PVs that will be used for database and backed up on a daily basis.
    • Some PVs that just store some 'not so important data'. Can be used from the VMs storage itself, but must be checked against how TalOS behaves on reset/upgrade etc.
    • Decision: Longhorn --> Use PV+PVCs backed by longhorn volumes for everything except of media files (movies, music, books etc.), those will be stored and accessed to/from a NFS
  • Testing of Longhorn + TalOS
    • Add a second disk to VM
    • Configure it in TalOS (adapt config scripts)
    • Verify talosctl reset does not delete contents of the disk
    • Create PV + PVC; write some data to it using a Deployment; configure and do backup of disk to another disk; check for files on disk; do a disaster recovery once
  • The 2 TB Data SSD must be exposed in the cluster to read/write data as well as via SMB share to be able to access the files via Windows and Mac. According to official K8s documentation it is possible to deploy a Samba Deployments. Current idea: Mount Data SSD in a worker VM, make the Samba Server Deployment stick to that Node. But how will other Pods be able to write to that Disk? --> New Idea: Longhorn use RWX for the data disk. This exposes the disk via NFS according to their documentation ---> Final implementation: VM which only acts as NFS share
  • Expose USB HDDs in network as backup drives. Currently the Fritzbox exposes them as Samba Shares. Requirements:
    • Longhorn can use the disks; according to Longhorn docs it should be possible to use a Samba Share
  • Final Setup for backups: Use RPI to host Openmediavault and connect USB HDDs to this machine.
  • successfully deploy storage provider
  • TalOS extract secrets and use templating mechanism
  • Proper scripts for setup of cluster and cluster installation
  • Script to uninstall everything from the cluster
  • Minimal README
  • Write a script that fills in secrets/values.yaml and secrets/secrets.yaml from defined vaultwarden
  • Script that stores current secrets/values.yaml and secrets/secrets.yaml in vaultwarden
  • Use traefik ingress objects instead of standard ingress
  • Use a single wildcard certificate instead of a certificate for each service
  • Successfully deploy Crowdsec
    • Traefik access logs are created and parsed by crowdsec
  • Successfully deploy Keel
  • Configure Renovate
  • Successfully deploy Vaultwarden incl. postgres communication
  • Successfully deploy Authentik incl. postgres communication
  • Successfully deploy Uptime-Kuma
  • Successfully deploy Signoz
  • Successfully deploy Portainer
  • Successfully deploy Mediastack incl. postgres communication
  • Successfully deploy Immich
  • Successfully deploy Spoolman
  • Successfully deploy Homepage
  • Successfully deploy Wallabag
  • Default Postgres deployment to be used by many services
  • Security Postgres deployment to be used by important services
  • Migrate stack from docker-compose to K8s
  • Once migrated, test End2End and configure:
    • Crowdsec
    • Vaultwarden
    • Authentik
    • Uptime-Kuma
    • Signoz
    • Mediastack
    • Immich
    • Spoolman
    • Homepage
  • Configure backups for Longhorn volumes
  • Enhanced security using RBAC
  • (Fun project) Write a golang cli to be able to remove the bash scripts. Especially for secrets pushing/pulling
  • Networkpolicies for security postgres