Art by @SkeletalGadget
IaC for my homelab and personal cloud
[ dotfiles · charts · containers · blog ]
This repository declares all of my infrastructure and Kubernetes clusters, both self-hosted and in Hetzner Cloud. I also host all of my documentation here.
Admittedly, both usages of "all" describe the end goal of this repo, not the current state. But, I will get there some day.
- KCL: Configuration language; with Helm support via kclipper.
- Talos: Immutable Kubernetes OS; built using talhelper.
- TrueNAS: Big ZFS storage; runs small Talos containers for direct I/O.
- Terraform: Declares any infrastructure not managed by Kubernetes.
- Renovate: Automatic updates for applications via pull requests.
- Argo CD: Reconciles Kubernetes clusters with this repository.
- Spegel: Stateless, fully transparent pull-through image cache.
- Reloader: Automatic rollouts on ConfigMap/Secret updates.
- Descheduler: Evicts Pods to maintain zone and node balance.
- Cilium: eBPF-based CNI, BGP control plane, firewall, and more.
- Envoy Gateway: Implements the Kubernetes Gateway API.
- Cert Manager: Automatic Let's Encrypt certificates.
- External DNS: Automatic DNS record management.
- AdGuard Home: DNS server with ad-blocking.
- Wireguard: Modern VPN tunnels.
- External Secrets: Synchronizes secrets from Doppler into Kubernetes.
- Tetragon: eBPF-based security observability and runtime enforcement.
- SecureCodeBox: Continuous and automated security testing with familiar tools like Nmap, ZAP.
- Loki: Log aggregation system.
- Grafana: Visualization platform.
- Tempo: Distributed tracing system.
- Mimir: Prometheus-compatible monitoring system and TSDB.
- Alloy: Grafana's distribution of OpenTelemetry collector.
- Beyla: Zero-touch eBPF auto-instrumentation (part of Alloy).
- Robusta: Alert and notification management.
- OpenEBS: Manages local and replicated persistent volumes.
- CloudNativePG: Manages highly-available, cloud-native Postgres clusters.
- Dragonfly: Highly-available, cloud-native Redis and Memcached implementation.
This repository implements a GitOps architecture, primarily orchistrated by Argo CD ApplicationSets defined as KCL with kclipper. The repo's structure directly informs ApplicationSet behavior via matrix generators. The libraries used are based on KCL's konfig.
This structure enables a readable application hierarchy where each tenant can effectively function independently, i.e. somewhat mirroring an actual production multi-tenant platform. However, what would be individual repositories with their own access controls, releases, and so on, are instead represented as folders in this monorepo.
.
├─📁 apps # KCL-based applications organized by tenants
│ ├─📁 argo # Tenant: argo project
│ │ ├─📁 _tenant # Tenant-level shared configuration
│ │ │ ├─📁 base # Base tenant resources
│ │ │ │ └─📄 .tenant.yaml # Configures this tenant's "apps" ApplicationSet
│ │ │ └─📁 shared # Shared tenant resources
│ │ │ └─📄 .tenant.yaml # Configures this tenant's "shared" ApplicationSet
│ │ └─📁 cd # Application: argo-cd namespace
│ │ ├─📁 base # Base app configuration
│ │ └─📁 mgmt # Management cluster environment
│ │ └─📄 .app.yaml # Configures this cluster's Argo CD Application
│ └─📁 ... # Additional tenants
├─📁 appsets # ArgoCD ApplicationSets for multi-cluster deployment
│ └─📄 tenants.yaml # Matrix generator deploying tenant ApplicationSets
├─📁 bootstrap # Cluster bootstrap configurations
│ └─📁 core # Essential components (Cilium, ArgoCD)
├─📁 charts # Kclipper wrappers for Helm charts
│ ├─📁 argo_cd # Auto-generated ArgoCD kclipper wrapper
│ ├─📁 ... # Additional auto-generated chart wrappers
│ └─📄 charts.k # Kclipper chart definitions
├─📁 clusters # Cluster configuration (Talos, KCL constants)
│ ├─📁 main # Main cluster config
│ └─📁 mgmt # Management cluster config
└─📁 konfig # Custom KCL library for Kubernetes abstractions
├─📁 models # Core data models
│ ├─📁 backend # Low-level Kubernetes resource models
│ ├─📁 frontend # High-level application abstractions
│ ├─📁 mixins # Reusable configuration mixins
│ ├─📁 protocol # Interface definitions
│ ├─📁 render # Rendering logic for YAML output
│ └─📁 templates # Model templates
└─📁 ... # Utility packages, etc.| Service | Use | Cost |
|---|---|---|
| Hetzner Cloud | Cloud compute and storage | $40/mo |
| Google Cloud | Cloud storage | $20/mo |
| Cloudflare | DNS, Certs, Proxy, WAF | Free |
| Doppler | Secrets with External Secrets | Free |
| GitHub | Hosting this repository and CI/CD workflows | Free |
| Robusta | Alerts and notifications | Free |
| Terraform Cloud | Storing Terraform state | Free |
| Grafana Cloud | Hosted Grafana / LGTM Stack | Free |
| Auth0 | IDP / Authentication and authorization platform | Free |
| Unifi Site Manager | Multi-site Unifi gateway management | Free |
| Total: $60/mo |
| Provider | Plan | Modem | Speed (Down) | Speed (Up) | Latency | Purpose | Cost |
|---|---|---|---|---|---|---|---|
| T-Mobile | Home Internet 5G | InvisaGig | 600 Mbps | 150 Mbps | ~ 50ms (100mi) | Primary | $40/mo |
| Spectrum | Basic Cable | Ubiquiti UCI | 300 Mbps | 10 Mbps | ~ 15ms (100mi) | Backup & Gaming | $50/mo |
| Total: $90/mo |
| Item | Consumption | Rate | Cost |
|---|---|---|---|
| Homelab | ~ 400W (Avg) | $0.14/kWh | $45/mo |
| Total: $45/mo |
| Count | Device | Memory | Disk | OS | Purpose |
|---|---|---|---|---|---|
| 3 | Turing Pi 2 | 128MB | 1GB NAND | TPi BMC | 4-Node Cluster Board |
| 1 | Raspberry Pi 4B | 4GB | 32GB SD Card | PiKVM | Network KVM |
| Count | Device | Memory | Disk | OS | Purpose |
|---|---|---|---|---|---|
| 3 | Raspberry Pi CM4 | 8GB | 32GB eMMC | Talos | Control Plane |
| 3 | Turing RK1 | 32GB | 1TB NVMe | Talos | Workers (arm64) |
| Count | Device | Memory | Disk | OS | Purpose |
|---|---|---|---|---|---|
| 3 | Turing RK1 | 32GB | 1TB NVMe | Talos | Control Plane |
| 3 | Supermicro M11SDV-8C+-LN4F | 128GB | 4TB SSD | Talos | Workers (x86) |
| 3 | Turing RK1 | 32GB | 1TB NVMe | Talos | Workers (arm64) |
| 1 | TrueNAS Mini R | 64GB | 200TB HDD | SCALE | Storage + Worker |
| Count | Device | Eth Interfaces | SFP Interfaces | Platform | Purpose |
|---|---|---|---|---|---|
| 1 | Ubiquiti UDM-SE | 1x 2.5G | 2x 10G | UniFi OS | Router & Security Gateway |
| 1 | Ubiquiti U6-Pro | 1x 1G | N/A | UniFi OS | WiFi 6 Access Point |
| 1 | Ubiquiti USW-Pro-Aggregation | N/A | 28x 10G | UniFi OS | L3 Aggregation Switch |
| 1 | Ubiquiti USW-Pro-24 | 24x 1G | 2x 10G | UniFi OS | L3 Switch |
| 1 | Ubiquiti USW-Pro-24-POE | 24x 1G | 2x 10G | UniFi OS | L3 PoE Switch |
| 2 | WattBox WB-800-IPVM-12 | 1x 1G | N/A | OvrC | IP Controlled Metered PDU |
| 2 | WattBox WB-800-IPVM-6 | 1x 1G | N/A | OvrC | IP Controlled Metered PDU |
| 1 | WattBox WB-800VPS-IPVM-18 | 1x 1G | N/A | OvrC | IP Controlled Metered PDU |
Over time I've taken a ton of inspiration from the K8s@Home / home-ops community: onedr0p, szinn, budimanjojo, buroa, coolguy1771, and many others.
Technically however, I hope this repo is quite unique. I've intentionally tried to make some uncommon choices to learn more and venture outside my comfort zone a bit. So, I hope that in the very least, this repo will provide anyone looking with some interesting and unique ideas. 🙂
This project is licensed under the Apache-2.0 license, primarily because it's very compatible with a lot of the projects I enjoy stealing code from.
For more details, see LICENSE.
Ultimately though, I have a WTFPL mindset about any content produced by/for myself. If you like anything you see here, feel free to use it however you want (yes, that includes the peepos), just don't sue me if my code blows up your cluster. If you're feeling especially nice, links back to this repo are always appreciated (for the SEO, or whatever).
