Monitoring of a photovoltaic system, car charging and heat pump
- 7 kW peak solar power system
- KOSTAL PLENTICORE plus inverter
- go-eCharger HOMEfix 11kW wallbox
- Nibe VVM310 + S2125-8 heat pump
- monitoring and integration into Home Assistent via nibe-mqtt-gateway
- Monitoring with Prometheus running on a k3s Kubernetes cluster
- standard Prometheus instance with 7d data retention and 30s scrape interval
- long-term Prometheus instance with infinite data data retention and 15min scrape interval connected to the first instance using federation
- inverter and wall box connected using modbus_exporter
- k3s lightweight Kubernetes cluster
- 3 k3s servers using embedded etcd
- nasbox: Celeron G3900 (2 core), 32G RAM, 1T SSD, 4T raid1 disks
- pi1: Raspberry Pi 4, 8G RAM, 512G SSD
- pi2: Orange Pi 5 Max, 16G RAM, 1T SSD
- pv-control for controlling electric car charger
- charge car by solar power only
- 1 and 3 phase charging to get a wide control range starting at 1.3 kW up to (theoretical) 11kW
- UI for controlling charge modes
- Home Assistant for additional home automation
- Paperless-ngx for taming the bureaucracy
- Immich for managing photos
- Kopia for backups
This repo is structured by namespace and app: pv-monitoring/<namespace>/<app>. Every directory contains a deploy.sh script which recursively deploys the k8s resources of the directory.
Software versions including helm chart versions are maintained in deploy.sh and yaml files. Versions are kept up-to-ate by the renovate bot.
Important release notes
- cloudnative-pg
- csi-driver-smb
- homeassistant
- immich
- k3s
- kube-prometheus-stack
- longhorn
- metallb
- paperless-ngx
- zot
- kubectl
- curl, yq
- regctl
regctl registry set --tls=disabled registry.fritz.box
Standard installation as described in https://rancher.com/docs/k3s/latest/en/quick-start/.
All http and tcp workloads are exposed via Traefik v2 which is deployed as daemon set on all nodes:
- k3s.fritz.box - via MetalLB
- nasbox.fritz.box, pi1.fritz.box, pi2.fritz.box - via node ports
- additional DNS names like homeassistant.fritz.box - translated to node IPs or MetalLB IP via dnsmasq + customized coredns
MetalLB is used as LB for special services that need an own IP. E.g. for dnsmasq which is used as internal DNS server as the Fritzbox doesn't allow to add additional host names.
k8s API is available via:
- k3s.fritz.box:6443, MetalLB as Cluster Load Balancer -> Traefik ->
kubernetesservice in default namespace - port 6443 on every node
WIP: OCI image registry: zot
- mirror for docker.io, ghcr.io etc. to avoid slow image downloads on e.g. node
# first server (nasbox)
curl -sfL https://get.k3s.io | sh - --disable coredns,servicelb --embedded-registry --cluster-init
# other servers (pi1, pi2)
curl -sfL https://get.k3s.io | K3S_URL=https://192.168.178.10:6443 \
K3S_TOKEN=<join_token> sh - --disable coredns,servicelb --embedded-registry --cluster-init
Config file /etc/rancher/k3s/config.yaml
disable: coredns,servicelb
embedded-registry: false
tls-san:
- "192.168.178.230"
- k3s.fritz.box
etcd-expose-metrics: true
# https://github.com/k3s-io/k3s/issues/3619#issuecomment-993977516
kube-controller-manager-arg:
- "bind-address=0.0.0.0"
kube-proxy-arg:
- "metrics-bind-address=0.0.0.0"
kube-scheduler-arg:
- "bind-address=0.0.0.0"
# https://github.com/k3s-io/k3s/discussions/10125
kubelet-arg:
- "config=/etc/rancher/k3s/kubelet-config.yaml"
Kubelet config file /etc/rancher/k3s/kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
# https://github.com/k3s-io/k3s/discussions/10125
imageMaximumGCAge: 168h # 1 week
# https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
Configure registry mirroring in file /etc/rancher/k3s/registries.yaml: see ./zot/zot/registries.yaml
Others:
- nasbox: Disable multipath for
sd[a-z0-9]+devices as described in Troubleshooting:MountVolume.SetUp failed for volumedue to multipathd on the node
(currently no agents)
# enable cgroups, see https://docs.k3s.io/installation/requirements?os=pi (not needed for Armbian)
# needed for longhorn
sudo apt install open-iscsi
curl -sfL http://get.k3s.io | K3S_URL=https://192.168.178.10:6443 \
K3S_TOKEN=<join_token> sh -
Config file /etc/rancher/k3s/config.yaml
# https://github.com/k3s-io/k3s/discussions/10125
kubelet-arg:
- "config=/etc/rancher/k3s/kubelet-config.yaml"
Kubelet config file /etc/rancher/k3s/kubelet-config.yaml: same as for server.
Registry mirroring /etc/rancher/k3s/registries.yaml: same as for server
Automatic k3s updates: https://rancher.com/docs/k3s/latest/en/upgrades/automated/
Maintain k3s version in upgrade plans: ./system-upgrade/k3s-upgrade.yaml