Add multiple node PVE templates (with local storage) capability to cluster classes.

Author: holmesb

docs/advanced-setups.md

@@ -197,6 +197,47 @@ For example, you can set the `TEMPLATE_TAGS="tag1,tag2"` environment variable. Y
 * Each node listed in the `allowedNodes` is expected to have a copy of the template stored locally.
 * If any of the nodes in `allowedNodes` is missing the template, the operation fails.
 
+### Template lookup and cluster classes
+With the ClusterClasses provided in this repo, in your cluster you can choose one of two ways to tell CAPMOX which 
+Proxmox VM template to clone from. 1) Selector mode (tag-based) — recommended for multi-node setups:
+```yaml
+spec:
+  topology:
+    variables:
+      - name: templateSelector
+        value:
+          matchTags: ["capmox","ubuntu-24.04","k8s-1.33"]
+```
+This enables `templateSelector.matchTags`, and removes `sourceNode` and `templateID`.
+
+Or 2) Explicit mode (sourceNode/templateID) — default if templateSelector is not set.
+```yaml
+spec:
+  topology:
+    variables:
+      - name: cloneSpec
+        value:
+          machineSpec:
+            controlPlane:
+              sourceNode: pve1
+              templateID: 100
+            workerNode:
+              sourceNode: pve1
+              templateID: 100
+            loadBalancer:
+              sourceNode: pve1
+              templateID: 100
+ ```
+
+If using local (not shared) storage on nodes, you can also set `localStorage: true` in the selector:
+```yaml
+spec:
+  topology:
+    variables:
+      - name: localStorage
+        value: true
+```
+See earlier `localStorage` section for details.
 
 ## Proxmox RBAC with least privileges
 
@@ -322,6 +363,71 @@ spec:
 You can set either `ipv4PoolRef` or `ipv6PoolRef` or you can also set them both for dual-stack.
 It's up for you also to manage the IP Pool, you can choose a `GlobalInClusterIPPool` or an `InClusterIPPool`.
 
+## Additional Volumes
+By default, only a boot volume is created in machines. If additional disks are required for data storage, they can be 
+specified in the ProxmoxMachineTemplates.
+
+```yaml
+kind: ProxmoxMachineTemplate
+spec:
+  template:
+    spec:
+      storage: local-lvm  # Optional: a default storage to use when a volume doesn't set .storage
+      disks:
+        additionalVolumes:
+          - disk: scsi1
+            sizeGb: 200
+          - disk: scsi2  # target slot (e.g. scsi1, sata1, virtio1, ide2)
+            sizeGb: 80  # capacity in gigabytes
+            # Optional flags:
+            storage: my-nfs  # Optional per-volume storage override. Uses .spec.template.spec.storage if omitted
+            format: qcow2  # Only specify if using file-backed storage. If omitted, default for disk is used.
+            discard: true
+            ioThread: true
+            ssd: true
+```
+In the same way, additionalVolumes can also be specified in ProxmoxClusters, ProxmoxClusterTemplates, 
+and ProxmoxMachines. Flags: format, discard, ioThread, and ssd are supported by this provider.
+See Proxmox [docs](https://pve.proxmox.com/pve-docs/qm.1.html#qm_hard_disk) for details about these flags.
+
+Alternatively if using cluster-class, define additionalVolumes in your cluster:
+```yaml
+kind: Cluster
+spec:
+  topology:
+    class: proxmox-clusterclass-cilium-v0.1.0
+    variables:
+      - name: workerAdditionalVolumes
+        value:
+          - { disk: scsi1, sizeGb: 80, storage: my-lvm }
+          - { disk: ide1, sizeGb: 80, storage: my-zfs }
+      - name: controlPlaneAdditionalVolumes
+        value:
+          - { disk: virtio1, sizeGb: 80, storage: my-zfs }
+      - name: loadBalancerAdditionalVolumes
+        value:
+          - { disk: sata1, sizeGb: 80, storage: my-nfs, format: qcow2 }
+```
+To use the same storage for all machines of a given type, can specify a `<type>Storage` variable and then omit `storage`
+from the `workerAdditionalVolumes`. Eg for workers:
+```yaml
+kind: Cluster
+metadata:
+  labels:
+    cluster.x-k8s.io/proxmox-cluster-cni: cilium
+  name: capmox-cluster
+spec:
+  topology:
+    class: proxmox-clusterclass-cilium-v0.1.0
+    variables:
+      - name: workerStorage
+        value: my-lvm
+      - name: workerAdditionalVolumes
+        value:
+          - { disk: scsi1, sizeGb: 80 }
+          - { disk: scsi2, sizeGb: 80 }
+```
+
 ## Notes
 
 * Clusters with IPV6 only is supported.

templates/cluster-class-calico.yaml

@@ -166,6 +166,29 @@ spec:
             mode:
               type: string
               enum: ["ipvs","iptables"]
+    - name: templateSelector
+      required: false
+      schema:
+        openAPIV3Schema:
+          description: When set, we use tag-based template selection (and drop sourceNode/templateID).
+          type: object
+          properties:
+            matchTags:
+              type: array
+              items:
+                type: string
+              minItems: 1
+          required:
+            - matchTags
+    - name: localStorage
+      required: false
+      schema:
+        openAPIV3Schema:
+          type: boolean
+          default: false
+          description: >
+            If true, CAPMOX will schedule clones from a template on the same node where the VM is created (works with 
+            templateSelector or explicit sourceNode/templateID). Defaults to false.
     - name: cloneSpec
       required: true
       schema:
@@ -187,8 +210,6 @@ spec:
               properties:
                 controlPlane: &machineSpec
                   type: object
-                  required:
-                  - sourceNode
                   properties:
                     disks:
                       type: object
@@ -512,20 +533,162 @@ spec:
                         path: /etc/kubernetes/admin.conf
                         type: FileOrCreate
                       name: kubeconfig
+  - name: ApplyLocalStorageFlag-ControlPlane
+    definitions:
       - selector:
           apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
           kind: ProxmoxMachineTemplate
           matchResources:
             controlPlane: true
         jsonPatches:
-          - op: replace
+          - op: add
+            path: /spec/template/spec/localStorage
+            valueFrom:
+              template: '{{ .localStorage | default false }}'
+
+  - name: ApplyLocalStorageFlag-Worker
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-worker
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/localStorage
+            valueFrom:
+              template: '{{ .localStorage | default false }}'
+
+  - name: ApplyLocalStorageFlag-LoadBalancer
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-loadbalancer
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/localStorage
+            valueFrom:
+              template: '{{ .localStorage | default false }}'
+
+# (A) Selector mode: if templateSelector is provided, set it and REMOVE sourceNode/templateID
+  - name: TemplateSelectorMode-ControlPlane
+    enabledIf: '{{ if .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            controlPlane: true
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/templateSelector
+            valueFrom:
+              variable: templateSelector
+          - op: remove
+            path: /spec/template/spec/sourceNode
+          - op: remove
+            path: /spec/template/spec/templateID
+  - name: TemplateSelectorMode-Worker
+    enabledIf: '{{ if .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-worker
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/templateSelector
+            valueFrom:
+              variable: templateSelector
+          - op: remove
+            path: /spec/template/spec/sourceNode
+          - op: remove
+            path: /spec/template/spec/templateID
+  - name: TemplateSelectorMode-LoadBalancer
+    enabledIf: '{{ if .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-loadbalancer
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/templateSelector
+            valueFrom:
+              variable: templateSelector
+          - op: remove
+            path: /spec/template/spec/sourceNode
+          - op: remove
+            path: /spec/template/spec/templateID
+
+  # (B) Explicit mode: when NO templateSelector, set sourceNode/templateID from cloneSpec
+  - name: ExplicitMode-ControlPlane
+    enabledIf: '{{ if not .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            controlPlane: true
+        jsonPatches:
+          - op: add
             path: /spec/template/spec/sourceNode
             valueFrom:
               variable: cloneSpec.machineSpec.controlPlane.sourceNode
-          - op: replace
+          - op: add
             path: /spec/template/spec/templateID
             valueFrom:
               variable: cloneSpec.machineSpec.controlPlane.templateID
+  - name: ExplicitMode-Worker
+    enabledIf: '{{ if not .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-worker
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/sourceNode
+            valueFrom:
+              variable: cloneSpec.machineSpec.workerNode.sourceNode
+          - op: add
+            path: /spec/template/spec/templateID
+            valueFrom:
+              variable: cloneSpec.machineSpec.workerNode.templateID
+  - name: ExplicitMode-LoadBalancer
+    enabledIf: '{{ if not .templateSelector }}true{{ end }}'
+    definitions:
+      - selector:
+          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
+          kind: ProxmoxMachineTemplate
+          matchResources:
+            machineDeploymentClass:
+              names:
+                - proxmox-loadbalancer
+        jsonPatches:
+          - op: add
+            path: /spec/template/spec/sourceNode
+            valueFrom:
+              variable: cloneSpec.machineSpec.loadBalancer.sourceNode
+          - op: add
+            path: /spec/template/spec/templateID
+            valueFrom:
+              variable: cloneSpec.machineSpec.loadBalancer.templateID
   - name: kube-proxy-setup
     description: "kube-proxy configuration"
     enabledIf: "{{ if eq .kubeProxy.mode \"ipvs\" }}true{{ end }}"
@@ -573,23 +736,6 @@ spec:
               template: |
                 - name: root
                   sshAuthorizedKeys: {{ .cloneSpec.sshAuthorizedKeys }}
-      - selector:
-          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
-          kind: ProxmoxMachineTemplate
-          matchResources:
-            controlPlane: false
-            machineDeploymentClass:
-              names:
-              - proxmox-worker
-        jsonPatches:
-          - op: replace
-            path: /spec/template/spec/sourceNode
-            valueFrom:
-              variable: cloneSpec.machineSpec.workerNode.sourceNode
-          - op: replace
-            path: /spec/template/spec/templateID
-            valueFrom:
-              variable: cloneSpec.machineSpec.workerNode.templateID
   - name: LoadBalancerSetup
     description: "Configure LoadBalancer Node Initialisation"
     definitions:
@@ -607,23 +753,6 @@ spec:
               template: |
                 - name: root
                   sshAuthorizedKeys: {{ .cloneSpec.sshAuthorizedKeys }}
-      - selector:
-          apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
-          kind: ProxmoxMachineTemplate
-          matchResources:
-            controlPlane: false
-            machineDeploymentClass:
-              names:
-              - proxmox-loadbalancer
-        jsonPatches:
-          - op: replace
-            path: /spec/template/spec/sourceNode
-            valueFrom:
-              variable: cloneSpec.machineSpec.loadBalancer.sourceNode
-          - op: replace
-            path: /spec/template/spec/templateID
-            valueFrom:
-              variable: cloneSpec.machineSpec.loadBalancer.templateID
   - name: ControlPlaneNodeSockets
     description: "Configure Control Plane Sockets"
     enabledIf: "{{ if .cloneSpec.machineSpecs.controlPlane.numSockets }}true{{ end }}"