diff --git a/deploy/helm/DEVELOPER.md b/deploy/helm/DEVELOPER.md new file mode 100644 index 0000000..c7cc1ad --- /dev/null +++ b/deploy/helm/DEVELOPER.md @@ -0,0 +1,138 @@ +# Developer Guide: Building, Packaging, and Publishing Helm Charts + +This guide helps you build a new Helm chart or update an existing chart for configuration changes. See `README.md` for user installation instructions. + +--- + +## Prerequisites +- Helm installed (`helm version`) +- Git access to this repository +- (Optional) GitHub Pages enabled for publishing the Helm repo index + +--- + +## Create a New Helm Chart +```sh +helm create +``` +Then customize: +- Chart.yaml - name, version (chart version), appVersion (driver version) +- values.yaml - defaults +- templates/ - manifests + +Lint & test: +```sh +helm lint +helm install --dry-run --debug +``` + +--- + +## Update an Existing Chart +```sh +git pull +# Edit values/templates/Chart.yaml as needed +helm lint +helm upgrade --dry-run --debug +``` + +Versioning rules (semantic): +- Bump Chart.yaml:version each release (e.g., 1.2.8 -> 1.2.9) +- Update appVersion when the CSI plugin version changes + +--- + +## Package a New Version + +Example layout: +``` +deploy/helm/repo/ +├─ v1.2.7/ # chart root +├─ index.yaml # Helm repo index (published) +└─ v1.2.7/hammerspace-csi-1.2.7.tgz # packaged chart(s) +``` + +1) Package the chart +```sh +cd deploy/helm/repo/v1.2.7 +helm package ./hammerspace-helm-chart +# Produces hammerspace-csi-.tgz +``` + +2) Update the repo index +```sh +cd .. +# Replace with the base URL of your Helm repo hosting +helm repo index . --url +``` + +3) Commit and push the artifacts +```sh +git add *.tgz index.yaml +git commit -m "chore(helm): release csi-driver " +git push +``` + +### If hosted on GitHub Pages +- Publish index.yaml and all *.tgz to your Pages branch (commonly gh-pages) or /docs folder on main +- Ensure Pages serves the exact directory containing index.yaml +- Optional: add artifacthub-repo.yml next to index.yaml for Artifact Hub + +Minimal artifacthub-repo.yml: +```yaml +repositoryID: "hammerspace-csi-helm-repo" +owners: + - name: "Your Name" + email: "you@example.com" +``` + +--- + +## Local Smoke Test from the Built Repo +After you publish the new package + index.yaml: +```sh +helm repo remove hscsi || true +helm repo add hscsi +helm repo update +helm search repo hscsi +helm install test-hscsi hscsi/csi-driver --dry-run --debug +``` + +--- + +## Optional: Makefile helpers +Create a Makefile in deploy/helm/repo: + +```make +CHART_DIR ?= csi-driver +REPO_URL ?= + +package: +\tcd $(CHART_DIR) && helm package . + +index: +\thelm repo index . --url $(REPO_URL) + +release: package index +\tgit add *.tgz index.yaml +\tgit commit -m "chore(helm): release $$(date +%Y.%m.%d-%H%M)" +\tgit push +``` + +Usage: +```sh +make package +make index +make release +``` + +--- + +## Release Checklist +- [ ] Chart.yaml version bumped +- [ ] appVersion set to the intended CSI plugin image tag +- [ ] helm lint passes +- [ ] helm install --dry-run --debug passes +- [ ] New .tgz generated and committed +- [ ] index.yaml updated with correct --url +- [ ] GitHub Pages serves the folder containing index.yaml diff --git a/deploy/helm/README.md b/deploy/helm/README.md new file mode 100644 index 0000000..cf86034 --- /dev/null +++ b/deploy/helm/README.md @@ -0,0 +1,100 @@ +# Hammerspace CSI Helm Charts + +This repository provides Helm charts to deploy the **Hammerspace CSI driver** in a Kubernetes cluster. +The chart installs both the **Controller** and **Node** plugins, and supports configuration of timeouts, retry intervals. + +--- + +## 🚀 Quickstart: Deploy the Chart + +### 1) Add the Helm repository +> Replace `` with your GitHub Pages URL where `index.yaml` is hosted (e.g., `https://hammer-space/csi-plugin.github.io/deploy/helm/`). +```bash +helm repo add hscsi https://hammer-space/csi-plugin.github.io/deploy/helm/ +helm repo update +helm install my-hammerspace-csi hscsi/hammerspace-csi --version 1.2.8 +``` + +### 2) Create your `values.yaml` +You can override default settings by creating a custom `values.yaml`. Example: + +```yaml +# values.yaml (example) + +controller: + replicaCount: 2 + resources: + requests: + cpu: "200m" + memory: "512Mi" + limits: + cpu: "1" + memory: "1Gi" + +node: + # Tolerate control-plane nodes if needed + tolerations: + - key: "node-role.kubernetes.io/control-plane" + operator: "Exists" + effect: "NoSchedule" + +``` + +### 3) Install the chart +```bash +helm install hscsi hscsi/hammerspace-csi \ + --namespace kube-system \ + --create-namespace \ + -f values.yaml +``` + +Alternatively, override values inline: +```bash +helm install hscsi hscsi/hammerspace-csi \ + --namespace kube-system \ + --create-namespace \ + --set controller.replicaCount=2 +``` + +### 4) Verify the deployment +```bash +kubectl get pods -n kube-system -l app.kubernetes.io/name=hscsi +kubectl get csidrivers | grep hammerspace || true +``` + +You should see both **csi-provisioner-0** and **csi-node-** plugin pods running. +--- + +## 🔧 Upgrade +```bash +helm upgrade hscsi -n kube-system -f values.yaml +``` + +## 🗑 Uninstall +```bash +helm uninstall hscsi -n kube-system +``` + +--- + +## ⚙️ Configuration Reference (common) +| Key | Type | Description | +| --- | --- | --- | +| `controller.replicaCount` | int | Number of controller replicas | +| `controller.resources` | map | Requests/limits for controller pods | +| `node.tolerations` | list | Tolerations for node daemonset | + +> For all options, see `values.yaml` in the chart. + +--- + +## 🧰 Troubleshooting +- Pods stuck in ImagePullBackOff: Verify image registry access and tag in the chart values. +- "csi-driver" not found: Run `helm repo update` and check that your `/index.yaml` is accessible. +- No nodes provisioned: Check node plugin DaemonSet tolerations/affinity and that nodes can reach the NFS endpoints. + +--- + +## 📚 References +- Helm: https://helm.sh/docs/ +- Kubernetes CSI: https://kubernetes-csi.github.io/docs/ diff --git a/deploy/helm/repo/artifacthub-repo.yaml b/deploy/helm/repo/artifacthub-repo.yaml new file mode 100644 index 0000000..9c8a793 --- /dev/null +++ b/deploy/helm/repo/artifacthub-repo.yaml @@ -0,0 +1,5 @@ +repositoryID: ba19dd93-de8e-41da-b66c-4f5c557ee0f3 # use `uuidgen` to generate +name: hammerspace-csi-chart +displayName: Hammerspace CSI Helm Chart +description: Helm chart for deploying the Hammerspace CSI plugin +url: https://github.com/hammer-space/csi-plugin/deploy/helm/repo \ No newline at end of file diff --git a/deploy/helm/repo/index.yaml b/deploy/helm/repo/index.yaml new file mode 100644 index 0000000..8e3068c --- /dev/null +++ b/deploy/helm/repo/index.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +entries: + hammerspace-csi: + - apiVersion: v2 + appVersion: v1.2.8 + created: "2025-08-26T06:46:19.212057988Z" + description: A Helm chart for the Hammerspace CSI driver + digest: b7f41604ea1d1b9b35bc183c5d5f1b8fd0d771a1887565b8a348982824fae058 + icon: https://www.hammerspace.com/favicon.ico + name: hammerspace-csi + urls: + - v1.2.8/hammerspace-csi-1.2.8.tgz + version: 1.2.8 + - apiVersion: v2 + appVersion: v1.2.7 + created: "2025-08-26T06:46:19.211443051Z" + description: A Helm chart for the Hammerspace CSI driver + digest: 1db738079a744d44fb7e25c70636568d4c136000b564b4845666071ef901b58c + icon: https://www.hammerspace.com/favicon.ico + name: hammerspace-csi + urls: + - v1.2.7/hammerspace-csi-1.2.7.tgz + version: 1.2.7 +generated: "2025-08-26T06:46:19.210659997Z" diff --git a/deploy/helm/repo/v1.2.7/hammerspace-csi-1.2.7.tgz b/deploy/helm/repo/v1.2.7/hammerspace-csi-1.2.7.tgz new file mode 100644 index 0000000..32e2a74 Binary files /dev/null and b/deploy/helm/repo/v1.2.7/hammerspace-csi-1.2.7.tgz differ diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/Chart.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/Chart.yaml new file mode 100644 index 0000000..1d96391 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: hammerspace-csi +description: A Helm chart for the Hammerspace CSI driver +version: 1.2.7 +appVersion: "v1.2.7" +icon: https://www.hammerspace.com/favicon.ico diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/configmap.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/configmap.yaml new file mode 100644 index 0000000..218ddd5 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/configmap.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-env-config + namespace: {{ .Values.namespace | default "kube-system" }} +data: + MOUNT_CHECK_TIMEOUT: "{{ .Values.env.MOUNT_CHECK_TIMEOUT }}" + UNMOUNT_RETRY_COUNT: "{{ .Values.env.UNMOUNT_RETRY_COUNT }}" + UNMOUNT_RETRY_INTERVAL: "{{ .Values.env.UNMOUNT_RETRY_INTERVAL }}" \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/rbac.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/rbac.yaml new file mode 100644 index 0000000..d953e9a --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/rbac.yaml @@ -0,0 +1,94 @@ +# This file is part of the Hammerspace CSI Driver project. +# It defines the RBAC roles and bindings for the CSI provisioner component. +# The provisioner is responsible for creating and managing PersistentVolumes +# based on PersistentVolumeClaims in Kubernetes. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} +--- +# This ClusterRole defines the permissions required by the CSI provisioner. +# It allows the provisioner to manage PersistentVolumes, PersistentVolumeClaims, +# and other related resources in the Kubernetes cluster. +# It also allows the provisioner to interact with storage classes and volume snapshots. +# The ClusterRole is bound to the csi-provisioner ServiceAccount. +# This ClusterRoleBinding binds the csi-provisioner ServiceAccount to the csi-provisioner ClusterRole. +# This allows the provisioner to perform the actions defined in the ClusterRole. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# The ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachments. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachment status. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# The ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: csi-provisioner +rules: + - apiGroups: [""] + resources: ["pods", "persistentvolumes", "persistentvolumeclaims", "nodes", "events", "endpoints", "secrets"] + verbs: ["list", "watch", "get", "create", "delete", "update", "patch"] + - apiGroups: ["storage.k8s.io", "snapshot.storage.k8s.io", "apiextensions.k8s.io"] + resources: ["storageclasses", "volumeattachments", "volumeattachments/status", "volumesnapshotcontents/status", "volumesnapshots", "volumesnapshotcontents", "volumesnapshotclasses", "customresourcedefinitions"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +--- +# This ClusterRoleBinding binds the csi-provisioner ServiceAccount to the csi-provisioner ClusterRole. +# It allows the provisioner to perform the actions defined in the ClusterRole. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachments. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachment status. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-provisioner-binding +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io +--- +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachment status. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-provisioner +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/service.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/service.yaml new file mode 100644 index 0000000..4ecc4f5 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/service.yaml @@ -0,0 +1,24 @@ +# This file is part of the Hammerspace CSI Driver Helm Chart. +# It defines the service for the CSI provisioner component. +# The service is a headless service that allows the StatefulSet to manage the pods. +kind: Service +apiVersion: v1 +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} + labels: + app: csi-provisioner +spec: + type: ClusterIP + clusterIP: None + # This is needed for the StatefulSet to work properly + # as it uses a headless service to manage the pods. + # The StatefulSet will create a DNS entry for the pods + # in the format ...svc.cluster.local + # where is the name of the pod, is the name of the service, + # and is the namespace of the service. + # This allows the pods to communicate with each other using DNS. + # The StatefulSet will also create a DNS entry for the service in the format ..svc.cluster.local + # which allows the pods to communicate with the service + selector: + app: csi-provisioner \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/statefulset.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/statefulset.yaml new file mode 100644 index 0000000..56e8618 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/controller/statefulset.yaml @@ -0,0 +1,113 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} +spec: + serviceName: "csi-provisioner" + replicas: {{ .Values.controller.replicas }} + selector: + matchLabels: + app: csi-provisioner + template: + metadata: + labels: + app: csi-provisioner + spec: + serviceAccountName: csi-provisioner + hostNetwork: true + containers: + - name: csi-provisioner + imagePullPolicy: Always + image: {{ .Values.image.provisioner }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--timeout=60s" # Recommended as shares may take some time to create + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-attacher + imagePullPolicy: Always + image: {{ .Values.image.attacher }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-snapshotter + imagePullPolicy: Always + image: {{ .Values.image.snapshotter }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-resizer + imagePullPolicy: Always + image: {{ .Values.image.resizer }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: hs-csi-plugin-controller + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: {{ .Values.image.csiPlugin }} + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + emptyDir: {} + - name: staging-dir + hostPath: + path: /tmp \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/csidriver.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/csidriver.yaml new file mode 100644 index 0000000..65229a0 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/csidriver.yaml @@ -0,0 +1,10 @@ +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +metadata: + name: com.hammerspace.csi +spec: + podInfoOnMount: true + requiresRepublish: true + volumeLifecycleModes: + - Persistent + storageCapacity: true \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/daemonset.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/daemonset.yaml new file mode 100644 index 0000000..904159b --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/daemonset.yaml @@ -0,0 +1,125 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: csi-node + namespace: {{ .Values.namespace }} +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + containers: + - name: csi-resizer + imagePullPolicy: Always + image: {{ .Values.image.resizer }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: driver-registrar + imagePullPolicy: Always + image: {{ .Values.image.registrar }} + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", '[ -n "$REG_SOCKET" ] && rm -rf "$REG_SOCKET" || echo "REG_SOCKET not set, skipping delete"'] + args: + - "--v=5" + - "--csi-address=$(CSI_ENDPOINT)" + - "--kubelet-registration-path=$(REG_SOCKET)" + securityContext: + privileged: true + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: REG_SOCKET + value: /var/lib/kubelet/plugins_registry/com.hammerspace.csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + - name: hs-csi-plugin-node + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: {{ .Values.image.csiPlugin }} + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: dev-dir + mountPath: /dev + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + - name: dev-dir + hostPath: + path: /dev + - name: staging-dir + hostPath: + path: /tmp \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/rbac.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/rbac.yaml new file mode 100644 index 0000000..bad30fe --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/templates/node/rbac.yaml @@ -0,0 +1,43 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-node + namespace: {{ .Values.namespace }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: csi-node +rules: + - apiGroups: [""] + resources: ["pods", "secrets", "nodes", "namespaces", "events", "persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status"] + verbs: ["get", "list", "create", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "create", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-node +subjects: + - kind: ServiceAccount + name: csi-node + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-node + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-provisioner-volumeattachment-status-binding +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io diff --git a/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/values.yaml b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/values.yaml new file mode 100644 index 0000000..454ca68 --- /dev/null +++ b/deploy/helm/repo/v1.2.7/hammerspace-helm-chart/values.yaml @@ -0,0 +1,21 @@ +namespace: kube-system + +credentialsSecretName: com.hammerspace.csi.credentials +# +env: + MOUNT_CHECK_TIMEOUT: "30s" + UNMOUNT_RETRY_COUNT: "5" + UNMOUNT_RETRY_INTERVAL: "1s" +# +image: + csiPlugin: hammerspaceinc/csi-plugin:v1.2.7 + provisioner: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + attacher: registry.k8s.io/sig-storage/csi-attacher:v4.4.0 + snapshotter: registry.k8s.io/sig-storage/csi-snapshotter:v8.1.0 + resizer: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + registrar: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 +# +controller: + replicas: 1 + + diff --git a/deploy/helm/repo/v1.2.8/hammerspace-csi-1.2.8.tgz b/deploy/helm/repo/v1.2.8/hammerspace-csi-1.2.8.tgz new file mode 100644 index 0000000..36cd7ef Binary files /dev/null and b/deploy/helm/repo/v1.2.8/hammerspace-csi-1.2.8.tgz differ diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/Chart.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/Chart.yaml new file mode 100644 index 0000000..f4c4770 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: hammerspace-csi +description: A Helm chart for the Hammerspace CSI driver +version: 1.2.8 +appVersion: "v1.2.8" +icon: https://www.hammerspace.com/favicon.ico diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/configmap.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/configmap.yaml new file mode 100644 index 0000000..218ddd5 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/configmap.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-env-config + namespace: {{ .Values.namespace | default "kube-system" }} +data: + MOUNT_CHECK_TIMEOUT: "{{ .Values.env.MOUNT_CHECK_TIMEOUT }}" + UNMOUNT_RETRY_COUNT: "{{ .Values.env.UNMOUNT_RETRY_COUNT }}" + UNMOUNT_RETRY_INTERVAL: "{{ .Values.env.UNMOUNT_RETRY_INTERVAL }}" \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/rbac.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/rbac.yaml new file mode 100644 index 0000000..995d38d --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/rbac.yaml @@ -0,0 +1,65 @@ +# This file is part of the Hammerspace CSI Driver project. +# It defines the RBAC roles and bindings for the CSI provisioner component. +# The provisioner is responsible for creating and managing PersistentVolumes +# based on PersistentVolumeClaims in Kubernetes. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} +--- +# This ClusterRole defines the permissions required by the CSI provisioner. +# It allows the provisioner to manage PersistentVolumes, PersistentVolumeClaims, +# and other related resources in the Kubernetes cluster. +# It also allows the provisioner to interact with storage classes and volume snapshots. +# The ClusterRole is bound to the csi-provisioner ServiceAccount. +# This ClusterRoleBinding binds the csi-provisioner ServiceAccount to the csi-provisioner ClusterRole. +# This allows the provisioner to perform the actions defined in the ClusterRole. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# The ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachments. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachment status. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# The ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. + +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: csi-provisioner +rules: + - apiGroups: [""] + resources: ["pods", "persistentvolumes", "persistentvolumeclaims", "nodes", "events", "endpoints", "secrets"] + verbs: ["list", "watch", "get", "create", "delete", "update", "patch"] + - apiGroups: ["storage.k8s.io", "snapshot.storage.k8s.io", "apiextensions.k8s.io"] + resources: ["storageclasses", "volumeattachments", "volumeattachments/status", "volumesnapshotcontents/status", "volumesnapshots", "volumesnapshotcontents", "volumesnapshotclasses", "customresourcedefinitions", "csinodes"] + verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] +--- +# This ClusterRoleBinding is specifically for the provisioner to manage volume attachment status. +# It allows the provisioner to update the status of volume attachments. +# This is necessary for the provisioner to properly manage the lifecycle of volumes +# and ensure that they are correctly attached to nodes. +# The ClusterRoleBinding is created in the same namespace as the provisioner ServiceAccount. +# This ClusterRoleBinding is necessary for the provisioner to have the required permissions +# to manage storage resources in the cluster. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-provisioner +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/service.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/service.yaml new file mode 100644 index 0000000..4ecc4f5 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/service.yaml @@ -0,0 +1,24 @@ +# This file is part of the Hammerspace CSI Driver Helm Chart. +# It defines the service for the CSI provisioner component. +# The service is a headless service that allows the StatefulSet to manage the pods. +kind: Service +apiVersion: v1 +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} + labels: + app: csi-provisioner +spec: + type: ClusterIP + clusterIP: None + # This is needed for the StatefulSet to work properly + # as it uses a headless service to manage the pods. + # The StatefulSet will create a DNS entry for the pods + # in the format ...svc.cluster.local + # where is the name of the pod, is the name of the service, + # and is the namespace of the service. + # This allows the pods to communicate with each other using DNS. + # The StatefulSet will also create a DNS entry for the service in the format ..svc.cluster.local + # which allows the pods to communicate with the service + selector: + app: csi-provisioner \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/statefulset.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/statefulset.yaml new file mode 100644 index 0000000..56e8618 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/controller/statefulset.yaml @@ -0,0 +1,113 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: csi-provisioner + namespace: {{ .Values.namespace }} +spec: + serviceName: "csi-provisioner" + replicas: {{ .Values.controller.replicas }} + selector: + matchLabels: + app: csi-provisioner + template: + metadata: + labels: + app: csi-provisioner + spec: + serviceAccountName: csi-provisioner + hostNetwork: true + containers: + - name: csi-provisioner + imagePullPolicy: Always + image: {{ .Values.image.provisioner }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--timeout=60s" # Recommended as shares may take some time to create + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-attacher + imagePullPolicy: Always + image: {{ .Values.image.attacher }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-snapshotter + imagePullPolicy: Always + image: {{ .Values.image.snapshotter }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-resizer + imagePullPolicy: Always + image: {{ .Values.image.resizer }} + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: hs-csi-plugin-controller + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: {{ .Values.image.csiPlugin }} + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + emptyDir: {} + - name: staging-dir + hostPath: + path: /tmp \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/csidriver.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/csidriver.yaml new file mode 100644 index 0000000..65229a0 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/csidriver.yaml @@ -0,0 +1,10 @@ +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +metadata: + name: com.hammerspace.csi +spec: + podInfoOnMount: true + requiresRepublish: true + volumeLifecycleModes: + - Persistent + storageCapacity: true \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/daemonset.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/daemonset.yaml new file mode 100644 index 0000000..72a1831 --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/daemonset.yaml @@ -0,0 +1,120 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: csi-node + namespace: {{ .Values.namespace }} +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + containers: + - name: driver-registrar + imagePullPolicy: Always + image: {{ .Values.image.registrar }} + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", '[ -n "$REG_SOCKET" ] && rm -rf "$REG_SOCKET" || echo "REG_SOCKET not set, skipping delete"'] + args: + - "--v=5" + - "--csi-address=$(CSI_ENDPOINT)" + - "--kubelet-registration-path=$(REG_SOCKET)" + securityContext: + privileged: true + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: REG_SOCKET + value: /var/lib/kubelet/plugins_registry/com.hammerspace.csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + - name: hs-csi-plugin-node + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: {{ .Values.image.csiPlugin }} + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: dev-dir + mountPath: /dev + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + - name: rootshare-dir # <-new with 1.2.8 + mountPath: /var/lib/hammerspace/ + mountPropagation: Bidirectional + + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + - name: dev-dir + hostPath: + path: /dev + - name: staging-dir + hostPath: + path: /tmp + - name: rootshare-dir # <-new with 1.2.8 + hostPath: + path: /var/lib/hammerspace/ \ No newline at end of file diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/rbac.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/rbac.yaml new file mode 100644 index 0000000..7fda37e --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/templates/node/rbac.yaml @@ -0,0 +1,31 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-node + namespace: {{ .Values.namespace }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: csi-node +rules: + - apiGroups: [""] + resources: ["pods", "secrets", "nodes", "namespaces", "events", "persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status"] + verbs: ["get", "list", "create", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status", "csinodes"] + verbs: ["get", "list", "watch", "create", "update", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: csi-node +subjects: + - kind: ServiceAccount + name: csi-node + namespace: {{ .Values.namespace }} +roleRef: + kind: ClusterRole + name: csi-node + apiGroup: rbac.authorization.k8s.io + diff --git a/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/values.yaml b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/values.yaml new file mode 100644 index 0000000..c6df9cf --- /dev/null +++ b/deploy/helm/repo/v1.2.8/hammerspace-helm-chart/values.yaml @@ -0,0 +1,21 @@ +namespace: kube-system + +credentialsSecretName: com.hammerspace.csi.credentials +# +env: + MOUNT_CHECK_TIMEOUT: "30s" + UNMOUNT_RETRY_COUNT: "5" + UNMOUNT_RETRY_INTERVAL: "1s" +# +image: + csiPlugin: hammerspaceinc/csi-plugin:v1.2.8-rc2 + provisioner: registry.k8s.io/sig-storage/csi-provisioner:v5.2.0 + attacher: registry.k8s.io/sig-storage/csi-attacher:v4.8.0 + snapshotter: registry.k8s.io/sig-storage/csi-snapshotter:v8.0.2 + resizer: registry.k8s.io/sig-storage/csi-resizer:v1.14.0 + registrar: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0 +# +controller: + replicas: 1 + + diff --git a/deploy/kubernetes/kubernetes-1.28/plugin.yaml b/deploy/kubernetes/kubernetes-1.28/plugin.yaml index af6d44b..80df4ac 100644 --- a/deploy/kubernetes/kubernetes-1.28/plugin.yaml +++ b/deploy/kubernetes/kubernetes-1.28/plugin.yaml @@ -56,7 +56,7 @@ spec: containers: - name: csi-provisioner imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + image: registry.k8s.io/sig-storage/csi-provisioner:v5.2.0 args: - "--csi-address=$(CSI_ENDPOINT)" - "--timeout=60s" # Recommended as shares may take some time to create @@ -69,7 +69,7 @@ spec: mountPath: /var/lib/csi/ - name: csi-attacher imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0 + image: registry.k8s.io/sig-storage/csi-attacher:v4.8.0 args: - "--csi-address=$(CSI_ENDPOINT)" - "--v=5" @@ -81,7 +81,7 @@ spec: mountPath: /var/lib/csi/ - name: csi-snapshotter imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1 + image: registry.k8s.io/sig-storage/csi-snapshotter:v8.0.2 args: - "--csi-address=$(CSI_ENDPOINT)" - "--v=5" @@ -93,7 +93,7 @@ spec: mountPath: /var/lib/csi/ - name: csi-resizer imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + image: registry.k8s.io/sig-storage/csi-resizer:v1.14.0 args: - "--csi-address=$(CSI_ENDPOINT)" - "--v=5" @@ -171,7 +171,7 @@ rules: verbs: ["get", "patch", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] - verbs: ["get", "list", "watch"] + verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] @@ -195,13 +195,13 @@ rules: verbs: ["get", "list", "watch", "update"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] - verbs: ["create", "list", "watch", "delete"] + verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments", "volumeattachments/status"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: ["storage.k8s.io"] - resources: ["storageclasses"] - verbs: ["get", "list", "watch", "update"] + resources: ["csinodes"] + verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 @@ -235,21 +235,10 @@ spec: serviceAccount: csi-node hostNetwork: true containers: - - name: csi-resizer - imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 - args: - - "--csi-address=$(CSI_ENDPOINT)" - - "--v=5" - env: - - name: CSI_ENDPOINT - value: /csi/csi.sock - volumeMounts: - - name: socket-dir - mountPath: /csi + # The node driver registar sidecar - name: driver-registrar imagePullPolicy: Always - image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 + image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0 lifecycle: preStop: exec: @@ -378,7 +367,7 @@ rules: resources: ["persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status","events"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: ["storage.k8s.io"] - resources: ["volumeattachments", "volumeattachments/status"] + resources: ["volumeattachments", "volumeattachments/status", "csinodes"] verbs: ["get", "list", "watch", "update", "patch"] --- @@ -397,20 +386,6 @@ roleRef: apiGroup: rbac.authorization.k8s.io --- -kind: ClusterRoleBinding -apiVersion: rbac.authorization.k8s.io/v1 -metadata: - name: csi-provisioner-volumeattachment-status-binding - namespace: kube-system -subjects: - - kind: ServiceAccount - name: csi-provisioner - namespace: kube-system -roleRef: - kind: ClusterRole - name: csi-provisioner-volumeattachment-status - apiGroup: rbac.authorization.k8s.io ---- apiVersion: v1 kind: ConfigMap metadata: diff --git a/docs/hammerspace_csi_upgrade_guide_v_1.2.8.md b/docs/hammerspace_csi_upgrade_guide_v_1.2.8.md new file mode 100644 index 0000000..048d694 --- /dev/null +++ b/docs/hammerspace_csi_upgrade_guide_v_1.2.8.md @@ -0,0 +1,285 @@ +# Hammerspace CSI Driver Upgrade Guide — v1.2.8 + +**Audience:** Cluster operators managing large Kubernetes clusters (400+ nodes). + +**Purpose:** Safe, repeatable process to test and roll out the Hammerspace CSI `v1.2.8` node plugin (bind-mount root-share change) across large clusters without mass manual restaging. + +--- + +## 1. Executive summary + +- v1.2.8 changes staging/publish behavior: the driver uses a **root share** mounted once per node and then **bind-mounts** per-volume subdirectories into pod targets. +- Old drivers (<= v1.2.7) mounted volumes directly (NFS direct mounts) and did not create the root share mount. +- After upgrading nodes, *existing volumes* on nodes previously staged with v1.2.7 may skip `NodeStageVolume` and only call `NodePublishVolume`, so the root-share may not exist on those nodes unless a lazy-stage is performed during publish. + +**Goal of this guide:** provide a safe single-node test and a staged cluster rollout with validation and rollback steps. Includes diagrams, commands, and the `csi-node` DaemonSet snippet. + +--- + +## 2. Diagrams (conceptual) + +### Old flow (v1.2.7 and earlier) + +``` +[IP:/pvc-XYZ mounted directly] --> /var/lib/kubelet/pods//volumes/.../mount (NFS direct) +``` + +### New flow (v1.2.8) + +``` +IP:/ --> /var/lib/hammerspace/rootmount (1 root NFS mount per node) + | + +--> bind mount /var/lib/hammerspace/rootmount/pvc-XYZ/ --> /var/lib/kubelet/pods//volumes/.../mount +``` + +**Key migration point:** If a node had direct NFS mounts from old driver, the root share directory will not exist until either `NodeStageVolume` runs or a lazy-stage is performed by `NodePublishVolume`. For large clusters we recommend lazy-stage logic inside `NodePublishVolume` to automatically mount root share and convert old direct NFS mounts to bind mounts. + +--- + +## 3. Phase 1 — Single-node testing (safe & reversible) + +**Pick a node:** + +- Choose a healthy, non-critical node that currently successfully mounts PVCs. +- Verify with: + +```bash +kubectl get nodes -o wide +kubectl describe node +``` + +**Steps:** + +1. Cordon the node so new pods don't land on it during validation: + +```bash +kubectl cordon +``` + +2. Label the node so DaemonSet can be targeted for the test: + +```bash +kubectl label node csi-test-node=true +``` + +3. Patch the `csi-node` DaemonSet to include the nodeSelector and set the new image. Example patch (JSON patch): + +```bash +kubectl patch daemonset csi-node -n kube-system \ + --type='json' \ + -p='[ + {"op":"add","path":"/spec/template/spec/nodeSelector","value":{"csi-test-node":"true"}}, + {"op":"replace","path":"/spec/template/spec/containers/2/image","value":"hammerspaceinc/csi-plugin:v1.2.8"} + ]' +``` + +*NB:* the `containers/2` index in the patch assumes container order; if you want safer, edit the DaemonSet YAML and change the `hs-csi-plugin-node` image and add `nodeSelector` under the template. + +4. Wait for the DaemonSet to schedule the updated pod onto the labeled node and verify the pod is running: + +```bash +kubectl get pods -n kube-system -o wide -l app=csi-node +kubectl logs -n kube-system -c hs-csi-plugin-node --tail=200 +``` + +5. Run a test Pod that mounts an existing PVC (or create a temporary PVC + Pod): + +```bash +kubectl run csi-test --image=nginx --restart=Never \ + --overrides='{"apiVersion":"v1","spec":{"volumes":[{"name":"test-vol","persistentVolumeClaim":{"claimName":""}}],"containers":[{"name":"nginx","image":"nginx","volumeMounts":[{"mountPath":"/data","name":"test-vol"}]}]}}' + +kubectl exec -it csi-test -- mount | grep /data +``` + +6. Validate: + +- Confirm the root share exists on the node (`ls /var/lib/hammerspace/rootmount`). +- Confirm the pod target path is a **bind mount** whose source path is under `/var/lib/hammerspace/rootmount/`. + +Example on node: `mount | grep hammerspace` or inside the pod: `mount | grep /data`. + +7. Roll back test targeting (remove nodeSelector and unlabel): + +```bash +kubectl patch daemonset csi-node -n kube-system --type='json' -p='[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]' +kubectl label node csi-test-node- +kubectl uncordon +``` + +--- + +## 4. Phase 2 — Cluster-wide rollout (recommended: staged) + +**Plan:** Rolling update in batches, e.g., `maxUnavailable: 10%` in DaemonSet rollingUpdate. With 400 nodes, this means \~40 nodes at a time. + +**Pre-checks:** + +- Ensure cluster-wide backups of manifests for quick rollback. +- Ensure kubelet/node health and that `csi-node` DaemonSet can tolerate brief restarts. + +**Step-by-step:** + +1. Backup current DaemonSet and controller manifests: + +```bash +kubectl get daemonset csi-node -n kube-system -o yaml > csi-node-backup.yaml +kubectl get statefulset csi-provisioner -n kube-system -o yaml > csi-provisioner-backup.yaml +``` + +2. Edit the DaemonSet update strategy: + +```yaml +spec: + updateStrategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 10% +``` + +3. Apply new image for the node plugin (and controller if needed): + +```bash +kubectl set image daemonset/csi-node -n kube-system hs-csi-plugin-node=hammerspaceinc/csi-plugin:v1.2.8 +kubectl set image statefulset/csi-provisioner -n kube-system csi-provisioner +``` + +4. Monitor rollout and logs: + +```bash +kubectl rollout status daemonset/csi-node -n kube-system +kubectl get pods -n kube-system -l app=csi-node -o wide +kubectl logs -n kube-system -l app=csi-node --tail=200 +``` + +5. Spot-check mounts on upgraded nodes: + +Choose a few nodes that report `Ready` with updated pods and run: + +```bash +kubectl debug node/ -n kube-system --image=registry.k8s.io/pause +# or ssh to node +mount | grep hammerspace +# check pod mounts +kubectl get pods -o wide --field-selector spec.nodeName= +kubectl exec -it -- mount | grep /var/lib/kubelet/pods +``` + +6. Validate PVCs after rollout: + +```bash +kubectl get pvc --all-namespaces +kubectl describe pvc -n +# confirm pods on upgraded nodes can access volumes and see bind mounts from /var/lib/hammerspace/rootmount +``` + +--- + +## 5. Validation & health checks (commands) + +### List PVCs and their bound PVs + +```bash +kubectl get pvc --all-namespaces -o wide +``` + +### Check mount points on a node (SSH or debug pod) + +```bash +mount | grep hammerspace +findmnt --target /var/lib/hammerspace/rootmount +``` + +### Inspect a pod's volume mount inside the pod + +```bash +kubectl exec -it -- mount | grep /data +``` + +### Count nodes with rootshare mounted + +```bash +# run as a DaemonSet job or remote SSH script to check per-node +# example: use kubectl debug or ssh +``` + +--- + +## 6. Troubleshooting + +- **Old mounts not replaced / bind mount failing**: ensure `NodePublishVolume` lazy-stage logic runs — check node logs for `[LazyStage]` entries. +- **ESTALE errors**: may appear on stale NFS mounts. Rebooting kubelet or remounting the underlying NFS may help. Prefer to unmount the *old direct mount at the pod target path* before bind-mounting; do NOT unmount the root share if other pods depend on it. +- **If many nodes show errors**: pause rollout, revert to backup DaemonSet manifest, and investigate logs. + +--- + +## 7. Rollback + +1. Revert the `csi-node` image to the backed-up image: + +```bash +kubectl set image daemonset/csi-node -n kube-system hs-csi-plugin-node= +``` + +2. Monitor rollout and confirm volumes become accessible again. + +--- + +## 8. Appendix — csi-node DaemonSet snippet (example) + +```yaml +# (snippet) Example csi-node DaemonSet for v1.2.8 +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: csi-node + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + containers: + - name: hs-csi-plugin-node + image: hammerspaceinc/csi-plugin:v1.2.8 + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: rootshare-dir # new added with 1.2.8 + mountPath: /var/lib/hammerspace/ + mountPropagation: Bidirectional + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: rootshare-dir # new added with 1.2.8 + hostPath: + path: /var/lib/hammerspace/ + - name: staging-dir + hostPath: + path: /tmp +```