Helm upgrade states secret patching during upgrade but its behavior is not matching with behavior of kubectl patch command

[rohitsharma382]

1. Create sample helm chart having one secret template like below:

$ cat nginx/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: data-test-secret
  labels:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
  annotations:
    test.com/product-name: "Test"
type: Opaque
data:
  testkey: "dGVzdAo="

2. Run helm install command to deploy secret

$ helm install nginxrs nginx

3. After helm install observe secret data.

$ kubectl get secret data-test-secret -o yaml
apiVersion: v1
data:
  testkey: dGVzdAo=
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: nginxrs
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-03-15T09:12:39Z"
  labels:
    app.kubernetes.io/instance: nginxrs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-test-secret
  namespace: test-system
  resourceVersion: "595791560"
  uid: 60a740ab-6e91-49c0-8a00-8ccabf36c18e
type: Opaque
$

4. Now create new template of secret with new data in helm chart.

$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: data-test-secret
  labels:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
  annotations:
    test.com/product-name: "Test"
type: Opaque
data:
  newtestkey: "dGVzdAo="

5. Now perform helm upgrade and observe helm is confirming that it is doing Secret patching.

[Output truncated ]

$ helm upgrade nginxrs nginx --debug

upgrade.go:153: [debug] preparing upgrade for nginxrs
upgrade.go:161: [debug] performing update for nginxrs
upgrade.go:354: [debug] creating upgraded release for nginxrs
client.go:393: [debug] checking 6 resources for changes
client.go:684: [debug] Looks like there are no changes for ServiceAccount "web-usr"
client.go:693: [debug] **Patch Secret "data-test-secret"** in namespace test-system

6. After upgrade check secret result and observed that old secret data was overrided by new data.

$ kubectl get secret data-test-secret -o yaml
apiVersion: v1
data:
  newtestkey: dGVzdAo=
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: nginxrs
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-03-15T09:17:00Z"
  labels:
    app.kubernetes.io/instance: nginxrs
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-test-secret
  namespace: test-system
  resourceVersion: "595807504"
  uid: b51f2fc7-4eda-4386-8ece-486f0ca07bc8
type: Opaque
$

7. Above behavior of secret patching via helm upgrade is different if we do manually via kubectl patch command.

For eg: In below we can see old key i.e testkey and new key i.e newtestkey both are present.

$ kubectl patch secret data-test-secret --patch-file 2data-test-secrets.yaml
secret/data-test-secret patched
$
$ kubectl get secret data-test-secret -o yaml
apiVersion: v1
data:
  newtestkey: dGVzdAo=
  testkey: dGVzdAo=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"testkey":"dGVzdAo="},"kind":"Secret","metadata":{"annotations":{},"name":"data-test-secret","namespace":"test-system"},"type":"Opaque"}
  creationTimestamp: "2024-03-15T07:49:39Z"
  name: data-test-secret
  namespace: test-system
  resourceVersion: "595595966"
  uid: d2580400-fe78-44b2-bfa8-68672e403eaf
type: Opaque
$

8. However if perform kubectl apply command then old data will be removed and replaced by new data.

For eg: In below we can see old key i.e testkey and new key i.e newtestkey both are present.

$ kubectl apply -f 2data-test-secrets.yaml
secret/data-test-secret configured

$ kubectl get secret -o yaml data-test-secret

apiVersion: v1
data:
  newtestkey: dGVzdAo=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"newtestkey":"dGVzdAo="},"kind":"Secret","metadata":{"annotations":{},"name":"data-test-secret","namespace":"test-system"},"type":"Opaque"}
  creationTimestamp: "2024-03-10T10:22:37Z"
  name: data-test-secret
  namespace: test-system
  resourceVersion: "578975054"
  uid: 1849763a-3296-4fd8-8462-ec411b09a7a3
type: Opaque

Hence, it seems like helm is stating that its doing patching while in actual it seems doing applying which is like wrong information presentation of actual behavior.

[gjenkins8]

Issues regarding Helm usage are best posted over on the helm software repo https://github.com/helm/helm. Please repost there. Thanks!