Inline blob mount producing error message claiming blobfuse2 is not in $PATH on Azure stack Azure Linux node

[sindreal]

Doing a simple inline blob mount test following the example at:
https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/deploy/example/e2e_usage.md#option3-inline-volume

What happened:
Pod won't start, due to the following error mesage:
Warning FailedMount 14s (x35 over 55m) kubelet MountVolume.SetUp failed for volume "volume-inline-blob-1" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

What you expected to happen:
That the pod started with the blob volume mounted.

Anything else we need to know?:
All csi-blob-node-xxxxx nodes seems to have blobfuse2 in their $PATH, and I can successfully mount the blob manually using blobfuse2 from the csi-blob-node. The secret is also present in the same namespace as the pod.

Environment:

• CSI Driver version: 1.25.1
• Kubernetes version (use kubectl version): 1.29.4
• OS (e.g. from /etc/os-release): AKS on Azure Local
• Install tools: Helm
• Others:

[andyzhangx]

@sindreal have you tried other example, e.g. option#1, option#2?
and what is AKS on Azure Local?

[andyzhangx]

I see where is the problem, have you set node.enableBlobfuseProxy=true in helm chart install? this driver is using blobfuse-proxy by default, and this driver would install blobfuse2 on the node by default, you are mounting inside blob csi driver pod, that works since blobfuse2 is already installed by default inside csi driver.
details: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/charts#tips

[sindreal]

The other examples also fails, and AKS on Azure Local is just renamed AKS on Azure Stack, which lacks access to this extension, hence the use of Helm.
I'll try to get node.enabelBlobFuseProxy=true enabled via helm, how may I easily check if it's installed?

[andyzhangx]

pls provide the csi driver logs first: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed, the blobfuse2 and blobfuse-proxy install logs are in install-blobfuse-proxy container

[sindreal]

kubectl logs daemonset/csi-blob-node -c blob -n kube-system
total of 130k lines, so this is just the end.

I0210 14:06:22.017012  372960 utils.go:104] GRPC call: /csi.v1.Node/NodePublishVolume
I0210 14:06:22.017028  372960 utils.go:105] GRPC request: {"target_path":"/var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"containerName":"blob-k8s-test","csi.storage.k8s.io/ephemeral":"true","csi.storage.k8s.io/pod.name":"deployment-fida-kol-1-5b49c84b6d-lqj4q","csi.storage.k8s.io/pod.namespace":"ns-fida-test","csi.storage.k8s.io/pod.uid":"06db8aad-3689-4e41-b4b1-ec591aa72045","csi.storage.k8s.io/serviceAccount.name":"default","csi.storage.k8s.io/serviceAccount.tokens":"***stripped***","mountOptions":"-o allow_other --file-cache-timeout-in-seconds=120","secretName":"blob-secret"},"volume_id":"csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968"}
I0210 14:06:22.017221  372960 nodeserver.go:102] NodePublishVolume: ephemeral volume(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) mount on /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount
I0210 14:06:22.017542  372960 blob.go:467] parsing volumeID(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) return with error: error parsing volume id: "csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968", should at least contain two #
I0210 14:06:22.017563  372960 blob.go:548] volumeID(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) authEnv: []
I0210 14:06:22.037704  372960 blob.go:955] got storage account(k8sblobstoragetest) from secret(blob-secret) namespace(ns-fida-test)
I0210 14:06:22.037753  372960 nodeserver.go:424] target /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount protocol  volumeId csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968
mountflags []
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true] volumeMountGroup 
args /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
serverAddress k8sblobstoragetest.blob.core.windows.net
I0210 14:06:22.037780  372960 nodeserver.go:166] start connecting to blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
I0210 14:06:22.038528  372960 nodeserver.go:175] begin to mount with blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
E0210 14:06:22.039036  372960 nodeserver.go:178] GRPC call returned with an error:rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH 
E0210 14:06:22.039076  372960 nodeserver.go:450] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output: 
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I0210 14:06:22.039164  372960 azure_metrics.go:118] "Observed Request Latency" latency_seconds=0.021905886 request="blob_csi_driver_node_stage_volume" resource_group="" subscription_id="" source="blob.csi.azure.com" volumeid="csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968" result_code="failed_csi_driver_node_stage_volume"
E0210 14:06:22.039179  372960 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output: 
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

[sindreal]

@andyzhangx

Finally got some time to continue troubleshooting.

First of all, when including the following helm values:

    node.enableBlobfuseProxy: true
    cloud: AzureStackCloud

It failed to install the helmrelease properly, and printed the following error message:

  Warning  InstallFailed     31s   helm-controller  Helm install failed for release kube-system/kube-system-rel-blob with chart blob-csi-driver@1.25.1: 1 error occurred:
           * DaemonSet.apps "csi-blob-node" is invalid: [spec.template.spec.containers[2].volumeMounts[5].mountPath: Invalid value: "/etc/ssl/certs": must be unique, spec.template.spec.containers[2].volumeMounts[6].name: Not found: "ssl-pki"]

After removing the "cloud: AzureStackCloud" it installed successfully, but I still get the original error message on the pod itself:

Warning  FailedMount             57s (x10 over 5m7s)  kubelet                  MountVolume.SetUp failed for volume "volume-inline-blob-1" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

Is the cloud value needed? What does it do?
And is the PATH error message related to the k8s-node itself, or the csi-blob-node?

Sinde reinstall, the output from "kubectl logs daemonset/csi-blob-node -c blob -n kube-system" is quite small, but I couldn't find a smoking gun:

Found 5 pods, using pod/csi-blob-node-869sp
I0217 11:16:56.077271 2890252 main.go:115] set up prometheus server on [::]:29635
I0217 11:16:56.077382 2890252 main.go:76] Sys info: NumCPU: 4 MAXPROC: 2
I0217 11:16:56.077395 2890252 main.go:79] driver userAgent: blob.csi.azure.com/v1.25.1 OSS-helm
W0217 11:16:56.077406 2890252 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0217 11:16:56.077506 2890252 util.go:348] set QPS(25.000000) and QPS Burst(50) for driver kubeClient
I0217 11:16:56.077834 2890252 azure.go:64] reading cloud config from secret kube-system/azure-cloud-provider
I0217 11:16:56.140852 2890252 azure.go:77] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0217 11:16:56.140880 2890252 azure.go:82] could not read cloud config from secret kube-system/azure-cloud-provider
I0217 11:16:56.140925 2890252 azure.go:88] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
W0217 11:16:56.140952 2890252 azure.go:95] load azure config from file(/etc/kubernetes/azure.json) failed with open /etc/kubernetes/azure.json: no such file or directory
I0217 11:16:56.140966 2890252 azure.go:101] no cloud config provided, error: open /etc/kubernetes/azure.json: no such file or directory, driver will run without cloud config
I0217 11:16:56.140976 2890252 azure.go:139] starting node server on node(moc-lcmbhwqdkv8)
I0217 11:16:56.140986 2890252 main.go:90] cloud: , location: , rg: , VnetName: , VnetResourceGroup: , SubnetName:
I0217 11:16:56.170148 2890252 mount_linux.go:334] Detected umount with safe 'not mounted' behavior
I0217 11:16:56.170795 2890252 driver.go:80] Enabling controller service capability: CREATE_DELETE_VOLUME
I0217 11:16:56.170812 2890252 driver.go:80] Enabling controller service capability: EXPAND_VOLUME
I0217 11:16:56.170817 2890252 driver.go:80] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I0217 11:16:56.170826 2890252 driver.go:80] Enabling controller service capability: CLONE_VOLUME
I0217 11:16:56.170835 2890252 driver.go:99] Enabling volume access mode: SINGLE_NODE_WRITER
I0217 11:16:56.170843 2890252 driver.go:99] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0217 11:16:56.170847 2890252 driver.go:99] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I0217 11:16:56.170850 2890252 driver.go:99] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I0217 11:16:56.170853 2890252 driver.go:99] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0217 11:16:56.170856 2890252 driver.go:99] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0217 11:16:56.170859 2890252 driver.go:99] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0217 11:16:56.170865 2890252 driver.go:90] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0217 11:16:56.170870 2890252 driver.go:90] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I0217 11:16:56.170874 2890252 driver.go:90] Enabling node service capability: VOLUME_MOUNT_GROUP
I0217 11:16:56.171719 2890252 blob.go:368]
DRIVER INFORMATION:
-------------------
Build Date: "2025-01-20T02:46:52Z"
Compiler: gc
Driver Name: blob.csi.azure.com
Driver Version: v1.25.1
Git Commit: 86a616fd8cddd4d9f2c0a063067773ece618d275
Go Version: go1.23.1
Platform: linux/amd64

Streaming logs below:
I0217 11:16:56.741893 2890252 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo
I0217 11:16:56.741910 2890252 utils.go:105] GRPC request: {}
I0217 11:16:56.744367 2890252 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.25.1"}
I0217 11:16:56.918048 2890252 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo
I0217 11:16:56.918072 2890252 utils.go:105] GRPC request: {}
I0217 11:16:56.918105 2890252 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.25.1"}
I0217 11:16:57.322438 2890252 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo
I0217 11:16:57.322453 2890252 utils.go:105] GRPC request: {}
I0217 11:16:57.322529 2890252 utils.go:111] GRPC response: {"node_id":"moc-lcmbhwqdkv8"}

[andyzhangx]

what is the output of following commands:

kubectl describe ds csi-blob-node -n kube-system
kubectl logs daemonset/csi-blob-node -c install-blobfuse-proxy -n kube-system

[sindreal]

kubectl describe ds csi-blob-node -n kube-system:

Name:           csi-blob-node
Selector:       app=csi-blob-node,app.kubernetes.io/instance=kube-system-rel-blob,app.kubernetes.io/name=blob-csi-driver
Node-Selector:  kubernetes.io/os=linux
Labels:         app=csi-blob-node
                app.kubernetes.io/component=csi-driver
                app.kubernetes.io/instance=kube-system-rel-blob
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=blob-csi-driver
                app.kubernetes.io/part-of=blob-csi-driver
                app.kubernetes.io/version=1.25.1
                helm.sh/chart=blob-csi-driver-1.25.1
                helm.toolkit.fluxcd.io/name=rel-blob
                helm.toolkit.fluxcd.io/namespace=flux-green-test-hci01
Annotations:    deprecated.daemonset.template.generation: 1
                meta.helm.sh/release-name: kube-system-rel-blob
                meta.helm.sh/release-namespace: kube-system
Desired Number of Nodes Scheduled: 5
Current Number of Nodes Scheduled: 5
Number of Nodes Scheduled with Up-to-date Pods: 5
Number of Nodes Scheduled with Available Pods: 5
Number of Nodes Misscheduled: 0
Pods Status:  5 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=csi-blob-node
                    app.kubernetes.io/component=csi-driver
                    app.kubernetes.io/instance=kube-system-rel-blob
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=blob-csi-driver
                    app.kubernetes.io/part-of=blob-csi-driver
                    app.kubernetes.io/version=1.25.1
                    helm.sh/chart=blob-csi-driver-1.25.1
  Service Account:  csi-blob-node-sa
  Init Containers:
   install-blobfuse-proxy:
    Image:      mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.25.1
    Port:       <none>
    Host Port:  <none>
    Command:
      /blobfuse-proxy/init.sh
    Environment:
      DEBIAN_FRONTEND:         noninteractive
      INSTALL_BLOBFUSE:        false
      BLOBFUSE_VERSION:        1.4.5
      INSTALL_BLOBFUSE2:       true
      BLOBFUSE2_VERSION:       2.4.0
      INSTALL_BLOBFUSE_PROXY:  true
      SET_MAX_OPEN_FILE_NUM:   true
      MAX_FILE_NUM:            9000000
      DISABLE_UPDATEDB:        true
      KUBELET_PATH:            /var/lib/kubelet
      MIGRATE_K8S_REPO:        false
      SET_READ_AHEAD_SIZE:     true
    Mounts:
      /host/etc from host-etc (rw)
      /host/usr from host-usr (rw)
      /host/usr/local from host-usr-local (rw)
  Containers:
   liveness-probe:
    Image:      mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.14.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=/csi/csi.sock
      --probe-timeout=3s
      --http-endpoint=localhost:29633
      --v=2
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /csi from socket-dir (rw)
   node-driver-registrar:
    Image:      mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.12.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
      --v=2
    Limits:
      memory:  100Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Environment:
      ADDRESS:               /csi/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/blob.csi.azure.com/csi.sock
    Mounts:
      /csi from socket-dir (rw)
      /registration from registration-dir (rw)
   blob:
    Image:      mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.25.1
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --endpoint=$(CSI_ENDPOINT)
      --blobfuse-proxy-endpoint=$(BLOBFUSE_PROXY_ENDPOINT)
      --enable-blobfuse-proxy=true
      --nodeid=$(KUBE_NODE_NAME)
      --drivername=blob.csi.azure.com
      --cloud-config-secret-name=azure-cloud-provider
      --cloud-config-secret-namespace=kube-system
      --custom-user-agent=
      --user-agent-suffix=OSS-helm
      --allow-empty-cloud-config=true
      --enable-get-volume-stats=false
      --append-timestamp-cache-dir=false
      --mount-permissions=511
      --allow-inline-volume-key-access-with-idenitity=false
      --enable-aznfs-mount=true
      --metrics-address=0.0.0.0:29635
    Limits:
      memory:  2100Mi
    Requests:
      cpu:     10m
      memory:  20Mi
    Liveness:  http-get http://localhost:29633/healthz delay=30s timeout=10s period=30s #success=1 #failure=5
    Environment:
      AZURE_CREDENTIAL_FILE:    <set to the key 'path' of config map 'azure-cred-file'>  Optional: true
      CSI_ENDPOINT:             unix:///csi/csi.sock
      BLOBFUSE_PROXY_ENDPOINT:  unix:///csi/blobfuse-proxy.sock
      KUBE_NODE_NAME:            (v1:spec.nodeName)
      AZURE_GO_SDK_LOG_LEVEL:
    Mounts:
      /csi from socket-dir (rw)
      /etc/kubernetes/ from azure-cred (rw)
      /lib/modules from lib-modules (ro)
      /mnt from blob-cache (rw)
      /opt/microsoft/aznfs/data from aznfs-data (rw)
      /var/lib/kubelet/ from mountpoint-dir (rw)
   aznfswatchdog:
    Image:      mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.25.1
    Port:       <none>
    Host Port:  <none>
    Command:
      aznfswatchdog
    Limits:
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /opt/microsoft/aznfs/data from aznfs-data (rw)
      /var/lib/kubelet/ from mountpoint-dir (rw)
  Volumes:
   host-usr:
    Type:          HostPath (bare host directory volume)
    Path:          /usr
    HostPathType:
   host-usr-local:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local
    HostPathType:
   host-etc:
    Type:          HostPath (bare host directory volume)
    Path:          /etc
    HostPathType:
   socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/blob.csi.azure.com
    HostPathType:  DirectoryOrCreate
   mountpoint-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  DirectoryOrCreate
   registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
   azure-cred:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  DirectoryOrCreate
   blob-cache:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt
    HostPathType:
   aznfs-data:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/microsoft/aznfs/data
    HostPathType:  DirectoryOrCreate
   lib-modules:
    Type:               HostPath (bare host directory volume)
    Path:               /lib/modules
    HostPathType:       DirectoryOrCreate
  Priority Class Name:  system-cluster-critical
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  39m   daemonset-controller  Created pod: csi-blob-node-cx2pk
  Normal  SuccessfulCreate  39m   daemonset-controller  Created pod: csi-blob-node-2fcww
  Normal  SuccessfulCreate  39m   daemonset-controller  Created pod: csi-blob-node-lvpzr
  Normal  SuccessfulCreate  39m   daemonset-controller  Created pod: csi-blob-node-869sp
  Normal  SuccessfulCreate  39m   daemonset-controller  Created pod: csi-blob-node-tvwv4

kubectl logs daemonset/csi-blob-node -c install-blobfuse-proxy -n kube-system

Found 5 pods, using pod/csi-blob-node-869sp
+ INSTALL_BLOBFUSE_PROXY=true
+ DISABLE_UPDATEDB=true
+ SET_MAX_OPEN_FILE_NUM=true
+ SET_READ_AHEAD_SIZE=true
+ MIGRATE_K8S_REPO=false
+ READ_AHEAD_KB=15380
+ KUBELET_PATH=/var/lib/kubelet
+ [ /var/lib/kubelet != /var/lib/kubelet ]
+ HOST_CMD=nsenter --mount=/proc/1/ns/mnt
+ nsenter --mount=/proc/1/ns/mnt cat /etc/os-release
+ grep ^ID=
+ cut -d= -f2
+ tr -d "
+ DISTRIBUTION=mariner
+ nsenter --mount=/proc/1/ns/mnt uname -m
Linux distribution: mariner, Arch: x86_64
skip install blobfuse/blobfuse2 for mariner....
+ ARCH=x86_64
+ echo Linux distribution: mariner, Arch: x86_64
+ . ./blobfuse-proxy/install-proxy.sh
+ set -xe
+ [ mariner != ubuntu ]
+ echo skip install blobfuse/blobfuse2 for mariner....
+ [ true = true ]
+ updateBlobfuseProxy=true
+ [ -f /host/usr/bin/blobfuse-proxy ]
+ sha256sum /host/usr/bin/blobfuse-proxy
+ awk {print $1}
+ old=b6ec99d3b48bde3279a67b7ba99553835d684afabd1407a74e810dc76d7a2b8d
+ sha256sum /blobfuse-proxy/blobfuse-proxy
+ awk {print $1}
+ new=b6ec99d3b48bde3279a67b7ba99553835d684afabd1407a74e810dc76d7a2b8d
+ [ b6ec99d3b48bde3279a67b7ba99553835d684afabd1407a74e810dc76d7a2b8d = b6ec99d3b48bde3279a67b7ba99553835d684afabd1407a74e810dc76d7a2b8d ]
+ updateBlobfuseProxy=false
+ echo no need to update blobfuse-proxy
+ [ false = true ]
+ updateService=true
+ [ -f /host/usr/lib/systemd/system/blobfuse-proxy.service ]
no need to update blobfuse-proxy
+ + awksha256sum {print $1}
 /host/usr/lib/systemd/system/blobfuse-proxy.service
+ old=fac872ebbd5d6e3d253a65a121b670b76ba32ec226a35339218b7ff71ddedc4c
+ sha256sum /blobfuse-proxy/blobfuse-proxy.service
+ awk {print $1}
+ new=fac872ebbd5d6e3d253a65a121b670b76ba32ec226a35339218b7ff71ddedc4c
+ [ fac872ebbd5d6e3d253a65a121b670b76ba32ec226a35339218b7ff71ddedc4c = fac872ebbd5d6e3d253a65a121b670b76ba32ec226a35339218b7ff71ddedc4c ]
+ updateService=false
+ echo no need to update blobfuse-proxy.service
no need to update blobfuse-proxy.service
+ [ false = true ]
+ nsenter --mount=/proc/1/ns/mnt systemctl daemon-reload
+ nsenter --mount=/proc/1/ns/mnt systemctl enable blobfuse-proxy.service
start blobfuse-proxy....
+ [ false = true ]
+ [ false = true ]
+ echo start blobfuse-proxy....
+ nsenter --mount=/proc/1/ns/mnt systemctl start blobfuse-proxy.service
+ [ true = true ]
+ nsenter --mount=/proc/1/ns/mnt sysctl -w fs.file-max=9000000
fs.file-max = 9000000
+ updateDBConfigPath=/host/etc/updatedb.conf
+ [ true = true ]
+ [ -f /host/etc/updatedb.conf ]
+ [ true = true ]
+ echo set read ahead size to 15380KB
set read ahead size to 15380KB
+ which awk
+ AWK_PATH=/usr/bin/awk
+ cat
+ nsenter --mount=/proc/1/ns/mnt udevadm control --reload

[andyzhangx]

+ [ mariner != ubuntu ]
+ echo skip install blobfuse/blobfuse2 for mariner....

this is Azure Linux VM, you need to install blobfuse2 on all nodes manually

[sindreal]

It's AKS nodes on Azure Local (new name for Azure Stack), I'm not supposed to do manual tasks on those, as they may rotate etc.

I'm assuming that the "cloud: AzureStackCloud" value does something related to this? If so, maybe follow that trail instead.

[andyzhangx]

cloud: AzureStackCloud is not related, you could disable blobfuse-proxy to make it work: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/deploy/blobfuse-proxy, if blobfuse-proxy is disabled, it would use the installed blobfuse2 binary inside the csi driver, that would work

[sindreal]

Okay, now I'm slightly confused.
Initially when opening the issue, the node.enableBlobfuseProxy option was not set, but I'll disable it again and check.

[sindreal]

Even without the helm value set, it seems like "--enable-blobfuse-proxy=true" is set in "kubectl describe ds csi-blob-node -n kube-system"
Tried setting it to false: Values: node.enableBlobfuseProxy=false

And yet, kubectl describe ds csi-blob-node -n kube-system shows --enable-blobfuse-proxy=true

Is that the correct way to disable it, and to check if it's actually disabled?

[andyzhangx]

just run kubectl edit ds csi-blob-node -n kube-system and set --enable-blobfuse-proxy=true