Inline blob mount producing error message claiming blobfuse2 is not in $PATH

[sindreal]

Doing a simple inline blob mount test following the example at:
https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/deploy/example/e2e_usage.md#option3-inline-volume

What happened:
Pod won't start, due to the following error mesage:
Warning FailedMount 14s (x35 over 55m) kubelet MountVolume.SetUp failed for volume "volume-inline-blob-1" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output:
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.

What you expected to happen:
That the pod started with the blob volume mounted.

Anything else we need to know?:
All csi-blob-node-xxxxx nodes seems to have blobfuse2 in their $PATH, and I can successfully mount the blob manually using blobfuse2 from the csi-blob-node. The secret is also present in the same namespace as the pod.

Environment:

• CSI Driver version: 1.25.1
• Kubernetes version (use kubectl version): 1.29.4
• OS (e.g. from /etc/os-release): AKS on Azure Local
• Install tools: Helm
• Others:

[andyzhangx]

@sindreal have you tried other example, e.g. option#1, option#2?
and what is AKS on Azure Local?

[andyzhangx]

I see where is the problem, have you set node.enableBlobfuseProxy=true in helm chart install? this driver is using blobfuse-proxy by default, and this driver would install blobfuse2 on the node by default, you are mounting inside blob csi driver pod, that works since blobfuse2 is already installed by default inside csi driver.
details: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/charts#tips

[sindreal]

The other examples also fails, and AKS on Azure Local is just renamed AKS on Azure Stack, which lacks access to this extension, hence the use of Helm.
I'll try to get node.enabelBlobFuseProxy=true enabled via helm, how may I easily check if it's installed?

[andyzhangx]

pls provide the csi driver logs first: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed, the blobfuse2 and blobfuse-proxy install logs are in install-blobfuse-proxy container

[sindreal]

kubectl logs daemonset/csi-blob-node -c blob -n kube-system
total of 130k lines, so this is just the end.

I0210 14:06:22.017012  372960 utils.go:104] GRPC call: /csi.v1.Node/NodePublishVolume
I0210 14:06:22.017028  372960 utils.go:105] GRPC request: {"target_path":"/var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"containerName":"blob-k8s-test","csi.storage.k8s.io/ephemeral":"true","csi.storage.k8s.io/pod.name":"deployment-fida-kol-1-5b49c84b6d-lqj4q","csi.storage.k8s.io/pod.namespace":"ns-fida-test","csi.storage.k8s.io/pod.uid":"06db8aad-3689-4e41-b4b1-ec591aa72045","csi.storage.k8s.io/serviceAccount.name":"default","csi.storage.k8s.io/serviceAccount.tokens":"***stripped***","mountOptions":"-o allow_other --file-cache-timeout-in-seconds=120","secretName":"blob-secret"},"volume_id":"csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968"}
I0210 14:06:22.017221  372960 nodeserver.go:102] NodePublishVolume: ephemeral volume(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) mount on /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount
I0210 14:06:22.017542  372960 blob.go:467] parsing volumeID(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) return with error: error parsing volume id: "csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968", should at least contain two #
I0210 14:06:22.017563  372960 blob.go:548] volumeID(csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968) authEnv: []
I0210 14:06:22.037704  372960 blob.go:955] got storage account(k8sblobstoragetest) from secret(blob-secret) namespace(ns-fida-test)
I0210 14:06:22.037753  372960 nodeserver.go:424] target /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount protocol  volumeId csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968
mountflags []
mountOptions [-o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true] volumeMountGroup 
args /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
serverAddress k8sblobstoragetest.blob.core.windows.net
I0210 14:06:22.037780  372960 nodeserver.go:166] start connecting to blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
I0210 14:06:22.038528  372960 nodeserver.go:175] begin to mount with blobfuse proxy, protocol: , args: /var/lib/kubelet/pods/06db8aad-3689-4e41-b4b1-ec591aa72045/volumes/kubernetes.io~csi/volume-inline-blob-1/mount -o allow_other --file-cache-timeout-in-seconds=120 --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968 --container-name=blob-k8s-test --pre-mount-validate=true --use-https=true
E0210 14:06:22.039036  372960 nodeserver.go:178] GRPC call returned with an error:rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH 
E0210 14:06:22.039076  372960 nodeserver.go:450] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output: 
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.
I0210 14:06:22.039164  372960 azure_metrics.go:118] "Observed Request Latency" latency_seconds=0.021905886 request="blob_csi_driver_node_stage_volume" resource_group="" subscription_id="" source="blob.csi.azure.com" volumeid="csi-b5b8a76b03633823ab412389660babe7ae70af18a706b67e37698765a157f968" result_code="failed_csi_driver_node_stage_volume"
E0210 14:06:22.039179  372960 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exec: "blobfuse2": executable file not found in $PATH , output: 
Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors.