NFS CSI (nfs.csi.k8s.io) complains of permission denied in tmp directory when writing a pvc.

[Spaghetto392]

Hi, when trying to create a NFS based PVC, I get an error saying that permission was denied for creating a directory in /tmp. I was wondering if /tmp was potentially made read only, or otherwise has some permissions that k8s fails to clear. That or is the CSI I'm using just incompatible with Talos?

[smira]

On the host /tmp is a tmpfs, so it's writable (but it's small, so it's better not to create anything big there).

[smira]

I would probably look into the logs of the CSI pods (if any) and also try to make nfs version lower.

[Spaghetto392]

Hi, on downgrading to version 3, I got the following:

  Type     Reason                Age                From                                                               Message
  ----     ------                ----               ----                                                               -------
  Normal   Provisioning          16s                nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  External provisioner is provisioning volume for claim "test/pvc-test"
  Normal   ExternalProvisioning  16s (x2 over 16s)  persistentvolume-controller                                        Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed    6s                 nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o hard,nfsvers=3 192.168.1.200:/mnt/Pool1/Test /tmp/pvc-d0155ea2-3582-4b00-a0bf-7b517d39f248
Output: mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.

if nolock is passed through, I get the original error again.

  Type     Reason                Age               From                                                               Message
  ----     ------                ----              ----                                                               -------
  Normal   Provisioning          8s (x4 over 15s)  nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  External provisioner is provisioning volume for claim "test/pvc-test"
  Warning  ProvisioningFailed    8s (x4 over 15s)  nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-e2e4d8dd-75ed-463c-b438-26c09da95106/pvc-e2e4d8dd-75ed-463c-b438-26c09da95106: permission denied
  Normal   ExternalProvisioning  7s (x2 over 15s)  persistentvolume-controller                                        Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Unfortunately the CSI logs are empty.

[smira]

yeah, interesting, I don't know, and it doesn't seem to be related to Talos in a way I can see. Probably worth trying opening an issue with the CSI provider?

[Spaghetto392]

I'll try opening an issue with the CSI, I'll post an update if I get anything that helps.

[MichaelTrip]

I'll try opening an issue with the CSI, I'll post an update if I get anything that helps.

I have the same problem. For that reason I mainly use the NFS subdir provisioner.

When I come home from my holiday I will create an issue at their GitHub repo.

[zviratko]

I am getting this error as well, semi-randomly. At first I thought it was because I was using Talos and thus even /tmp was immutable. I changed it to /dev/shm but looking at it closer, it is not a hostpath but an emptydir which should always work, so I changed it back and it works... no idea why I saw this error previously. Some sort of race condition?

[andyzhangx]

could you check whether readOnlyRootFilesystem: true is set in nfs container? that could be root cause, I have remove this setting in PR: kubernetes-csi/csi-driver-nfs#720

[zviratko]

Yes, I see it set in that container and it is indeed read only
overlay on / type overlay (ro,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/156/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/155/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/77/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/284/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/284/work,uuid=on)

Isn't creating and mounting an emptydir a better solution?

[andyzhangx]

readOnlyRootFilesystem: true is not a necessary setting for nfs container