Toolbox Run failed to get Container Device Interface containerEdits for NVIDIA

[simon3z]

Describe the bug
The command toolbox run fails with "Error: failed to get Container Device Interface containerEdits for NVIDIA".

$ toolbox run ls
...
level=debug msg="Generating Container Device Interface for NVIDIA: failed to get containerEdits: failed to create 
discoverer for common entities: failed to create discoverer for driver files: failed to create discoverer for driver libraries: failed to get libraries for driver version: failed to locate libcuda.so.575.57.08: pattern libcuda.so.575.57.08 not found\nlibcuda.so.575.57.08: not found"
Error: failed to get Container Device Interface containerEdits for NVIDIA

My system has nvidia kernel drivers loaded:

$ lsmod | grep nvidia
nvidia_uvm           4218880  4
nvidia_drm            159744  5
nvidia_modeset       2150400  1 nvidia_drm
nvidia              12976128  9 nvidia_uvm,nvidia_modeset
drm_ttm_helper         16384  2 nvidia_drm,xe
video                  81920  4 asus_wmi,xe,i915,nvidia_modeset

But CUDA libraries are NOT installed.

Steps how to reproduce the behaviour

1. Load nvidia kernel drivers
2. Make sure not to have CUDA libraries installed
3. Run a toolbox run command
4. See error

Expected behaviour
The toolbox run command is expected to be executed regularly.

Actual behaviour
The command toolbox run fails with "Error: failed to get Container Device Interface containerEdits for NVIDIA"

Output of toolbox --version (v0.0.90+)
toolbox version 0.1.2

Toolbx package info (rpm -q toolbox)
toolbox-0.1.2-1.fc42.x86_64

Output of podman version
e.g.,

Client:        Podman Engine
Version:       5.5.2
API Version:   5.5.2
Go Version:    go1.24.4
Git Commit:    e7d8226745ba07a64b7176a7f128e4ef53225a0e
Built:         Tue Jun 24 02:00:00 2025
Build Origin:  Fedora Project
OS/Arch:       linux/amd64

Podman package info (rpm -q podman)
podman-5.5.2-1.fc42.x86_64

Info about your OS
Fedora Linux 42 (Workstation Edition)

Additional context
Note that toolbox create, toolbox enter, etc. all works perfectly fine.
A possible fix would be failing with an ErrPlatformUnsupported in GenerateCDISpec when CUDA libraries are not found in the host system so that the error is managed gracefully.

diff --git a/src/pkg/nvidia/nvidia.go b/src/pkg/nvidia/nvidia.go
index c2cfe19..5593d5b 100644
--- a/src/pkg/nvidia/nvidia.go
+++ b/src/pkg/nvidia/nvidia.go
@@ -108,7 +108,7 @@ func GenerateCDISpec() (*specs.Spec, error) {
        commonEdits, err := cdi.GetCommonEdits()
        if err != nil {
                logrus.Debugf("Generating Container Device Interface for NVIDIA: failed to get containerEdits: %s", err)
-               return nil, errors.New("failed to get Container Device Interface containerEdits for NVIDIA")
+               return nil, ErrPlatformUnsupported
        }
 
        spec, err := nvspec.New(nvspec.WithEdits(*commonEdits.ContainerEdits))

[debarshiray]

Interesting. I see two parts in this bug report. I think I somewhat understand one, and don't understand the other. :)

The command toolbox run fails with "Error: failed to get Container Device Interface containerEdits for NVIDIA".

$ toolbox run ls
...
level=debug msg="Generating Container Device Interface for NVIDIA: failed to get containerEdits: failed to create 
discoverer for common entities: failed to create discoverer for driver files: failed to create discoverer for driver libraries: failed to get libraries for driver version: failed to locate libcuda.so.575.57.08: pattern libcuda.so.575.57.08 not found\nlibcuda.so.575.57.08: not found"
Error: failed to get Container Device Interface containerEdits for NVIDIA

This is the first part, and I think I understand it.

Note that toolbox create, toolbox enter, etc. all works perfectly fine.

This is the other part that I don't understand.

Are you saying that toolbox enter works and toolbox run doesn't? It's surprising because they share most of the code, and the code for the proprietary NVIDIA driver is common between the two. I am assuming that toolbox create always works because it creates the container that both run and enter use.

[debarshiray]

Let's look at the first part more closely.

From where did you get the proprietary NVIDIA driver? Did you get it from NVIDIA's own repository or RPM Fusion or somewhere else? I am asking because NVIDIA and RPM Fusion package the driver somewhat differently.

I am guessing that you didn't get it from RPM Fusion, because I have the RPM Fusion packages on my NVIDIA machine. :)

I apologize in advance if I say something stupid. While I understand some aspects of the proprietary NVIDIA driver, the CUDA toolchain, etc., I am by no means an expert, and there's a lot that I don't know.

The command toolbox run fails with "Error: failed to get Container Device Interface containerEdits for NVIDIA".

$ toolbox run ls
...
level=debug msg="Generating Container Device Interface for NVIDIA: failed to get containerEdits: failed to create 
discoverer for common entities: failed to create discoverer for driver files: failed to create discoverer for driver libraries: failed to get libraries for driver version: failed to locate libcuda.so.575.57.08: pattern libcuda.so.575.57.08 not found\nlibcuda.so.575.57.08: not found"
Error: failed to get Container Device Interface containerEdits for NVIDIA

My system has nvidia kernel drivers loaded:

$ lsmod | grep nvidia
nvidia_uvm           4218880  4
nvidia_drm            159744  5
nvidia_modeset       2150400  1 nvidia_drm
nvidia              12976128  9 nvidia_uvm,nvidia_modeset
drm_ttm_helper         16384  2 nvidia_drm,xe
video                  81920  4 asus_wmi,xe,i915,nvidia_modeset

But CUDA libraries are NOT installed.

Some context ...

I don't have the CUDA toolchain either, or at least what RPM Fusion considers (more on that below) as CUDA. I have the proprietary driver from RPM Fusion, because that's the default source for it in Fedora Workstation. The CUDA toolchain has to be installed from NVIDIA's repository, with their packaging of the driver, because it's different from RPM Fusion's.

RPM Fusion has a xorg-x11-drv-nvidia-cuda-libs package that contains /usr/lib64/libcuda.so, /usr/lib64/libcudadebugger.so.1, etc.. This is required by xorg-x11-drv-nvidia-libs that contains the OpenGL, VDPAU, Vullkan, etc. libraries, and that's required by xorg-x11-drv-nvidia that contains a dracut configuration snippet, the firmware, systemd unit for falling back to Nouveau, udev rules, etc..

Even though RPM Fusion offers libcuda.so as part of the driver, it certainly doesn't have the NVIDIA CUDA Compiler (or nvcc) or the rest of the CUDA toolchain. I can't compile a CUDA source code. I don't know if I can run an existing CUDA binary, though.

So, this seems to imply that libcuda.so is somehow needed for the proprietary NVIDIA driver to work, or that it's somehow different from the rest of the CUDA toolchain. I don't know, and that's why I am curious.

Steps how to reproduce the behaviour

1. Load nvidia kernel drivers

2. Make sure not to have CUDA libraries installed

Exactly how did you do this? Did your packaging of the proprietary NVIDIA driver not include libcuda.so or did you manually do something to remove it? Is the driver working on the host?

Looking at the NVIDIA Container Toolkit code, the error comes from getVersionLibs(). The way the code is written, it looks like libcuda.so is essential for the driver to work.

A possible fix would be failing with an ErrPlatformUnsupported in GenerateCDISpec when CUDA libraries are not found in the host system so that the error is managed gracefully.

diff --git a/src/pkg/nvidia/nvidia.go b/src/pkg/nvidia/nvidia.go
index c2cfe19..5593d5b 100644
--- a/src/pkg/nvidia/nvidia.go
+++ b/src/pkg/nvidia/nvidia.go
@@ -108,7 +108,7 @@ func GenerateCDISpec() (*specs.Spec, error) {
        commonEdits, err := cdi.GetCommonEdits()
        if err != nil {
                logrus.Debugf("Generating Container Device Interface for NVIDIA: failed to get containerEdits: %s", err)
-               return nil, errors.New("failed to get Container Device Interface containerEdits for NVIDIA")
+               return nil, ErrPlatformUnsupported
        }
 
        spec, err := nvspec.New(nvspec.WithEdits(*commonEdits.ContainerEdits))

I agree that it would be good to produce a more human-readable error message or fall back with a warning.

I wish the NVIDIA Container Toolkit would provide some well-known errors that we can specifically look for and handle separately. Currently, they don't wrap the errors with fmt.Errorf("... %w", err) as they are propagated through the call chain, which is a factor. Maybe they should?

[simon3z]

@debarshiray unless I am missing something, you did a perfect analysis, same as mine, the NVIDIA Container Toolkit should provide more formalized errors. It's worth following up with that project for an enhancement but for now I think the patch I proposed is the best we can do to avoid this toolbox unexpected failure.

Should one of us send that PR?

Thanks.

[debarshiray]

the NVIDIA Container Toolkit should provide more formalized errors. It's worth following up with that project for an enhancement

Yes, please feel free to file an NVIDIA Container Toolkit issue.

but for now I think the patch I proposed is the best we can do to avoid this toolbox unexpected failure.

I am still curious about the questions above. eg., from where did you get the proprietary NVIDIA driver and how did you end up without libcuda.so?

There is the bigger question whether the driver is even supposed to work without libcuda.so. RPM Fusion and NVIDIA Container Toolkit seem to imply that libcuda.so is mandatory. If that's really true, then the short-term work around will be for you to install it.

[simon3z]

I am still curious about the questions above. eg., from where did you get the proprietary NVIDIA driver and how did you end up without libcuda.so?
There is the bigger question whether the driver is even supposed to work without libcuda.so. RPM Fusion and NVIDIA Container Toolkit seem to imply that libcuda.so is mandatory. If that's really true, then the short-term work around will be for you to install it.

For harness and keep the host Fedora system as tidy and possible I am installing just the drivers, and all the CUDA applications are running inside either podman containers or toolbox.

I think it's a valid use case and toolbox shouldn't be failing.

I'd also call out the difference between the working toolbox create/enter and the failing toolbox run.

[debarshiray]

So, this seems to imply that libcuda.so is somehow needed for the proprietary NVIDIA driver to work, or that it's somehow different from the rest of the CUDA toolchain. I don't know, and that's why I am curious.

I filed an NVIDIA Container Toolkit issue to clarify this. See: NVIDIA/nvidia-container-toolkit#1242

[simon3z]

I am curious if libcuda.so is mandatory for the proprietary NVIDIA driver to work? eg., purely for driving the display with OpenGL and Vulkan.

@debarshiray of course user-space libraries are NOT mandatory for the kernel drivers to work properly.

[debarshiray]

I am curious if libcuda.so is mandatory for the proprietary NVIDIA driver to work? eg., purely for driving the display with OpenGL and Vulkan.

@debarshiray of course user-space libraries are NOT mandatory for the kernel drivers to work properly.

A GPU driver consists of both a kernel and a user space part. The kernel space driver is useless on its own. This is the same as for CPUs. Support for a CPU in the kernel is useless if the user space runtime (eg., the dynamic linker, interpreters, etc.) can't handle it.

In the context of Toolbx, we are only interested in the user space driver, because the kernel space driver will always be shared with the host OS.

I am still curious about the questions above. eg., from where did you get the proprietary NVIDIA driver and how did you end up without libcuda.so?
There is the bigger question whether the driver is even supposed to work without libcuda.so. RPM Fusion and NVIDIA Container Toolkit seem to imply that libcuda.so is mandatory. If that's really true, then the short-term work around will be for you to install it.

For harness and keep the host Fedora system as tidy and possible I am installing just the drivers

How exactly are you installing just the drivers? I am trying to understand if NVIDIA supports this method that you are using or are you somehow overriding it?

and all the CUDA applications are running inside either podman containers or toolbox.

I think it's a valid use case and toolbox shouldn't be failing.

As I described earlier, based on how the driver is packaged and how the NVIDIA Container Toolkit works today, just because a shared library is called libcuda.so, it might not mean that it's not needed by the proprietary NVIDIA driver or it might not be a logically separate entity like the CUDA toolchain.

It will be either very difficult or impossible for Toolbx to deviate from what NVIDIA considers a supported configuration for its proprietary driver, because it's proprietary; and the NVIDIA Container Toolkit is NVIDIA's way of formally advertising what their user space driver needs to work.

Based on whether libcuda.so is needed by the proprietary NVIDIA driver or not, this becomes a question of a better error message or handling the error and carrying on.

[debarshiray]

According to the discussion in NVIDIA/nvidia-container-toolkit#1242 , there is movement towards a potential solution in NVIDIA/nvidia-container-toolkit#1451