“Nvidia-smi not found in $PATH” error

[darribas]

NOTE - Cross-posting from docker-snap#184 here as it might be more related to the container toolkit. I have tried removing the Docker snap and installing the engine through apt/get as suggested in Docker's docs, and the result is the same.

I’m running into the following issue:

dani@ymir:~$ docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.
[0.498s] 127  dani@ymir:~$

Here’re my specs

• OS version

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

• Kernel

Linux ymir 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• NVIDIA driver/CUDA on the host

Thu Sep 26 17:05:57 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
[…]

• Snap version

snap version 
snap    2.63.1+24.04
snapd   2.63.1+24.04
series  16
ubuntu  24.04
kernel  6.8.0-45-generic

• Docker info

snap info docker
name:      docker
summary:   Docker container runtime
publisher: Canonical✓
store-url: https://snapcraft.io/docker
contact:   https://github.com/docker-snap/docker-snap/issues?q=
license:   (Apache-2.0 AND MIT AND GPL-2.0)
description: |
  Build and run container images with Docker.
  
  **Usage**
  
  * This build can only access files in the home directory. So Dockerfiles and all other files used
  in commands like `docker build`, `docker save` and `docker load` need to be in $HOME.
  * You can change the configuration of this build by modifying the files in
  `/var/snap/docker/current/`.
  * Additional certificates used by the Docker daemon to authenticate with registries need to be
  added in `/var/snap/docker/current/etc/docker/certs.d` (instead of `/etc/docker/certs.d`). This
  directory can be accessed by other snaps using the `docker-registry-certificates` content
  interface.
  
  **Running Docker as normal user**
  
  By default, Docker is only accessible with root privileges (`sudo`). If you want to use docker as
  a regular user, you need to add your user to the `docker` group.
  
      sudo addgroup --system docker
      sudo adduser $USER docker
      newgrp docker
      sudo snap disable docker
      sudo snap enable docker
  
  **Warning:** if you add your user to the `docker` group, it will have similar power as the `root`
  user. For details on how this impacts security in your system, see
  https://docs.docker.com/engine/security/#docker-daemon-attack-surface
  
  **Authors**
  
  This snap is built by Canonical based on source code published by Docker, Inc. It is not endorsed
  or published by Docker, Inc.
  
  Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United
  States and/or other countries. Docker, Inc. and other parties may also have trademark rights in
  other terms used herein.
commands:
  - docker.compose
  - docker
  - docker.help
services:
  docker.dockerd:                  simple, enabled, active
  docker.nvidia-container-toolkit: oneshot, enabled, inactive
snap-id:      sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ
tracking:     latest/stable
refresh-date: 22 days ago, at 04:47 UTC
channels:
  latest/stable:    24.0.5   2024-09-03 (2932) 138MB -
  latest/candidate: 24.0.5   2024-09-03 (2932) 138MB -
  latest/beta:      27.2.0   2024-09-19 (2963) 146MB -
  latest/edge:      27.2.0   2024-09-20 (2969) 146MB -
  core18/stable:    20.10.17 2023-03-13 (2746) 146MB -
  core18/candidate: ↑                                
  core18/beta:      ↑                                
  core18/edge:      ↑                                
installed:          24.0.5              (2932) 138MB -

Do you have any sense what might be going on? Thanks!

[darribas]

I think this might be related to #679