Can not find device after 565+ on GH200 NVL2

[zilingzhang]

NVIDIA Open GPU Kernel Modules Version

nvidia-driver-565-open(565.57.01), nvidia-driver-570-open (570.86.15)

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.1 LTS

Kernel Release

Linux 6.11.0-1002-nvidia-64k #2-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 23 19:17:25 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GH200 144G HBM3e | GPU 1: NVIDIA GH200 144G HBM3e

Describe the bug

Installing ubuntu 24.04 on a new GH200 NVL2 system, using apt NVIDIA open kernel driver.
Both GPUs not found when using the nvidia-driver-565-open or nvidia-driver-570-open apt package.
But both GPU found when using nvidia-driver-560-open

To Reproduce

sudo apt install nvidia-driver-565-open
nvidia-smi
No devices were found

sudo apt install nvidia-driver-570-open
nvidia-smi
No devices were found

sudo apt install nvidia-driver-560-open
nvidia-smi
both GPUs found.

Both GPUs are at 96.00.A0.00.01 VBIOS
Which should be newer than 96.00.68.00.xx
https://docs.nvidia.com/datacenter/tesla/tesla-release-notes-565-57-01/index.html#known-issues

nvidia-bug-report.log.gz

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

[aritger]

Could you also generate an nvidia-bug-report.log.gz for 560? It may help to compare logs between working and failing configurations.

[zilingzhang]

nvidia-bug-report.log.gz
This is from 560.35.05, but collection hang, let me know if it's good for diagnosis

[aritger]

If I'm reading the log correctly, 560.35.05 has the same symptom as 570.86.15:

[    6.488488] kernel: NVRM: numa memblock size of zero found during device start
[    6.488492] kernel: [drm:nv_drm_load [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00090100] Failed to allocate NvKmsKapiDevice
[    6.504217] kernel: [drm:nv_drm_register_drm_device [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00090100] Failed to register device

Once the GPU gets into this state, I think the problem will persist until a reboot. Could I trouble you to do the following for each of 560.35.05 and 570.86.15?

• install driver
• reboot
• run nvidia-smi
• run nvidia-bug-report.sh

[zilingzhang]

I did reboot between install, or nvidia-smi will show driver mismatch.
Currently I'm running 560.35.05 (New bug report) and GPUs show up in nvidia-smi and serving llama model correctly.

I will try to perform the sequence again for both versions tonight at down time and get back to you. Thanks!

[zilingzhang]

• Installed 570.86.15
• reboot
• run nvidia-smi -> No devices were found
• run nvidia-bug-report.sh

570.nvidia-bug-report.log.gz

• Installed 565.57.01
• reboot
• run nvidia-smi -> No devices were found
• run nvidia-bug-report.sh

565.nvidia-bug-report.log.gz

• Installed 560
• reboot
• run nvidia-smi -> both GPUs were found
• run nvidia-bug-report.sh

560.nvidia-bug-report.log.gz