Qualify NVIDIA GPUs by per-device CUDA compute capability

[joelagnel]

Problem

The NVIDIA hardware probe records the global CUDA driver API version from nvmlSystemGetCudaDriverVersion_v2 and copies its major version onto every detected GPU as CudaMajorVersion.

openclaw-windows-node/src/OpenClaw.Shared/Inference/NvmlHostHardwareProbe.cs

Lines 174 to 207 in 31dffdb

	var initialize = GetDelegate<NvmlInitialize>(library, "nvmlInit_v2");
	shutdown = GetDelegate<NvmlShutdown>(library, "nvmlShutdown");
	var getCount = GetDelegate<NvmlDeviceGetCount>(library, "nvmlDeviceGetCount_v2");
	var getHandle = GetDelegate<NvmlDeviceGetHandleByIndex>(library, "nvmlDeviceGetHandleByIndex_v2");
	var getName = GetDelegate<NvmlDeviceGetString>(library, "nvmlDeviceGetName");
	var getUuid = GetDelegate<NvmlDeviceGetString>(library, "nvmlDeviceGetUUID");
	var getMemory = GetDelegate<NvmlDeviceGetMemoryInfo>(library, "nvmlDeviceGetMemoryInfo");
	var getDriver = GetDelegate<NvmlSystemGetString>(library, "nvmlSystemGetDriverVersion");
	var getCuda = GetDelegate<NvmlSystemGetCudaDriverVersion>(library, "nvmlSystemGetCudaDriverVersion_v2");
	if (initialize() != NvmlSuccess)
	return NvmlProbeResult.Empty;
	initialized = true;
	string? driverVersion = ReadSystemString(getDriver, DriverVersionCapacity);
	int? cudaMajorVersion = getCuda(out int cudaDriverVersion) == NvmlSuccess && cudaDriverVersion > 0
	? cudaDriverVersion / 1000
	: null;
	if (getCount(out uint count) != NvmlSuccess)
	return new NvmlProbeResult([], driverVersion, cudaMajorVersion);
	var devices = new List<NvmlGpuSnapshot>();
	for (uint index = 0; index < count; index++)
	{
	if (getHandle(index, out IntPtr device) != NvmlSuccess ||
	getMemory(device, out NvmlMemory memory) != NvmlSuccess ||
	memory.Total == 0)
	{
	continue;
	}
	string? name = ReadDeviceString(device, getName, DeviceNameCapacity);
	if (string.IsNullOrWhiteSpace(name))

Eligibility then compares that global driver value with the runtime CUDA version and reports CudaCapabilityTooLow:

openclaw-windows-node/src/OpenClaw.Shared/Inference/Catalog/LocalInferenceEligibility.cs

Lines 121 to 138 in 31dffdb

	if (!Version.TryParse(gpu.DriverVersion, out Version? driverVersion) ||
	driverVersion < MinimumNvidiaDriverVersion)
	{
	return UnsupportedCandidate(
	gpu,
	LocalInferenceEligibilityFailureCode.DriverTooOld,
	totalMemoryBytes,
	freeMemoryBytes);
	}
	if (gpu.CudaMajorVersion < runtime.CudaVersion.Major)
	{
	return UnsupportedCandidate(
	gpu,
	LocalInferenceEligibilityFailureCode.CudaCapabilityTooLow,
	totalMemoryBytes,
	freeMemoryBytes);
	}

The CUDA driver API version and a GPU's compute capability are different facts. The current code does not call the per-device nvmlDeviceGetCudaComputeCapability API and therefore does not prove that each selected GPU architecture is supported by the pinned llama-server CUDA build.

NVIDIA documents the per-device query here:

https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html

Impact

A sufficiently new installed driver can satisfy the global CUDA-version check without establishing that a particular GPU is compatible with the architecture set supported by the pinned runtime artifacts. On mixed-generation systems, eligibility can select a device without explicit per-device compatibility evidence and discover the incompatibility only after downloading and launching the runtime.

Expected behavior

• Record the CUDA driver API version separately and name it accordingly.
• Probe compute-capability major and minor for every NVML device.
• Define the minimum or supported compute capabilities for each pinned runtime variant.
• Require both a compatible driver and compatible selected device.
• Fail closed when required per-device capability evidence is unavailable.
• Keep selection and launch pinned to the same stable GPU UUID.

Tests

Add coverage for:

• Compatible driver and compatible GPU
• Compatible driver but unsupported GPU compute capability
• Mixed-generation multiple-GPU selection
• Missing or unsupported NVML compute-capability query
• Clear separation of driver-version and device-capability failure reasons

Tracking requested by @joelagnel.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 21, 2026, 4:18 PM ET / 20:18 UTC.

Summary
Keep open. This is a concrete compatibility requirement for active Local AI work, while current main and the latest release contain no Local AI/NVML inference implementation; maintainers must choose the missing-query hardware policy.

Reproducibility: no. Current main and the latest release have no Local AI/NVML inference surface to exercise; the report instead provides an audit path against an unmerged feature revision.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.

Maintainer decision needed

Question	Recommendation
Should the initial Local AI release fail closed when NVML cannot provide per-device CUDA compute capability, instead of retaining a driver-version-only fallback?	Fail closed on missing capability data: Require per-device capability evidence for CUDA runtime selection and report a clear unsupported-hardware reason.

Why: This choice determines which NVIDIA systems may run the new local runtime and cannot be inferred from the report alone.

Next step
Prior review context identifies active Local AI implementation work as the right landing boundary, while the missing-query policy requires maintainer direction; an independent repair PR would risk duplicating it.

Review details

Best possible solution:

Implement per-device capability qualification in the active Local AI work, with separately named driver data, pinned-runtime support rules, stable UUID selection, and coverage for compatible, incompatible, mixed-generation, and unavailable-query cases.

Do we have a high-confidence way to reproduce the issue?

No. Current main and the latest release have no Local AI/NVML inference surface to exercise; the report instead provides an audit path against an unmerged feature revision.

Is this the best way to solve the issue?

Unclear. Per-device capability evidence is the narrowest architecture check, but maintainers must first decide the compatibility behavior when the NVML query is unavailable.

AGENTS.md: found, but no applicable review policy affected this item.

Remaining risk / open question:

• Failing closed when NVML cannot expose compute capability changes the supported-hardware policy, so the active Local AI work needs an explicit maintainer choice and mixed-GPU validation before landing.

Codex review notes: model internal, reasoning high; reviewed against d3ed6c78bf14.

Label changes

Label justifications:

• P2: This is a bounded local-inference eligibility requirement with limited blast radius outside the unshipped Local AI path.
• impact:other: The issue concerns hardware and pinned-runtime compatibility, which does not fit a more specific owned impact category.

Evidence reviewed

What I checked:

• Current-main boundary: Current main has no tracked NVML, Local Inference, CUDA eligibility, or GPU implementation files, so it cannot already provide the requested per-device qualification. (d3ed6c78bf14)
• Reported feature-revision boundary: The issue identifies the affected probe and eligibility files in a revision distinct from current main, indicating that this audits active Local AI work rather than a shipped default-branch feature. (src/OpenClaw.Shared/Inference/NvmlHostHardwareProbe.cs:174, 31dffdb0dbd9)
• Release check: The latest release tag is v2026.7.1 at this commit, and its tracked tree also has no NVML or Local Inference eligibility surface. (4206611f2b2e)
• Feature-history availability: A history search found no Local AI, NVML, CUDA, or GPU feature commit on the locally available default-branch history; the prior review context records an active Local AI implementation PR, but its URL was not supplied. (d3ed6c78bf14)
• Maturity check: The required maturity helper could not evaluate this item because qa/maturity-scores.yaml is absent from this checkout; no maturity label is applied. (qa/maturity-scores.yaml, d3ed6c78bf14)

Likely related people:

• Joel: Recent work on the shared subsystem is commit 3880b62; no Local AI feature history is present on current main. (role: recent adjacent OpenClaw.Shared contributor; confidence: low; commits: 3880b62dd517; files: src/OpenClaw.Shared)
• joelagnel: The issue identifies the exact proposed hardware-probe and eligibility boundary, but no merged feature-history ownership is available locally. (role: reporter and implementation tracker; confidence: low; files: src/OpenClaw.Shared/Inference/NvmlHostHardwareProbe.cs, src/OpenClaw.Shared/Inference/Catalog/LocalInferenceEligibility.cs)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.