feat(mocknvml): decode GPU fabric health for nvidia-smi -q

[giuliocalzo]

Fixes #677

Summary

Every row of the nvidia-smi -q Fabric → Health block read N/A, so a consumer could not tell a healthy fabric from an unknown one, and degraded-fabric handling could not be exercised at all. FabricInfo already carried a HealthSummary that the bridge copied into the v3 struct, but nothing set it — and a zero summary makes nvidia-smi treat the whole block as unreported, which also made the existing fabric.health_mask key silently ineffective.

• Health is configured and decoded. A fabric: block with no health keys means a healthy fabric, so the shipped Grace-Blackwell profiles report Summary : Healthy / Bandwidth : Full. fabric.health faults one named condition at a time (degraded_bandwidth, route_recovery, route_unhealthy, access_timeout_recovery, incorrect_configuration: <name>), the Summary row is derived from the conditions unless pinned with fabric.health_summary, and a raw fabric.health_mask stays available as an escape hatch — an explicit 0 still means "the driver reported no health".
• Runtime injection. nvml-mock-ctl fabric-health --gpu <idx> route_unhealthy degrades a fabric under a running workload and ... fabric-health --gpu <idx> healthy restores it without a reset, so the device's other overrides survive. It is authoritative like throttle: the named conditions are set and the rest cleared, so one condition flips its own row and only its own row. Injecting a fault also releases a profile-pinned summary, since a Healthy summary next to a faulted condition is not a state hardware can be in.
• v1/v2 callers are unaffected now that the summary is non-zero: healthSummary starts exactly where a v2 buffer ends, pinned by a unit test alongside the shared-field offsets the v2 branch's reinterpret cast depends on, plus a check that a v2 tag never takes the v3 path.
• e2e assertions read the -q -x document (following test(e2e): assert nvidia-smi state from -q -x instead of scraping text #658) rather than the human table. The fabric health elements are decoded in schema.go, and the checks compare a whole block per GPU, so one observation covers both the degraded device and its untouched neighbours — a fault applied to the wrong device fails rather than passes.

Verification against a real nvidia-smi

Built the mock library from this branch, layered it onto the mock image (which bundles the real nvidia-smi 580.65.06) and ran the binary against the gb200 profile config.

Before:

        Health
            Summary                       : N/A
            Bandwidth                     : N/A
            Route Recovery in progress    : N/A
            Route Unhealthy               : N/A
            Access Timeout Recovery       : N/A
            Incorrect Configuration       : N/A

After, with nvml-mock-ctl fabric-health --gpu 1 route_unhealthy applied — GPU 0 untouched, GPU 1 degraded, from the same -q -x document (this capture is the new testdata/qx-gb200-fabric-degraded.xml fixture):

<health>                                    <health>
  <summary>Healthy</summary>                  <summary>Unhealthy</summary>
  <bandwidth>Full</bandwidth>                 <bandwidth>Full</bandwidth>
  <route_recovery_in_progress>False</..>      <route_recovery_in_progress>False</..>
  <route_unhealthy>False</route_unhealthy>    <route_unhealthy>True</route_unhealthy>
  <access_timeout_recovery>False</..>         <access_timeout_recovery>False</..>
  <incorrect_configuration>None</..>          <incorrect_configuration>None</..>
</health>                                   </health>

fabric-health healthy restored it; no_partition rendered <incorrect_configuration>No Partition</incorrect_configuration> with Summary : Unhealthy on the target GPU only; degraded_bandwidth rendered Bandwidth : Degraded with Summary : Limited Capacity.

Note on the Partition Assigned acceptance criterion

There is no row or element to assert. The string Partition Assigned does not exist in the 580.65.06 nvidia-smi the image bundles, and neither PARTITION_ASSIGNED_TRUE nor _FALSE in the health mask makes it appear; the reference tray prints it because it runs 580.173.02. The mock reports the field as NOT_SUPPORTED, which is what that healthy tray reports and what a newer nvidia-smi renders as N/A, and schema.go deliberately decodes no element whose name this package cannot pin against a captured document. Documented in docs/configuration.md.

Test plan

• make test — engine health resolution written TDD; mask field shifts/widths and summary constants pinned against go-nvml; mockctl.FabricHealthPatch and the fabric-health CLI covered; the XML check covered against both the new degraded fixture and the pre-[Feature]: decode GPU fabric health so nvidia-smi -q reports Summary, Bandwidth, and route state instead of N/A #677 capture already in testdata, which it must reject
• pkg/gpu/mocknvml/bridge/fabric_tail_test.go — struct-tail boundary, shared-field offsets, version dispatch
• make test-mocknvml-bridge — C ABI harness exercises v1/v2/v3 against a healthy and a faulted device (51 passed)
• make lint-fix — 0 lint issues (govulncheck reports pre-existing Go stdlib advisories; unmodified main fails identically)
• make helm-tests — 169 tests, 15 snapshots (profile checksums refreshed)
• Real nvidia-smi 580.65.06 against the built library, as above
• New Ginkgo spec in CI: healthy baseline per profile (nvidiasmi.FabricHealth), then route_unhealthy injected on one GPU with its neighbours asserted unchanged, cleared, and a named misconfiguration. Gated on p.HasFabric(); selected by --label-filter="nvidia-smi" or "failure-injection".

Docs

docs/configuration.md gains a Fabric Health section (conditions table, summary derivation, raw mask, the Partition Assigned caveat); docs/nvml-mock-ctl.md documents fabric-health with a worked example and its reset semantics; CHANGELOG.md entry under [Unreleased].

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ArangoGutierrez]

Nothing here blocks merge.

The mask encoding is the part I looked at hardest and it holds up: each condition gets its own shift/width slot, deriveFabricHealthSummary keeps degraded bandwidth at limited-capacity instead of promoting it to unhealthy, and mask == 0 still reports not-supported so the raw escape hatch keeps its old meaning. fabric_tail_test.go is the test I would have asked for - pinning healthSummary at exactly the end of a v2 struct, plus a v2 tag never classifying as v3, is what stops the now non-zero summary from writing past a v2 caller's buffer, and the shared-offset table covers the reinterpret cast the v2 branch depends on. On the e2e side, comparing a whole block per GPU out of one -q -x document with the neighbours pinned to healthy means a fault applied to the wrong device fails instead of quietly passing. One thing on the runtime injection path is worth picking up, inline - it does not change the shape of the change, so I am not holding the review for it.

[ArangoGutierrez]

The patch releases a pinned health_summary to "auto" but leaves health_mask alone, and resolveFabricHealthMask returns a non-nil mask outright before ever looking at the health block. On a config that still pins health_mask the injection merges cleanly and is then discarded, so the command exits 0 having done nothing; with the health_mask: 0 that every shipped profile carried until this PR removed it, the derived summary comes back NotSupported and the whole block renders N/A, which is the pre-#677 behaviour this change is fixing. Worth writing health_mask as null alongside health_summary so the same reasoning that releases the pinned summary also releases the field that outranks it.