poc: Mokka + AICR pre-silicon integration preflight - #503
Draft
ArangoGutierrez wants to merge 6 commits into
Draft
poc: Mokka + AICR pre-silicon integration preflight#503ArangoGutierrez wants to merge 6 commits into
ArangoGutierrez wants to merge 6 commits into
Conversation
Runs the AICR validate suite against an nvml-mock cluster and classifies what each check actually proves when no GPU is present, so a coverage claim can be reviewed rather than asserted. The harness is built so an incomplete run degrades to "not run" rather than to a favourable number. A check the run never reported can never be recorded as passing; CTRF "pending" and "other" map to not-run rather than being laundered into a pass; a hardware-dependent check reporting green under simulation is flagged suspect; and every record carries sim or silicon provenance so the two cannot be conflated. Bucket A is split by gpuDependent. Some AICR checks are meaningful but GPU-independent (gang-scheduling is deliberately CPU-only upstream, platform-health is pure control-plane), so counting them as coverage Mokka unlocked would inflate the headline. Catalog rows carry the analytical judgment with source evidence; the harness supplies only the observed outcome and never edits a bucket. Includes the 2026-07-25 gb200 run evidence: 9 of 21 checks executed, 5 pass and 4 fail, with the raw CTRF report and validate log. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
The GPU Operator scenario checked three GPU Feature Discovery labels through NodeLabelSoft, which only prints a warning and has no assertion in any branch. GFD could have been removed entirely and the scenario would still have gone green, so the check contributed no signal. Assert the labels hard, with expected values derived from the profile rather than read back off the node. Deriving them independently is what makes the comparison discriminate: reading the labels and asserting they equal themselves would be the same theater in a different shape. Adds Profile.GFDProductName (spaces to dashes, as GFD publishes it) and Profile.MemoryMiB (from device_defaults.memory.total_bytes), plus ExpectedGFDLabels, DiffGFDLabels and WaitGFDLabels. DiffGFDLabels reports every mismatching label rather than stopping at the first, and treats an empty value as missing. Expected values are pinned to what GFD actually published on a live gb200 Mokka cluster on 2026-07-25: NVIDIA-GB200, 196608, 8. Closes NVIDIA#500 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
Customer-facing documentation for the Mokka + AICR pre-silicon integration preflight, plus the POC findings note. The framing is deliberately bounded. Preflight covers GPU Operator install, DRA and resource advertisement, NVML-facing components and control-plane behaviour. It does not cover CUDA execution, firmware and driver compatibility, NCCL / NVLink / NVLS and fabric, or TTFT and throughput, and those determine first token. AICR validation on real silicon remains the final readiness gate. The claim is that preflight reduces time to first token, never that it is hit on day one. Numbers come from the 2026-07-25 run and match the catalog exactly: 21 checks, A 14, B 0, C 4, G 3. Today 66.7 percent, reachable 81.0 percent once gaps close, and 42.9 percent is what Mokka specifically unlocks since 5 of the bucket-A checks are GPU-independent. Only 9 of 21 executed, which the docs state rather than paper over. Five diagrams in mermaid and hand-authored SVG. Text is real text, each carries a dark-scheme block, and none relies on colour alone. Scale caveat is stated everywhere it could be misread: the mock needs a real kubelet and cannot run on KWOK nodes, this POC ran on one node, so fidelity at KWOK scale is unproven and is not claimed. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
The original framing of issue NVIDIA#498 was wrong. It said Mokka lacks an nvidia-caps-imex-channels device class and proposed inventing a way to fake /proc/devices, sized M. Both the cause and the remedy were off. The DRA driver already supports mock NVML environments. Its chart exposes altProcDevices, which mounts an alternate proc-devices file and passes ALT_PROC_DEVICES_PATH, and the documented example path is literally /var/lib/nvml-mock/imex/proc-devices. Its CI builds the whole surface against nvml-mock in hack/ci/mock-nvml/setup-mock-gpu.sh: the proc-devices entries, a fabric-imex-mgmt capability file, and 2048 channel device nodes. So the gap is packaging, not capability, and it is smaller: nvml-mock does not ship that surface, and the released NGC chart v25.12.0 that this repo's e2e uses has no altProcDevices key. Both halves exist and neither is reachable from the nvml-mock chart. Size drops M to S. Reproducing the surface by hand and mounting it over /proc/devices in the container is rejected by runc ("cannot be mounted because it is inside /proc"), which is exactly why the env-var indirection exists. Evidence committed alongside the run. Buckets and coverage numbers are unchanged (A 14, B 0, C 4, G 3): a user of the nvml-mock chart alone still cannot run the check, so dra-support stays G. Only the root cause, size, and proposed fix change. Credit to Carlos for asking why the DRA driver's own CI works against nvml-mock if this were a hard blocker. It does, and that question is what surfaced the correct diagnosis. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
ArangoGutierrez
marked this pull request as draft
July 25, 2026 08:39
…ain plugin
The NVIDIA DRA driver's compute-domain kubelet plugin reads a device
major for nvidia-caps-imex-channels out of /proc/devices at startup. On
a mock node there is no NVIDIA kernel module, so the entry is absent and
the plugin aborts with "error parsing '/proc/devices': unexpected regex
match: []". That is what blocks AICR's dra-support check under Mokka.
The DRA driver already supports a substitute file through
ALT_PROC_DEVICES_PATH, wired by its chart's altProcDevices value, and
its own CI builds this surface against nvml-mock. This ships our half so
the surface comes from the chart rather than from a consumer's CI
scripts.
Adds pkg/system/mockimex/render for the proc-devices rewrite, a
render-imex-procdevices command, and an opt-in imex.mockChannels value
that also creates the fabric-imex-mgmt capability file and the channel
device nodes. Off by default, since it creates channelCount nodes per
node.
The renderer tests use the DRA driver's own parsing regex as the oracle,
copied verbatim from its internal/common/nvcaps.go. Deriving the
expectation from the consumer's parser rather than from this
implementation is what makes them discriminate: a mutation that appends
the entries after the Block devices header still contains the right
text, and four tests catch it.
Bind-mounting over /proc/devices is not a workaround; runc rejects it
("cannot be mounted because it is inside /proc"), which is why the
consumer uses an env-var indirection.
This does not by itself make ComputeDomains work: altProcDevices is on
the DRA driver's main branch but absent from v25.12.0, so no released
driver can consume the file yet. Documented in values.yaml.
Refs NVIDIA#498
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
…efect
Expanded the cluster from 2 components to 7 (cert-manager, NFD,
kube-prometheus-stack, prometheus-adapter alongside GPU Operator and the
DRA driver) and re-ran. Executed coverage moves from 5 pass / 4 fail to
6 pass / 3 fail. Bucket counts are unchanged: A 14, B 0, C 4, G 3.
The progression is the strongest evidence in the POC that the A-bucket
checks carry real signal. Each fix moved a check to its next real
requirement rather than flipping it green:
GPU Operator + DRA 5/4 baseline
+ cert-manager, NFD, Prometheus 5/4 platform-health 8 missing
namespaces to 5;
ai-service-metrics moves to
"no DCGM time series"
+ dcgm-exporter ServiceMonitor 5/4 moves to "custom metrics
API not available"
+ prometheus-adapter 6/3 ai-service-metrics passes
A check that passed trivially against a mock could not have failed four
different ways for four different real reasons.
That progression also surfaced a defect in AICR's own recipe, recorded
in a new section 2b. The generated kind+gb200 recipe selects
ai-service-metrics as a conformance check and declares both
kube-prometheus-stack and prometheus-adapter as components, but never
wires dcgm-exporter into Prometheus, so the check cannot pass from a
clean deploy of its own bundle. Verified three ways: no serviceMonitor
block in the gpu-operator overrides, no additionalScrapeConfigs or
widened selector in the kube-prometheus-stack overrides, and no DCGM
scrape configuration anywhere in the recipe.
Not filed: NVIDIA/aicr is Mark's repo, so it is raised in the findings
note for him to confirm or correct rather than opened as an issue.
Also fixes the Dockerfile to COPY pkg/system/mockimex, without which the
image build fails on the new renderer.
Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>
This was referenced Jul 25, 2026
ArangoGutierrez
added a commit
to ArangoGutierrez/k8s-test-infra
that referenced
this pull request
Jul 26, 2026
Issue NVIDIA#506 asks the mock to stop reading as "obviously fake" on a dashboard. Half of that is telling the reader what is fake and why, which nothing in the tree did: a user could not tell that DCGM_FI_PROF_SM_OCCUPANCY is 0.75 x utilization.gpu rather than a measurement, nor that temperature moves on a completely idle node. A new Metric Fidelity section in docs/configuration.md splits every reported value into three buckets and names the mechanism for each: - Simulated: temperature, power and utilization, but only with a dynamic_metrics block, and driven by elapsed time rather than load; plus NVLink counters accruing from MOCK_NVML_EPOCH / btime. - Static: everything resolved through the effective config, changeable within one override TTL via nvml-mock-ctl. - Deliberately fixed: the per-metric fractions of utilization.gpu behind DCGM_FI_PROF_*, and the identity/topology fields baked at construction. SM occupancy and tensor-core activity describe kernels this library never runs, so deriving them from anything but the configured utilization would be fabrication, not simulation. Recorded as by design so it is not mistaken for an open gap. A fourth subsection records the actual gap: no value is workload-aware. Nothing in the deployment observes allocation -- the nvml-mock DaemonSet stages the driver tree and then sleeps, and the optional NRI plugin subscribes only to container creation and mounts the overlay read-only -- so a pod holding an nvidia.com/gpu claim moves nothing. Fractions and mechanisms are transcribed from pkg/gpu/mocknvml/engine (gpm.go gpmRatio* constants, dynamic_metrics.go, nvlink_counters.go), not estimated. NVIDIA#506 asks for this statement in docs/aicr-preflight/coverage.md, which does not exist on this base -- that tree arrives with PR NVIDIA#503. Placing it in the existing configuration reference avoids creating a competing aicr-preflight tree that NVIDIA#503 would then have to reconcile. Refs NVIDIA#506 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit c11dd412a9fd5d9bbacef36f797286830baa5b1b)
ArangoGutierrez
added a commit
that referenced
this pull request
Jul 26, 2026
…t POC (#512) * fix(image): copy all of pkg/ and cmd/ into the build The Dockerfile enumerated 8 of the 9 top-level pkg/ directories and 7 cmd/ paths, so any new package reachable from a built binary was missing at build time. go build -mod=vendor reports that as inconsistent vendoring, which names the wrong cause and costs a debugging cycle. Copy both trees wholesale. This invalidates the COPY layer on any pkg/ or cmd/ edit, which is an accepted cost: vendor/ is already copied wholesale and CI uses cache-from: type=gha. Verified with a throwaway package imported from cmd/mock-ib -- the image build fails before this change and succeeds after. Refs #508 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit c524059232d8a3e7eb222669f24a9addaf31e1ef) * fix(engine): add go vet to CI and drop two unsafe.Pointer round-trips go vet never ran in CI. It reports 8 possible-misuse-of-unsafe.Pointer findings. The //nolint:govet comments sitting on them do nothing for go vet itself, which has no //nolint mechanism -- but they are live suppressions under golangci-lint, which does analyse these CGo packages. Measured on this base: golangci-lint reports 0 govet findings with the comments present and 3 with them stripped, 3 rather than 8 only because --max-same-issues defaults to 3. HandleTable stored only a uintptr, so the validate and free paths had to convert back to unsafe.Pointer. Store the pointer alongside the device in an unexported handleEntry instead -- no public API changes, no caller changes, and both engine findings go away. The 6 remaining findings are in the CGo bridge and need the handle type itself changed; a later commit on this branch does exactly that and removes the CI exclusion introduced here, so this step is superseded within the same change set. Refs #511 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> * refactor(nvml-mock): type device handles as unsafe.Pointer, not uintptr go vet's unsafeptr check reported 6 "possible misuse of unsafe.Pointer" findings in the CGo bridge, all the same shape: the engine hands out a device handle as uintptr and the bridge converts it back with (*C.struct_nvmlDevice_st)(unsafe.Pointer(handle)). Adjudicated per site, since #511 asks for verdicts rather than a green count. All six are sound. HandleTable.Register allocates the block with calloc, so the referenced memory is libc heap the Go garbage collector never scans, moves or frees -- the hazard unsafeptr exists to catch does not apply. Separately, Lookup checks map membership before anything dereferences the address, so a pointer the table never issued is rejected rather than followed. unsafeptr cannot see either fact and reports the conversions anyway. Soundness was not the problem; the type was. uintptr is documented as an integer that does not hold a reference, and these values are pointers: isValidHandle dereferences them and C callers receive them as nvmlDevice_t. Typing them unsafe.Pointer end to end removes the conversion instead of suppressing the diagnostic. That distinction is load-bearing because bare go vet has no per-site suppression mechanism -- no //nolint, no pragma -- so the only way to keep the six sites and still gate on vet would have been excluding the whole package, which #511 rules out. The 7 //nolint:govet comments in the bridge are gone too, and it is worth being precise about what they did, because it is easy to assume they were decoration. They were not. Measured on base b0d811c, golangci-lint reports 0 govet findings with them present and 3 with them stripped, so they were live suppressions for golangci-lint's copy of the unsafeptr analyzer; bare go vet is simply the tool that ignores them. (golangci-lint caps repeats of one finding at 3 by default, which is why it showed 3 where go vet showed 8.) With the handle typed correctly there is nothing left for either tool to suppress. Public API change (Go API only). Eight engine signatures change their handle type from uintptr to unsafe.Pointer -- DeviceGetHandleByIndex, DeviceGetHandleByUUID, DeviceGetHandleByPciBusId, LookupDevice, LookupConfigurableDevice, TopologyNearestGpus, TopologyGpuSet, PendingXidEvent -- as do HandleTable's Register, Lookup and HandleFor. The bridge is the only in-tree consumer. The mock's C ABI is untouched, so nothing that dlopens libnvidia-ml.so is affected. Negative-path tests could no longer fabricate handles from integer literals (Lookup(999)); the naive translation unsafe.Pointer(uintptr(999)) would reintroduce the exact misuse being removed. They now use a real Go allocation absent from the table, which lands on the same branch. go vet ./... is clean across all 37 packages, down from 6 findings. Refs #511 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit bb199b1995c928fa4910c511c50edc590fe2efb3) * test(nvml-mock): pin the handle table's trust boundary Adjudicating the six unsafe.Pointer sites for #511 turned up an invariant that nothing pinned: what makes a device handle trustworthy is membership in HandleTable.devices, not the magic number in the block it points at. That distinction is easy to lose. The magic-number check reads like the validation step, and the map lookup above it is commented as a checkptr crash-avoidance measure, so a future reader could reasonably drop it or move it after the C call. The bridge forwards whatever pointer the C caller supplied -- every nvmlDeviceGetX(nvmlDevice_t device, ...) export reads device.handle straight out of the caller's struct -- so losing that check means any address whose first word happens to be 0x4E564D4C is accepted. The test hands Lookup a Go-allocated block carrying exactly that magic value. Mutation-checked rather than assumed: deleting the `if !ok` guard makes Lookup fall through to isValidHandle, which passes, and it then returns the map's zero value. Observed failure is actual : <nil>(<nil>) i.e. a nil nvml.Device where the null-object contract promises InvalidDeviceInstance -- every bridge caller nil-derefs instead of returning ERROR_INVALID_ARGUMENT. Restoring the guard returns it to green. Also gives TestHandleTable_ConcurrentClear a real postcondition. It previously asserted nothing at all ("Should not crash - that's the main test"). Clear() and Register() serialise on the mutex, so only two orderings are reachable and the table must end with 0 or 1 entries; a higher count means a pre-Clear entry survived. When an entry does survive it has to round-trip device -> handle -> device, which is what catches Clear() resetting one of its two maps but not the other. Refs #511 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit a3cefd6bf7edbfc7df848a4084df63fd085551c5) * ci(golang): vet every package, drop the bridge exclusion 00f7075c added the go vet step but excluded pkg/gpu/mocknvml/bridge, which is the one thing #511 rules out -- a package-level exclusion is a blanket suppression, and the issue sequences the CI gate after the list is clean, not instead of cleaning it. The bridge is now vet-clean on its own merits, so the exclusion has nothing left to hide and is gone. Worth noting the exclusion was also written as an unanchored substring match (grep -v '/mocknvml/bridge'). A future pkg/.../bridge/internal, or a sibling package whose path merely contained that text, would have been dropped from vet silently. Deleting it removes that failure mode rather than anchoring around it. The step's comment now records why bare go vet earns its place alongside golangci-lint's govet instead of duplicating it. Both reasons were measured on the pre-#511 tree rather than assumed: golangci-lint honours //nolint:govet (0 findings with the bridge's 7 comments present, findings as soon as they are stripped), and it caps repeats of one finding at 3 by default, so it showed 3 where go vet showed 8. Bare go vet cannot be silenced per site and does not cap. Verified with the exact command CI runs, no environment overrides: $ go vet ./... rc=0 $ go list ./... | wc -l 37 Refs #511 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 77cad535c63b901de764385c954edab981e776f4) * docs(nvml-mock): explain the hand-written NFD pci-10de label The kubectl label at setup.sh read as an unexplained shortcut around NFD. Reading NFD's source (v0.19.0, byte-identical to master for the files below) shows it is not a shortcut: NFD structurally cannot derive the label on a mock node. NFD's PCI source reads exactly one path, fixed at link time. source/pci/utils.go:74 resolves hostpath.SysfsDir.Path("bus/pci/devices"), and pkg/utils/hostpath/hostpath.go:30 builds SysfsDir from the private `pathPrefix` var (hostpath.go:24), settable only by linker -X. Upstream's container build passes HOSTMOUNT_PREFIX=/host- (Makefile:58,81; Dockerfile:87), so the shipped worker reads /host-sys/bus/pci/devices -- the real host /sys, hostPath-mounted read-only (worker.yaml:139-141,169-171). The published v0.19.0 binary confirms it: its embedded build info records hostpath.pathPrefix=/host-. Our rendered tree at /var/lib/nvml-mock/sys/bus/pci/devices is reachable only via libpcimocksys.so. The NRI plugin does inject it into the NFD worker -- injection is opt-out and NFD is not in an excluded namespace (pkg/nri/nvmlmock/adjust.go:189-204,105) -- but the injection is inert: the shim rewrites only paths starting "/sys/" (shim.c:27-32,61), which "/host-sys/..." never matches, and nfd-worker is linked -extldflags=-static (Dockerfile:87), so it has no PT_INTERP and ignores LD_PRELOAD entirely. `file` on the shipped binary reports "statically linked" with zero ld-linux references. Notably the label itself is correct and would be produced on real hardware. NFD's default device-class whitelist ["03","0b40","12"] (pci.go:48) admits class 0302 by prefix match (pci.go:120), GPU Operator configures deviceLabelFields:[vendor] yielding exactly pci-10de.present, and our renderer writes all five attributes NFD treats as mandatory (utils.go:31). Only visibility is missing, so the line stays. No behaviour change; comment only. Refs #505 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 0fefc8e1978b07b1d3e5c8df2e4073a2c1f0a262) * docs(nvml-mock): scope the NFD pci-10de rationale to NFD's PCI source Review follow-up to 0fefc8e1, which explained why the pci-10de label is written by hand. Comment text only; no executable line changes. The previous wording claimed NFD "structurally cannot derive it on a mock node", but every reason it then gave is a property of NFD's PCI source. NFD's *local* source reads plain feature files from a different path, and that route is open to us. Verified at the v0.19.0 tag pinned in go.mod:17: source/local/local.go:68 featureFilesDir = "/etc/kubernetes/node-feature-discovery/features.d/" source/local/local.go:257 os.ReadDir(featureFilesDir) That literal is not wrapped in hostpath.HostDir, unlike the PCI source's hostpath.SysfsDir, so it carries no /host- prefix; NFD's worker mounts the host directory at the identical in-container path: deployment/components/common/worker-mounts.yaml:22-24 hostPath .../features.d/ deployment/components/common/worker-mounts.yaml:50-52 mountPath .../features.d/ This repo already mounts that exact host directory read-write as the GFD mock's output dir (tests/e2e/gfd-mock.yaml:51-52 and :60-63, duplicated at tests/e2e/go/assets/gfd-mock.yaml:52,62). Writing a feature file there is therefore a supported way to retire the kubectl call, needing no node RBAC. That option previously existed only in a gitignored working note, so it would never have reached the repo. Also corrected in the comment: - The "full analysis" pointer sent readers to issue #505, which poses the question and does not answer it; no analysis was ever posted there. It now names the command that reaches the analysis in 0fefc8e1's message, and says plainly what #505 is. - Reasons (a) and (b) were joined by "; and", which reads as a conjunction. They are independently sufficient; reworded to say so. - Added the "as of NFD v0.19.0" vintage, plus the caveat that the deployed worker image comes from GPU Operator and may differ. - Moved "removing this line silently drops the label" from the 20th line of the comment to the top, with the mechanism after it. Verification (run against the final state): shellcheck -f gcc -s sh deployments/nvml-mock/scripts/setup.sh | grep -c . => 3, the unchanged baseline (SC2012 x2 at :36/:46, SC2129 at :166) sh -n deployments/nvml-mock/scripts/setup.sh => clean non-comment, non-blank lines are byte-identical to HEAD (diff empty), and the #504 toolkit-ready marker does not appear in the diff. Refs #505 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 8c8302de968236cce746dac6750e0d41747751e6) * feat(nvml-mock): gate the hand-written NFD pci-10de label behind a value setup.sh writes feature.node.kubernetes.io/pci-10de.present=true by hand because NFD's PCI source structurally cannot see a mock GPU (analysis in the step 7 comment, verified against NFD v0.19.0). That was unconditional, so a cluster running a real NFD got a forged label it could not reconcile, and the preStop hook deleted a key the mock may not have owned. Add nodeLabels.pciVendorPresent (default true, preserving today's behaviour). It renders MOCK_NFD_PCI_LABEL=on|off into the DaemonSet, which setup.sh and cleanup.sh both branch on, so the label is written and removed by the same switch — the mock never deletes a label it did not create. Following the MOCK_FABRICMANAGER precedent, setup.sh normalises the value and aborts on anything but on/off rather than silently disabling the label on a typo; values.schema.json types it as a boolean so a string fails at `helm template` with a pointed message instead of a raw ternary error. Document the value, its default and the reason the label is hand-written in the chart README's new "Node Labels" section, and correct the Known Limitations row: the gap is not that PCI sysfs is missing (we render it) but that NFD reads a link-time-fixed /host-sys path our shim cannot reach. Two helm-unittest cases cover both states. Removing the DaemonSet env block makes both fail (verified: 5 failures = 2 new + 3 snapshots), so they discriminate rather than merely assert presence. Refs #505 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 0a5dff3a6ad52aafdb7bb317866a5e4c0b456323) * docs(nvml-mock): correct the toolkit-ready marker rationale The comment above the toolkit-ready touch claimed that nvidia-container-toolkit writes this marker and that no other component writes it under nvml-mock. Both sentences are false. The gate is real: six operand DaemonSets ship an unconditional toolkit-validation init container that polls for /run/nvidia/validations/toolkit-ready (gpu-operator v26.3.0 assets/state-device-plugin/0500_daemonset.yaml:29,31 and five siblings). But its writer is GPU Operator's own nvidia-validator, not the toolkit. It runs as the operator-validator DaemonSet's toolkit-validation init container with COMPONENT=toolkit (assets/state-operator-validation/0500_daemonset.yaml:59-69), and Toolkit.validate() deletes the marker (cmd/nvidia-validator/main.go:1134) before re-creating it on success (main.go:1153). The validator's preStop hook removes every *-ready file on shutdown (0500_daemonset.yaml:133-136). So nvml-mock's pre-creation is not durable and validates nothing; it only lets operands clear the gate before that check has run. Say so, so nobody reads a green operand as evidence of toolkit validation. Comment only. No behaviour change. Refs #504 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 1041f10afcacbc4be5bdf86e31aad4169d0790d8) * fix(nvml-mock): stop pre-creating the GPU Operator toolkit-ready marker setup.sh pre-created /run/nvidia/validations/toolkit-ready at nvml-mock install time. That marker is not ours to write: GPU Operator's own nvidia-validator writes it from Toolkit.validate(), which deletes it (cmd/nvidia-validator/main.go:1134), runs nvidia-smi, and re-creates it only on success (main.go:1153). The operator-validator DaemonSet that runs it is deployed unconditionally. Pre-creating it let the six operand DaemonSets clear their toolkit-validation gate before any toolkit check had run, turning an ordering barrier into a no-op and making a green operand look like a validated one. That is the hollow pass in #504. The mkdir goes with it: every DaemonSet mounting that path declares hostPath type DirectoryOrCreate, and the validator does os.Mkdir itself (main.go:524). cleanup.sh's matching rm goes too. It was declared the counterpart to setup.sh step 8b, so it is now dead; worse, as nvml-mock's preStop hook it would delete state owned by another component on every restart. Validated on a live single-node Kind cluster (gb200, kindest/node v1.36.1), not from source reading alone: make e2e-gpu-operator -> rc=0 Ran 6 of 46 Specs in 466.175 seconds SUCCESS! -- 6 Passed | 0 Failed | 0 Pending | 40 Skipped With the touch gone, /run/nvidia/validations did not exist while nvml-mock was already Running. The device-plugin's toolkit-validation init container then genuinely blocked -- started 17:11:46, finished 17:11:51 -- clearing only after the operator-validator wrote toolkit-ready at 17:11:47. All operands reached Ready and the node advertised 8 allocatable GPUs. Refs #504 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 3ae94c0aabe61fab2b24818b4657d0a80227df1a) * docs(nvml-mock): narrow two overstated marker comments to what is verified Round-2 review found two comment claims stronger than the evidence behind them. Comment text only -- no executable line of either script changes. cleanup.sh: the NOTE said removing the marker would "re-arm the operand gate with nothing scheduled to re-satisfy it". This task's own live run refutes that clause. cleanup.sh also drops the nvidia.com/gpu.present node label, which recycles the operator-validator on the same preStop; the validator then rewrites the marker (observed live: 17:15:20-17:15:23, toolkit-ready rewritten at 17:15:21.542). Restores the hedge the round-2 report carried and the shipped comment dropped: probably benign in this scenario, dead code either way, hazard real where the validator does not restart. setup.sh: "every DaemonSet that mounts it declares hostPath type DirectoryOrCreate" was a false universal. Only 2 of the 6 gated operands mount the validations dir directly (device-plugin, mig-manager), plus the validator. The other four mount the parent /run/nvidia -- gfd, dcgm and mps as type Directory, dcgm-exporter with no type -- which step 8's retained mkdir supplies. No bug follows, but a reader checking GFD would find the claim did not match. Also disambiguates the three cited files that share the basename 0500_daemonset.yaml. Adds one debugging pointer: nvidia-validator's cleanup-all flag (CLEANUP_ALL, default false) os.RemoveAll's and recreates the output dir, replacing the directory inode, which the two direct bind-mounters would not follow. Unreachable by default -- CLEANUP_ALL appears nowhere in assets/ or controllers/ -- so it is framed as "check this first if an operand ever hangs here", not as a live defect. All GPU Operator citations re-verified at tag v26.3.0 (the version the comment names), not at the checkout's HEAD, which is 17 commits past it. Verification: executable-line diff vs 3ae94c0a (comments/blanks stripped): IDENTICAL non-comment changed lines in git diff -U0: 0 shellcheck -f gcc -s sh | grep -c . setup.sh 3, cleanup.sh 1 (baseline) sh -n OK on both Refs #504 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit e26e2366a200fc77496dc508de292769c3888e9b) * docs(nvml-mock): correct the CLEANUP_ALL hazard mechanism The cleanup-all debugging pointer claimed os.RemoveAll replaces the validations directory inode, leaving already-gated pods polling an unlinked inode forever. That mechanism is impossible. outputDirFlag defaults to /run/nvidia/validations (main.go:146, :313), and that path is a bind mount in every nvidia-validator container (state-operator-validation/0500_daemonset.yaml:54-55, 73-74, 96-97, 123-124, 138-139, over the DirectoryOrCreate hostPath at :142-145). RemoveAll unlinks the children, then unlinkat(AT_REMOVEDIR) on the mount point itself returns EBUSY. os.IsNotExist(EBUSY) is false, so main.go:516-520 returns the error and start() aborts before reaching the os.Mkdir at :524 — the host inode is never replaced and no bind mount dangles. The real symptom is every *-ready marker wiped, the validator exiting non-zero into CrashLoopBackOff, and gated operands hanging because the marker is genuinely absent. A reader following the old pointer would hunt a stale-inode symptom that cannot occur and walk right past a crash-looping validator sitting in kubectl get pods. Also disambiguate the preStop citation: three cited files share the basename 0500_daemonset.yaml, and one of them (gpu-feature-discovery) is only 132 lines long, so a reader grepping the bare name lands on the wrong file. Add the state-operator-validation/ prefix already carried by the sibling citations. Comment text only. The comment-stripped code body is byte-identical before and after (sha256 cd89aadb8bb6141eb554ab2e1b0d1cbd649ca97b). Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 655161f73f87f51ec95259f4a9b27c5fcdf34e81) * test(engine): pin memory.used_bytes to the effective config GetMemoryInfo and GetMemoryInfo_v2 return Total/Free/Used from a struct baked once in NewConfigurableDevice, so `nvml-mock-ctl set --gpu 0 memory.used_bytes=N` writes the override file, the engine merges it into the effective config, and the getter keeps reporting the profile value until the pod restarts. Nothing in the suite noticed: no test asserted a memory Used/Free value anywhere. These three cases fail RED on that behaviour (Used == 0 where 1 GiB was set) and pin the revert path and the legacy no-memory-block path so the fix cannot regress into caching or zeroing. Refs #506 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit 5c326f4f82653838331d766753142bf04ba8dd22) * fix(engine): read device memory from the effective config GetMemoryInfo and GetMemoryInfo_v2 returned Total/Free/Used from d.MemoryInfo, written once in NewConfigurableDevice. Every other config-derived getter in this type reads d.cfg(), the override-merged effective config; memory did not, so `nvml-mock-ctl set --gpu 0 memory.used_bytes=N` validated, wrote the override file, and was merged into the effective config while the getter kept reporting the profile value until the pod restarted. v2 was affected on the same three fields; it already sourced Reserved from d.cfg(). A shared memoryInfo() helper now resolves the triple from d.cfg().Memory, falling back to d.MemoryInfo when the device was built without a memory block so legacy/default mode is unchanged. Values are reported verbatim rather than derived: free_bytes is an explicit profile field, and recomputing it from total-used would make it dead. The BAR1 aperture stays baked at construction, so the documented v1 limitation narrows to bar1_memory rather than disappearing. Refs #506 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit da82fa458fd7b664692ad426d76f2f4037ed3400) * docs(configuration): state which metric values are simulated vs fixed Issue #506 asks the mock to stop reading as "obviously fake" on a dashboard. Half of that is telling the reader what is fake and why, which nothing in the tree did: a user could not tell that DCGM_FI_PROF_SM_OCCUPANCY is 0.75 x utilization.gpu rather than a measurement, nor that temperature moves on a completely idle node. A new Metric Fidelity section in docs/configuration.md splits every reported value into three buckets and names the mechanism for each: - Simulated: temperature, power and utilization, but only with a dynamic_metrics block, and driven by elapsed time rather than load; plus NVLink counters accruing from MOCK_NVML_EPOCH / btime. - Static: everything resolved through the effective config, changeable within one override TTL via nvml-mock-ctl. - Deliberately fixed: the per-metric fractions of utilization.gpu behind DCGM_FI_PROF_*, and the identity/topology fields baked at construction. SM occupancy and tensor-core activity describe kernels this library never runs, so deriving them from anything but the configured utilization would be fabrication, not simulation. Recorded as by design so it is not mistaken for an open gap. A fourth subsection records the actual gap: no value is workload-aware. Nothing in the deployment observes allocation -- the nvml-mock DaemonSet stages the driver tree and then sleeps, and the optional NRI plugin subscribes only to container creation and mounts the overlay read-only -- so a pod holding an nvidia.com/gpu claim moves nothing. Fractions and mechanisms are transcribed from pkg/gpu/mocknvml/engine (gpm.go gpmRatio* constants, dynamic_metrics.go, nvlink_counters.go), not estimated. #506 asks for this statement in docs/aicr-preflight/coverage.md, which does not exist on this base -- that tree arrives with PR #503. Placing it in the existing configuration reference avoids creating a competing aicr-preflight tree that #503 would then have to reconcile. Refs #506 Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> (cherry picked from commit c11dd412a9fd5d9bbacef36f797286830baa5b1b)
ArangoGutierrez
added
priority/p1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The question this answers
Agreed with @mchmarny on 2026-07-24:
An honest negative result was an acceptable outcome. Nothing here was tuned toward a favourable answer.
Answer
It does more than prove deployment against mocked APIs, but by less than the headline number suggests, and the strongest results are two real integration defects found pre-silicon rather than a green suite. Verdict: go with scope, on a narrower claim than the original pitch.
The single most persuasive evidence is not the coverage percentage. It is that coverage improved monotonically as components were added, and each fix moved a check to its next real requirement rather than flipping it green:
platform-health8 missing namespaces to 5;ai-service-metricsmoves to "noDCGM_FI_DEV_GPU_UTILtime series"ai-service-metricspassesA check that passed trivially against a mock could not have failed four different ways for four different real reasons.
Headline numbers
Real run, 2026-07-25, provenance
sim. 1-node Kind cluster, Kubernetes v1.36.1, linux/arm64 (correct for GB200, since Grace is ARM),nvml-mockgb200profile, GPU Operator with ClusterPolicyready,nvidia-dra-driver-gpuv25.12.0, AICR at06256cc8.Of 21 AICR checks: A 14 · B 0 · C 4 · G 3
Two qualifications that cut against the headline, stated up front because they matter more than the percentage:
gang-schedulingis deliberately CPU-only upstream. Counting them as coverage Mokka bought would inflate the number, so the harness tracksgpuDependentseparately.The most useful result was a failure
dra-supportfailed. The real DRA driver installs and publishes all 8 simulated GB200 devices as ResourceSlices, but the check also wants anvidia-dra-driver-gpu-controllerDeployment, which chart v25.12.0 templates only whenresources.computeDomains.enabled=true. Enabling that crashes the kubelet plugin:This took two corrections to get right, and the second one matters.
The first reading was that AICR carried a stale expectation. Checking the chart templates disproved it: the controller is gated on
resources.computeDomains.enabled.The second reading was that Mokka lacks an
nvidia-caps-imex-channelsdevice class and that closing it meant inventing a way to fake/proc/devices, sized M. That was also wrong. The DRA driver already supports mock NVML: its chart exposesaltProcDevices, whose documented example path is literally/var/lib/nvml-mock/imex/proc-devices, and its CI builds the whole surface againstnvml-mockinhack/ci/mock-nvml/setup-mock-gpu.sh.So the real gap is packaging, not capability, and it is smaller (S, not M):
nvml-mockdoes not ship that surface, and the released NGC chart v25.12.0 this repo's e2e uses has noaltProcDeviceskey. Both halves exist; neither is reachable from thenvml-mockchart. Reproducing the surface by hand and mounting it over/proc/devicesis rejected by runc, which is exactly why the env-var indirection exists.Buckets and coverage are unchanged, because a user of the
nvml-mockchart alone still cannot run the check. Tracked as #498, rewritten.On the positive side,
check-nvidia-smipassed, exercising device-plugin allocation, CDI injection, container-toolkit wiring, anddlopenresolution oflibnvidia-ml.so.1. That is exactly the chain that broke in #455, where three missing//exportsymbols madeRTLD_NOWabort while build, unit tests, and lint all stayed green.A defect in AICR's own recipe
@mchmarny this one is for you specifically, and it is not filed anywhere.
The generated
kind+gb200recipe selectsai-service-metricsas a conformance check and declares bothkube-prometheus-stackandprometheus-adapteras components, but never wires the GPU Operator's dcgm-exporter into Prometheus. No GPU time series is ever produced, so the check cannot pass from a clean deploy of its own bundle.Verified three ways against the generated recipe, since one negative grep is not evidence:
gpu-operatorcomponent'sdcgmExporteroverrides contain onlyconfig: {create: false, data: '', name: ''}. NoserviceMonitorblock, and the chart default is disabled. On the live cluster nonvidia-dcgm-exporterServiceMonitor existed until enabled by hand.kube-prometheus-stackoverrides touch only alertmanager, defaultRules, grafana, prometheus (resources, retention, storage) and prometheusOperator. NoadditionalScrapeConfigs, no widenedserviceMonitorSelector.Prometheus selects ServiceMonitors labelled
release: kube-prometheus-stack; with none created at all, no selector helps. SettingdcgmExporter.serviceMonitor.enabled=truewith that label madeDCGM_FI_DEV_GPU_UTILappear (carryingDCGM_FI_DRIVER_VERSION: 580.65.06and the GB200 model name off the mock), and adding prometheus-adapter made the check pass.I have not opened an issue on NVIDIA/aicr. It is your repo, and you may know a deployment path that configures this which the generated recipe simply does not express. If it is real, say so and I will file it properly.
Back-test: caught 2 of 4, on a proxy set
The primary back-test is pending. No captured DSX-OS GB200 NVL72 bring-up failure set exists in any searchable location, because DSX-OS configs live on internal GitLab. No failure set was synthesized. Tracked as #501, which needs a human rather than engineering.
A clearly-relabelled secondary proxy ran against real hardware-encountered failures on cloud GB200 (
p6e-gb200.36xlarge) from NVIDIA/aicr issues:operator-health,expected-resourcesresource.k8s.io/v1requirementdra-supportnvidia_uvmwedged on driver-managed nodesBoth misses are explained by category, not by a fixable gap. Four caveats apply: cloud GB200 rather than an NVL72 rack, n=4, two of the four are failures of AICR's own tooling caught by AICR's own validators (partly circular), and a devil's-advocate review recommended against running any proxy at all. That objection is recorded in the findings note because you may share it.
Scale caveat
The mock replaces
libnvidia-ml.so.1on the host and needs a real kubelet, so it cannot run on KWOK nodes. This POC ran on one node. It says nothing about GPU-stack fidelity at KWOK-scale node counts. That combination is a claim to prove, not to assert, and nothing here asserts it.Claims supported, and not
WITH_WORKLOAD=falseGap backlog
Summary issue: #502. Gaps filed: 3. Closed in this PR: 1.
nvml-mockdoes not ship the mock IMEX surface the DRA compute-domain plugin needs#500 is worth a look on its own:
NodeLabelSoftonly printed a warning, so GFD could have been deleted entirely and the GPU Operator scenario would still have gone green. It now asserts hard against values derived from the profile. It did not move a bucket, because it was a defect in our e2e rather than in Mokka, and it is reported that way.Deliberately not filed: MIG. Coverage is zero, but no AICR check requires it, so closing it would unlock nothing. Recorded so nobody refiles it as a blocker.
Diagrams
Stack layering: AICR is the workload, Mokka is the substrate
Direction matters: the common misreading is that simulation lives inside AICR. It does not.
Preflight timeline: where integration failures get found
flowchart LR subgraph WITHOUT["WITHOUT preflight"] W1["~3 months: engineers idle"] --> W2["silicon arrives"] --> W3["integration failures found HERE"] --> W4["hardware-dependent validation"] end subgraph WITH["WITH preflight"] P1["~3 months: AICR on Mokka, integration failures found and fixed"] --> P2["silicon arrives"] --> P3["hardware-dependent validation only. AICR on silicon = FINAL GATE"] endNo day-one first-token claim is drawn, because simulation does not prove it.
Five diagrams live in
docs/aicr-preflight/diagrams/, each as mermaid plus a hand-authored SVG for email and slides.Review ask
@mchmarny, the part I most want your eye on is
docs/aicr-preflight/, specifically the AICR description and where I have drawn the final-gate boundary. Those are yours to validate and I would rather be corrected now than in front of a customer.Two specific calls I would like you to check:
slinky-slurm-imex-channelis bucketed G, not C. My reasoning: whether the fabric carries traffic is hardware-dependent and stays C (that is whatnccl-all-reduce-bw-nvlsis for), but whether channel allocation hands out distinct channels is driver-surface plumbing and is simulatable. This is the closest call in the catalog and sits nearest your boundary.How to re-run
aicr recipe --service kind --accelerator gb200 --intent inference -o recipe.yaml && aicr validate --recipe recipe.yaml --phase deployment --phase conformance --fail-on-error=false --output ctrf/validate.jsonFull prerequisites in
tests/aicr-preflight/README.md. Raw run output is committed underresults/2026-07-25-gb200-kind/.Design note
The harness is built so an incomplete run degrades to "not run" rather than to a favourable number: a check the run never reported can never be recorded as passing, CTRF
pendingandothermap to not-run, a bucket-C check reporting green undersimis flagged suspect, and every record carries provenance. Both invariants are mutation-tested. The catalog carries the analytical judgment with source citations; the harness supplies only the observed outcome and never edits a bucket.Verification
go build ./...,go test ./...,go test -tags=e2eon the touched packages, andgolangci-lint run ./...all pass (0 issues). All commits are DCO signed-off and GPG signed.Not for merge as-is: this is a POC for review, and the back-test section is openly pending.