gke-gpu-nic-networks verifies Network objects exist, not that the TCPXO fabric is usable

[yuanchen8911]

Scope note. Out of the current #2217 plan (#2295 / #2296 / #2297). This issue is independent of it: the gke-gpu-nic-networks check asserting a control-plane census rather than fabric usability is true regardless of who authors the training runtime. References below to observation or drift checks may point at closed issues.

Answers D-E (AICR verifies, provisioning owns) for GKE GPU NIC networks. The design question is settled — this issue is about making the verification actually verify capability rather than count objects.

Problem

The gke-gpu-nic-networks deployment check (added in #2248) verifies that eight Network objects named gpu-nic* exist. That is a control-plane census: it proves the cluster can name eight GPU NIC networks, not that the GPUDirect-TCPXO fabric is actually usable.

Two states pass the check today while TCPXO is degraded or non-functional.

1. A GPU NIC PCI slot is displaced, so a node delivers 7/8 GPUs.

When the GPU node pool is provisioned with a gVNIC additional network, it takes a GPU NIC PCI slot (0000:06:00.0), leaving only 7 of 8 GPUs available to TCPXO. All eight Network objects still exist, so the deployment check passes clean. This is already a documented failure mode — docs/integrator/gke-tcpxo-networking.md covers it under "RxDM detects 7/8 GPUs" — but nothing detects it automatically. It surfaces later as reduced bandwidth or an RxDM error at runtime.

Nothing in the codebase reads node NIC topology: networking.gke.io/nic-info has zero references across validators/ and pkg/, and appears only in the docs (gke-tcpxo-networking.md:283) as a command for operators to run by hand.

2. A Network exists but is unready, or its binding is broken.

DiscoverGPUNICNetworks filters on strings.Contains(name, "gpu-nic") and nothing else. It inspects neither the object's status nor spec.parametersRef, so an unready Network, or one whose GKENetworkParamSet is missing or misconfigured, counts toward the eight. The failure message added in #2248 discloses this ("this check counts Network names; it does not verify the GKENetworkParamSet binding or readiness"), so the limitation is visible — but it is not tracked.

3. The Network names do not match what the shipped runtime references.

DiscoverGPUNICNetworks filters on the gpu-nic substring and the check compares only the count (gke_gpu_nic_networks.go:93, len(gpuNICs) < RequiredGPUNICNetworks). It prints the discovered names as evidence but never compares them to anything.

A cluster publishing aicr-demo2-gpu-nic-0..7 therefore has eight substring-matching networks and passes clean, while a runtime referencing gpu-nic0..7 names objects that do not exist. Both naming conventions are real: AOR generates gpu-nic<N> across the DGXC fleet, and AICR's own demo clusters used the <cluster>-gpu-nic-<N> form — which is precisely why discovery filters by substring rather than assuming names.

This case becomes load-bearing with #2296, which ships fabric wiring in the ClusterTrainingRuntime referencing network names by value. Note the failure shape: missing networks fail loudly at the deployment phase, but mismatched names fail only when a workload actually runs — a late hard failure — a Pod referencing a nonexistent GKE Network fails creation/scheduling. The gate moves it earlier and makes remediation explicit; it does not convert a silent success into a failure.

Why one issue

All three are the same defect in different clothes: the gate counts objects rather than verifying capability. They share a code path (validators/internal/gkenet, relocating to pkg/gkenet was proposed under the now-closed the closed #2273), a check, and a phase, and a single change is likely to address them together.

Proposed direction

Move the check from existence-counting toward capability-verification:

• Verify node-level NIC topology against the expected GPU NIC count. networking.gke.io/nic-info is the existing signal and is already the documented manual remedy for case 1.
• Verify Network readiness and the GKENetworkParamSet binding, or deliberately retain the limitation with rationale recorded in code.
• Verify that the network names the shipped runtime references are actually present, not merely that eight substring-matching networks exist. Both halves are already in hand: DiscoverGPUNICNetworks yields the live names and the rendered runtime yields the referenced ones.

Worth deciding as part of this: whether the node-topology check belongs in the same gke-gpu-nic-networks check or a sibling, given one reads cluster-scoped CRs and the other reads node annotations.

Acceptance criteria

• A GPU node pool whose gVNIC displaces a GPU NIC PCI slot (7/8 GPUs usable) fails validation at the deployment phase, with a message naming the displacement
• An unready Network, or one with a missing/broken GKENetworkParamSet binding, either fails the check or is documented as an accepted limitation with rationale in code
• A cluster whose GPU NIC networks are named differently from what the shipped runtime references fails at the deployment phase, with a message naming the mismatch and the override needed to correct it
• The check remains inert where TCPXO does not apply — it still skips when the recipe does not declare gke-nccl-tcpxo, and still blocks (never skips) on RBAC, timeout, or transport errors
• Any new operator-facing failure message names the remediation, consistent with the existing one

Non-goals

• Making RequiredGPUNICNetworks SKU-aware. It is fixed at 8 for the a3-megagpu-8g shape, and only h100-gke-cos-training declares gke-nccl-tcpxo today. The general "fixed value applied to SKU-agnostic recipes" problem was considered and closed NOT_PLANNED in nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256, where the conclusion was that fabric/transport class, not GPU count, is the axis that matters.
• Wiring multi-NIC into the shipped kubeflow-trainer runtime — that is [Epic]: No fabric-wired runtime ships, so validation can only measure a fixture #2217, a distinct gap that survives a complete fix to this one.

Context

Follow-up to #2248, which added the deployment-phase detection and recorded both of these as out of scope. Original prerequisite gap: #2216.

Relationship to #2217

Updated — the earlier framing has been reversed by #2217's decisions.

#2296 now ships its wiring in a named full-node runtime that users opt into by name, so there is no default to flip and case 3 is not a merge blocker for it. What case 3 gates is the contents: the named runtime must not reference networks the cluster does not publish. Under D-B those names come from observation, so a mismatch is a discovery failure rather than a wrong default.

The earlier text here said #2296 "is contingent on case 3 being closed" and that "the decision recorded there is to enable the wiring by default." Both are now wrong.

Also corrected: the earlier claim that gpu-nic<N> is "correct for every cluster in the DGXC fleet but not universal" understated the problem. gkenet.go states that network names are "chosen by whoever provisions the cluster, not assigned by GKE — Google's own sample manifests name them vpc1..vpc8", and calls the substring a "documented provisioning REQUIREMENT, not an observation." So it is one provisioning stack's convention, not a fleet-wide fact — which is exactly why case 3 must compare names rather than count objects.

Sequencing

Wave 1 — startable now, nothing blocks it. Informs #2296 but does not gate it.

Note: the closed #2279 was closed. Case 3 and the closed #2279's recipe-versus-cluster drift check both compare what the recipe declares against what the cluster observes. Read both before building either, or the work is done twice.

#2290 resolves a floor this check enforces — RequiredGPUNICNetworks = 8 fails a correctly provisioned 4-NIC TCPX cluster.

[yuanchen8911]

Promoted to P1 / Ready: this is now a release-blocking dependency of #2274.

Added case 3 to the body — the check compares network count, not names (gke_gpu_nic_networks.go:93), so a cluster publishing aicr-demo2-gpu-nic-0..7 passes clean while a runtime referencing gpu-nic0..7 names objects that do not exist. Same defect class as the two cases already here: the gate counts objects rather than verifying capability.

#2274 ships fabric wiring in the shipped ClusterTrainingRuntime with network names as a static default, enabled by default. That decision is only safe if this check compares names — otherwise a mismatch passes validation and fails when a workload runs, which is the same late-and-silent shape as the defect #2217 exists to fix. If case 3 cannot land in the same release, #2274 ships with its wiring disabled by default instead.

[yuanchen8911]

Recommend splitting case 3 into its own issue, and gating #2274 on that alone.

#2274 depends only on case 3 (comparing referenced network names against published ones). This issue's acceptance criteria require all three cases, including case 1 — node NIC topology via networking.gke.io/nic-info, which this issue itself notes has zero references anywhere in validators/ or pkg/. That is greenfield work of unknown size.

As it stands, a P1 with a written, well-understood fix is blocked behind two harder unwritten ones, and #2274's documented fallback (ship the wiring disabled by default) becomes the likely outcome by default rather than by choice.

This cuts against the "Why one issue" argument above, which is sound on the merits — all three are the same defect class and share a code path. The counterweight is only the dependency: case 3 has a release-blocking consumer and the others do not.