[Epic]: No fabric-wired runtime ships, so validation can only measure a fixture

[yuanchen8911]

Problem

Multi-NIC fabric binding exists only inside the performance validator, which discovers the GPU NIC networks and injects the annotations at apply time, on a runtime it builds for the test and deletes afterward. The ClusterTrainingRuntime AICR ships (torch-distributed) carries none of it.

So on a correctly provisioned cluster, performance validation passes while a TrainJob against the shipped runtime still runs single-NIC. The benchmark's bandwidth number describes a throwaway test fixture, not anything the recipe delivers.

Intra-node NVLink is unaffected, so nothing errors — the loss is silent, and every acceptance criterion can pass with the defect intact.

This epic assumes the Network / GKENetworkParamSet CRs exist and are bound. Those are an infrastructure prerequisite AICR does not manage — docs/integrator/gke-tcpxo-networking.md scopes them to "Infrastructure Prerequisites … not managed by AICR".

What is not wrong

The recipe works. gke-nccl-tcpxo installs the NCCL plugin and NRI device injector on GPU nodes; kubeflow-trainer installs the operator and CRDs. A workload that supplies the pod-level wiring gets working inter-node TCPXO on the recipe exactly as it stands. The validator is the existence proof — it adds nothing to the cluster but a namespaced TrainingRuntime CR.

A workload can express this. An earlier revision of this epic argued that TCPX(O) and NVLS wiring is not workload-expressible, and built a large design on that basis: an ownership test, four named runtimes, observed-attachment discovery, artifact selection, render-time gating. That premise was false.

It is true at the TrainJob level — ContainerPatch exposes only Name, Env, VolumeMounts, SecurityContext, so a TrainJob cannot add the sidecar container. It is false at the Runtime level: TrainingRuntime is a normal namespaced CRD, and a user can author a wired one exactly as the validator does.

Seven issues built on that premise have been closed. The reasoning is preserved on them and in the appendix below.

What is actually wrong

1. Nobody has written down what a workload must do.	The docs publish the pod-level wiring only as a raw kind: Pod. There is no TrainingRuntime or TrainJob example in the repo. → #2295
2. Validation measures a test fixture.	The benchmark applies its own runtime and deletes it, so the bandwidth number describes nothing AICR delivers. Its install/teardown decision also comes from live cluster state, so a stray trainer is reused silently and the same recipe behaves differently. Applicability is already recipe-derived (supportedNCCLCombinations) and must not be changed — the nccl-all-reduce-bw* constraint is declared on six overlays and zero -kubeflow ones. → #2297
3. There is no artifact for validation to measure.	Fixing (2) properly requires something to derive the benchmark runtime from. → #2296

Why ship a runtime at all

Since a workload can author one, shipping is a choice rather than a necessity. Two reasons hold:

• Validation must measure what the recipe offers. With a shipped runtime, the benchmark derives its wiring from it and the number becomes evidence about the delivered artifact. Without one, the measurement is permanently about a test fixture.
• The daemon/plugin version pairing is a cluster fact AICR owns. gke-nccl-tcpxo pins plugin v1.0.15; Google publishes a coupled release pair with directional upgrade ordering (plugin first), not strict one-to-one compatibility. AICR's own validator pins v1.0.20, which is stale and uncoupled — nothing ties it to the installer version the recipe ships, so it drifts unnoticed.

That is the same test the runtime already passes for nodeSelector and tolerations: workload-expressible, but a cluster property the user cannot know, identical for every job.

Children

In plan

Issue	Role
#2295	Document the workload fabric contract — GKE TCPXO example plus the EFA/RDMA contract. Interim, lands first, rewritten when #2296 ships
#2296	Ship torch-distributed-tcpxo (framework torch) plus the required network-name recipe value it depends on — a typed recipe carrier, fail-closed enforced in Go, and the correct daemon/plugin pairing
#2297	Validator: recipe-driven install/teardown and evidence classification (applicability unchanged), then derive the benchmark runtime from the shipped one

Order — three of four PRs can start now

#2295        docs + example                    ── independent
#2297 (3.1)  recipe-driven install/teardown    ── independent
#2296        ship torch-distributed-tcpxo
   └──►      #2297 (3.2) derive benchmark from it

Only the derivation waits. The MPI benchmark runtime already works against the AICR-deployed trainer — trainer_lifecycle.go recognises the recipe's Helm layout — it discovers the live namespace and locates both the controller and JobSet Deployments by label, which is what makes it layout-independent (the pinned chart actually sets jobset.fullnameOverride: jobset rather than deriving from the release) — so on a -kubeflow recipe the probe leaves the deployed trainer alone and runs against it. That path is exercised today.

Scope 3.1 honestly. Its install behaviour barely moves: -kubeflow recipes already use the deployed trainer, and base recipes on a clean cluster already self-install. Two cases change: a base recipe with a stray trainer (silently reused today, now reused deliberately rather than incidentally), and a declared-but-missing trainer (self-installed today, now fails — masking a broken Kubeflow deployment is the defect being corrected). Evidence classification is decided separately, by runtime source (see #2297), not by whether the recipe declares Trainer. What 3.1 actually fixes is the decision source and the reporting semantics — the same recipe's evidence classification stops depending on what is already installed (execution still differs: a complete stray installation is reused rather than duplicated), and a result carries an explicit provenance class — keyed on runtime source (#2297), not on Trainer ownership — rather than reading as validation of a delivered artifact by default. It also puts in place the branch 3.2 fills. Applicability does not change at all — it stays with supportedNCCLCombinations. And the declared-but-missing case is a behavioural change: self-installed today, failing under the new policy.

Out of plan — real, but not this work

Issue	Why separate
#2277	GB200 NVLS/IMEX — the one case where the "cannot express" argument genuinely holds: a TrainJob cannot set pod-level spec.resourceClaims, and container claims are erased on the apply path
#2290	GKE TCPX (A3 High/Edge) — a recipe-selection defect: SKU-agnostic recipes pick TCPXO artifacts for shapes that cannot use them. Fails loudly today
#2265	gke-gpu-nic-networks asserts a control-plane census, not fabric usability — passes while a displaced PCI slot leaves 7/8 GPUs attached

Closed

Issue	Why
#2274, #2278, #2288	superseded by #2296, #2297, #2295
#2293, #2294	render-gating and artifact selection served the falsified premise
#2279	partially — artifact selection was falsified, but its network-name observation survives and is now the later autofill path for #2296's required network-name value — #2296 chose a required recipe value, and observation would populate the same field
#2292	subsumed — deriving the benchmark from the shipped runtime makes it behavioural proof
#2291	B200/GKE discovery — speculative; no pending implementation to route
#2275, #2276, #2280	EFA, RDMA and GPU sizing are workload-owned; the remedy is documentation, now #2295

Notes for whoever picks this up

• The Trainer 2.2.0 → 2.3.0 bump is a scheduled correctness event. The claim-erasure trigger differs: the runtime's claim is destroyed by any non-nil resourcesPerNode, on both versions. v2.2.0 has two paths — core/trainingruntime.go assigns the TrainJob's ResourceRequirements wholesale on non-nil, so the runtime's claim goes and the TrainJob's survives; jobset/builder.go then fires on limits or requests and drops the TrainJob's too. The runtime-owned claim is what NVLS depends on, which is why the rule is to omit the field entirely. Read GB200 NVLS/IMEX: pod-level resourceClaims in the named runtime, EFA combined on EKS #2277 at the bump.

• The frameworks differ for a reason. The shipped runtime is torch because that is what users run; the benchmark is MPI because all_reduce_perf requires MPI bootstrap and the bandwidth floors are calibrated to its busbw methodology. Do not converge them — Performance validation measures a test fixture, not the runtime the recipe ships #2297 copies the wiring, not the launcher.

• This plan's load-bearing premises — re-check them, they are cheap. Each would change scope if wrong:

  • The daemon/plugin pairing is a Google-published coupled pair. Verified against their release notes, but Google publishes no machine-readable artifact, so it can drift — which is why Ship a TCPXO-wired GKE runtime with a required network-name value #2296 asks for a mechanical coupling test.
  • RecipeConfiguration accepts an ordered mapping as cleanly as accounting.go suggests. Unverified; this is why Ship a TCPXO-wired GKE runtime with a required network-name value #2296 prototypes the carrier first. If it does not fit, the required-value decision reopens.
  • ContainerPatch's four fields hold across the Trainer 2.2.0 → 2.3.0 bump. Verified on both today, but the bump is already flagged above as a scheduled correctness event.

• Why premise-checking is worth the habit. The original version of this epic — sixteen children, five design decisions, a naming scheme — rested on "fabric wiring is not workload-expressible." The disproof was in the repository the whole time: validators/performance/testdata/h100/gke/runtime.yaml is a namespaced, user-authorable TrainingRuntime carrying the full wiring, applied and torn down on every performance run — and the epic was already citing that file as the source of the benchmark fixture. Checking required no upstream research and no cluster, only reading a file already in hand. The check cost nearly nothing; skipping it cost sixteen issues.

• Before applying a fix, ask what the current text is doing correctly that the fix might stop doing — a distinct question from "does the fix address the finding." Three fixes in this epic passed the second and failed the first.

• Verifying a decision — check it appears in exactly one operative place. The reversed-term grep below catches survivors that repeat the old wording. It misses survivors that encode the old rule in different language — which is how a superseded binary evidence split survived in two tables and an acceptance criterion while the corrected three-source mapping sat elsewhere in the same issue. Grep every mention of the decision's subject and confirm they all trace to one definition. That is a uniqueness check rather than a term search, so it does not depend on knowing how the old rule was phrased — the thing you cannot know when the phrasing is what changed.

• Verifying a reversal — use a negative check. When a decision here reverses, the fix reliably lands in the issue where it was argued, and the issues that merely reference it go stale. That happened in four consecutive rounds, three times in this epic body. Confirming the new wording is present does not catch it — a positive check only confirms the edit you already know you made. Grep every issue in the set for the reversed term instead; that does not depend on knowing which documents carry the claim, which is exactly the model that is wrong when this fails. Add the old term to the list at the moment of reversal, then grep — kept that way the list is a byproduct of reversing a decision rather than something to remember to maintain. Reversed so far: funcmap in scope, OwnedPaths applies, applicability changes, log-only diff, option 8 viable, divergence impossible by construction, cluster-only mitigation.

• A note on the appendix. The design record below is history: it argues from the premise this epic has since falsified. It is kept because the upstream facts, version matrices and API surfaces it establishes remain accurate and were expensive to gather. Do not treat its conclusions as current.

---

Open decisions

Note on issue numbers below. #2287 (named runtime artifact) and #2273 (shared plumbing) were later consolidated into #2274. References to them in this section are left as written — they record what was proposed at the time, and rewriting them would falsify the history.

Recorded as options, not picks. Several are coupled — the first one constrains most of the others.

D1. UNRESOLVED — replacement is confirmed; no supported user path preserves container resources

Settled against upstream source — but the mechanism differs by version, and only the outcome is common.

• v2.2.0 (pinned): trainingruntime.go assigns the whole Resources struct; jobset/builder.go assigns via generated WithResources, whose requirements never carries Claims.
• v2.3.0: resource application left builder.go entirely — Builder.Trainer() now touches only image, command, args and env. The work moved to trainingruntime.go, and it is a strategic merge (MergeResourceRequirements), not a reconstruction. Its base is built from limits and requests only and its write-back carries only those two.

Claims are dropped on both once the path fires — the trigger differs: v2.2.x only on limits or requests, v2.3.0 on any non-nil resourcesPerNode. Do not cite the builder mechanism when reasoning about v2.3.0. Both gates key on container name node and the trainjob-ancestor-step: trainer label, which AICR's manifest matches exactly.

Three corrections to the earlier framing. The first two soften it; none rescues container resources:

• Replacement is conditional, not unconditional. A TrainJob that omits resourcesPerNode leaves the runtime's container Resources — including Claims — completely intact.
• env merges rather than replaces (apply.UpsertEnvVars, upserted by name, always). Every NCCL_*, FI_*, and LD_LIBRARY_PATH the runtime sets survives, even when the user also sets env. AICR's documented TrainJob sets no env at all.
• Replacement is scoped to the single container named node. Sidecars and init containers — the tcpxo-daemon included — are never touched, resources and all.

Per-fabric consequence:

Fabric	Under resourcesPerNode	Notes
GKE TCPXO	pod scaffold survives; activation does not	Needs no container resources; annotations + NRI survive. The erasable piece is the worker command sourcing nccl-env-profile.sh. An env entry cannot execute that script — only transcribe a static snapshot, which the validator deliberately avoids so settings track the installed TCPXO version. Superseded: see D1a below — deliverable via direct env from Google's published list, version-coupled to the pinned installer
EFA	erased	Extended resources exist only in container requests/limits; there is no pod-level form
RDMA	erased	Same
NVLS	erased, subtly	Pod spec.resourceClaims survives, but the container resources.claims entry that injects the device does not — the claim is allocated and consumed by nobody

Option dispositions:

• D1a — viable, and activation is deliverable. GKE's pod-level attachment scaffold survives replacement. Command-based profile activation does not — an env entry cannot execute nccl-env-profile.sh. But that does not make activation unresolved:

  • Google publishes the variable set as data and presents it as the primary form. The GPUDirect-TCPXO documentation lists all 26 variables under "set them as environment variables to optimize performance", and offers the script only as an optional shortcut — the inverse of the framing used earlier here.
  • envFrom survives every TrainJob override. Verified in upstream v2.2.0: Builder.Trainer() mutates exactly Image, Command, Args, Resources, and Env on the node container, and never touches EnvFrom (its only WithEnvFrom calls target the initializer containers). AICR uses envFrom nowhere today, so this is unused headroom. Delivery: a ConfigMap mounted via envFrom, generated at bundle time from the pinned plugin version. Caveat: a Pod can only reference a ConfigMap in its own namespace, so a cluster-scoped runtime referencing one requires either a fixed namespace contract, namespace replication, or direct env entries in the runtime instead. Direct env is the simpler answer and carries the same version-coupling requirement. Kubernetes precedence is correct — container env overrides envFrom, so a user's merging spec.trainer.env still wins.
  • NCCL_CONF_FILE is only a partial channel (NCCL >= 2.23). It reaches variables NCCL itself reads, but LD_LIBRARY_PATH and CUDA_VISIBLE_DEVICES — both in Google's required list — are consumed by the loader and CUDA before NCCL init. Not a sole mechanism.

  Residual risk is drift between the pinned installer image and the transcribed list. Google publishes no versioned machine-readable artifact, so AICR must assert the coupling — extract the list from the installer image at build time, or pin-and-test. That is a testable, BOM-adjacent problem AICR already solves for images, not an unresolved design question.

• D1b (admission layer) — technically works, but means shipping a mutating webhook in the creation path of every TrainJob, whose failure mode is blocking all TrainJob creation. Against the repo's operational-simplicity principle; argue it down unless everything else fails.

• D1c (users repeat resources) — honest, and now one of only three surviving options for EFA/RDMA. Its cost is that users must know per-cluster values, and EFA count varies by instance family (4 on p4d, 32 on p5), so the contract cannot be a single documented constant.

• D1d (upstream change) — better supported than assumed. Upstream flags its own mechanism as unsatisfactory at builder.go:143-144: "TODO (andreyvelich) … Eventually, we should find better way to propagate resources from TrainJob to JobSet." Cite that.

• D1e (pin/gate on version) — retired. Presumes some version merges; nothing suggests one does.

• D1f (pod-level forms) — retired. Good hypothesis, pays off for no fabric: EFA and RDMA are extended resources with no pod-level form, and NVLS needs the container claim entry specifically.

D1g. REJECTED — runtimePatches cannot patch container resources

Verified against upstream v2.2.0 source. ContainerPatch exposes exactly four fields:

type ContainerPatch struct {
    Name            string
    Env             []corev1.EnvVar
    VolumeMounts    []corev1.VolumeMount
    SecurityContext *corev1.SecurityContext
}

There is no resources field. runtimePatches does use a real strategic merge, but it can only reach fields that restricted type exposes.

A submitted resources block is silently pruned, not rejected. The TrainJob CRD carries no preserveUnknownFields and no x-kubernetes-preserve-unknown-fields, so structural-schema pruning is active: a user who writes resources: under a runtime patch gets no error — the field vanishes and the job runs unwired. That is materially worse than rejection, and it is another instance of exactly the silent-failure class this epic exists to eliminate.

The Env restriction on the node container is enforced in Go validation, not merely documented (jobset.go: "must not have envs for the dataset-initializer, model-initializer, node containers"). An earlier revision of this entry said it was unenforced; that was wrong. It also does not bind in practice, because spec.trainer.env already provides a supported merging path via apply.UpsertEnvVars. Precisely: runtimePatches cannot adjust the node container's resources at all; its env is nominally off-limits there but reachable through spec.trainer.env.

What survives of the idea: omitting resourcesPerNode does preserve the runtime's resources, including Claims. What dies is the second half — there is no merge-preserving per-job override. Users either accept the runtime's fixed resources, or set resourcesPerNode and lose the fabric. There is no middle path in v2.2.0.

Consequence at the time: EFA and RDMA appeared to need D1c, D1b, or D1d. Superseded by D-A — those fabrics are workload-expressible, so none of those options is needed; they were closed rather than solved. NVLS still needs D1b or D1d for its container claim. The #2280 prerequisite claim is withdrawn, and #2280 itself is closed.

D1 therefore remains open. The mechanism is settled; what to do about it is not. Surviving options: D1b (admission layer), D1c (users repeat resources), D1d (upstream merge). An earlier revision marked D1 resolved via D1g — that was premature.

D2. How the fabric block is conditioned

• D2a. Evidence-gated rendering. Render the block only when values carry cluster-derived evidence, using the {{- with $kft.acceleratedNodeSelector }} pattern already eight lines above in the same manifest and documented as deliberate at registry.yaml:723. No evidence → block omitted → byte-identical to today. It narrows the single-node blast radius rather than dissolving it: no-evidence bundles are byte-identical, but on an evidence-positive cluster the same cluster-scoped runtime still serves every TrainJob, so single-node jobs inherit sidecars, resource requests, claims, and hard scheduling dependencies they do not need.

Note also that a {{- with }} guard proves only that a value is non-empty — not that it came from observation rather than a static default, a --set, or a stale snapshot, since all sources merge into one values map before render. Gating should therefore key on an explicit attachment profile carrying source/provenance, node cohort, observation time, resource names and counts — not a bare non-empty check. It relocates rather than dissolves gate reachability: on the criteria-only path (first-class here, and what the chainsaw CUJ tests exercise) nothing renders, so the original defect persists undetected. The failure mode moves from "breaks loudly" to "silently does nothing" — the same class as the bug this epic exists to fix. That cost is not minor, and D2a is only acceptable paired with D4a, which makes that path loud.

• D2b. Static default plus enable flag (the current shape). Simpler to write; keeps D3 alive and requires the gate work.
• D2c. --dynamic / cluster-values.yaml. The repo's sanctioned home for a per-cluster value unknown at bundle time (docs/contributor/component.md:243). Stronger than D2a on one axis: it never guesses, so there is no staleness window. But the value is absent from the recipe, so gke-gpu-nic-networks verifies Network objects exist, not that the TCPXO fabric is usable #2265's name-comparison gate cannot see it — the --set-json/gate blindness reappearing through a sanctioned channel — and it fails this epic's core complaint that the recipe should describe what gets deployed. Not currently available for this manifest. The runtime is a recipe-side raw manifest rendered at bundle time from the original component values (renderInputFor captures c.Values; manifest.Render consumes the raw manifest). --dynamic only splits paths between values.yaml and cluster-values.yaml — it does not defer rendering. By install time the {{- with }} block has already been included or omitted. Using D2c would require a second-stage rendering change: convert the runtime to a real Helm template with escaped inner expressions, or otherwise defer rendering. Do not present cluster-values.yaml as a present-day fallback.
• D2d. No conditioning at all — see D9; if fabric wiring lives in separate named runtimes, the generic runtime never changes and there is nothing to gate.

D2e. Attachment profile — use the ADR-015 mechanism AICR already ships

This is not new machinery. pkg/recipe/profile.go already defines exactly the abstraction D2, D6 and D9 keep reaching for:

type SelectedProfile struct {
    Name       string
    Value      string
    Advertiser string              // who chose it — provenance
    OwnedPaths map[string][]string // declaration-wide lock surface
}

Selected once at generation with --profile <name>=<value>, persisted in RecipeResult.Metadata.SelectedProfile. The shipped gpuStack profile in recipes/overlays/aks.yaml is a working template, and its own comment describes this epic's problem: the mode is "selected once at recipe generation … recorded in metadata.selectedProfile, and the paths below become profile-owned — a bundle-time --set diverging from the selected value is rejected instead of silently producing the unqualified hybrid". It also states that "unavailable, unknown, or mixed pool values fail closed. Criteria-only generation (no --snapshot) selects explicitly."

Shape: --profile fabricAttachment=gke-tcpxo-multinetwork | eks-efa-device-plugin | aks-rdma-shared | gke-managed-dranet | none.

Problem recorded in this epic	How profiles answer it
D2: {{- with }} proves non-emptiness, not provenance	Partially — values are profile-owned and locked. But Advertiser is not provenance: it has a closed GPU-advertiser meaning (recipe-owned vs external), not "who chose this"
D2a: criteria-only path silently renders nothing	Partially — omitting --profile selects the declared default, so the current API cannot require explicit selection. Required-selection semantics would have to be added
D3: default-on vs default-off	Moot — wiring ships in a named runtime users opt into by reference; there is no default
--set-json / gate blindness	OwnedPaths makes a diverging --set a rejection, not a silent override
D6: recipe as single source of truth	The choice is recorded in the emitted recipe, auditable
D9b: "provisioning-profile-specific" runtimes	Literally the profile abstraction, already named that way
Unowned: B200/GKE multi-network vs DRANET; OCI hostNetwork vs SR-IOV	Mutually exclusive values on one profile, carrying each one's version constraints

One limitation that constrains #2274. ProfileComponentRef is {Name, Overrides} only, and its godoc is explicit: "A profile may only assign values on an existing component; component identity, deployment shape, manifests, health checks, and ordering are not profile effects." So a profile cannot swap manifestFiles. #2274's current "separate manifest, selected per overlay" approach is therefore not profile-selectable — under D2e the fabric manifest must be attached unconditionally and gated internally by a profile-set value. hasYAMLObjects() already drops an empty release cleanly, so this works, but it contradicts #2274 as written and must be reworked if D2e is chosen.

BLOCKER: AICR supports exactly one profile per recipe, and the GKE chain has already spent it.

• --profile is a cli.StringFlag (pkg/cli/recipe.go), single-valued and rejected if repeated.
• Metadata.SelectedProfile is a singular pointer (pkg/recipe/metadata.go:929).
• The overlay spec key is singular profile:, not a list.
• Two overlays declare one today: aks.yaml and gke-cos.yaml, which declares gpuStack — and h100-gke-cos-training-kubeflow inherits from that chain.

So the one recipe family where TCPXO lives cannot take a fabricAttachment profile without extending the mechanism from one selected profile to N. That touches the recipe schema, the CLI flag, the metadata field, and the OwnedPaths conflict logic. It is a design task, not a detail.

Three ways forward, none obviously right:

1. Extend profiles to N per recipe. Cleanest conceptually; changes ADR-015's contract and its ownership/conflict model, so it needs its own design note.
2. Fold fabric attachment into gpuStack as extra values. Cheap but wrong — driver installation and fabric attachment are independent axes, and cross-multiplying them produces a combinatorial value set.
3. Declare fabric attachment outside the profile mechanism. Loses OwnedPaths, the --set rejection, and recorded provenance — which are precisely the properties that made D2e attractive.

Recommendation: D2e remains the right shape — gpuStack and fabricAttachment are the same kind of decision, and the fail-closed-on-unknown behaviour is what D-B needs. But it is not adoptable until the one-profile constraint is resolved. D3 and D4 dissolve under it if it lands; they do not dissolve today.

Note the single-valued shape only works at all because of the D-A narrowing: GB200/EKS has both EFA and NVLS on the hardware, and one profile value could not express both. That is a reason not to reopen EFA into runtime scope later without revisiting this.

D3. Default-on versus default-off — MOOT

Dissolved by the delivery-vehicle decision, not by argument. Fabric wiring ships in a named full-node runtime (#2287, contents from #2274), which a user opts into by referencing it in runtimeRef. The generic torch-distributed is unchanged. There is nothing to enable or disable, so no default exists to argue about.

What survives is discoverability — a runtime users must name by hand reaches only users who know it exists, which is the same weakness default-off had. That is now #2288's job, and it is why #2288 is a dependency rather than a parallel task.

The material below is retained as the record of how this was argued before the vehicle was settled.

D3 (historical). Default-on versus default-off

Currently recorded as disabled by default, flipping per fabric as gates land. Note D2a would make this moot.

• D3a. Keep default-off with the flip criterion.
• D3b. Default-on once each fabric's gate exists (rejected earlier: the gate is unreachable on the bundle-and-deploy path, and the blast radius exceeds the defect's).
• D3c. Moot under D2a.

D4. Capability gate: blocking or advisory

• D4a. Advisory — SELECTED (as D4c, advisory default with blocking opt-in). Report "the cluster advertises the fabric but the deployed runtime references none," naming the remediation. Cannot break anyone; strictly more informative than a gate, which can only say "what you referenced is missing."

• D4b. Blocking gate, as currently written in the children.

• D4c. Advisory by default, blocking opt-in — DECIDED, but the mechanism does not exist yet. Lowest risk of the three: it cannot break a working cluster and preserves both behaviours; a blocking default would fail clusters whose users legitimately do not use the fabric.

  Correction: an earlier revision claimed aicr validate's existing fail-on switch provides the opt-in. It does not — --fail-on-error only inspects statuses that are already failing (ctrf.IsFailingStatus) to decide the exit code; it cannot promote an advisory. D4c therefore requires a new policy or severity input that changes the check's own result, or two distinct checks. Owned by Workload fabric contract for EFA/RDMA: document it, and advertise cluster capability #2288, and it is real work rather than a flag reuse.

D5. NVLS / ComputeDomain (#2277)

Note on the precedent, so the record is not misread: the slinky-slurm fixed ComputeDomain is sound — it is correctly scoped, because its consumers are namespaced by construction and co-located with it. The finding is the mismatch between a cluster-scoped runtime and a namespaced claim, not any defect in that precedent. D1 adds a third independent kill for D5c: even with the CD in the right namespace, container resources.claims is erased by any TrainJob setting resourcesPerNode, so D5c also requires the D1g contract change. NVLS has the most stacked blockers of any fabric; sequencing it last is right.

Three independent blockers: namespace scope (a namespaced CD cannot serve a cluster-scoped runtime), per-workload lifetime, and container-claim erasure under D1.

• D5a. Document as a user prerequisite and do not ship — follows the existing precedent where Network / GKENetworkParamSet are cluster-provisioning's responsibility (gke-nccl-tcpxo health check passes on clusters with no GPU NIC networks #2216).
• D5b. Per-workload CD/RCT creation in the TrainJob namespace, requiring a controller or admission integration AICR does not have.
• D5c. Ship a fixed CD via preManifestFiles, mirroring slinky-slurm exactly, and document that TrainJobs must run in the runtime's namespace to use it.
• D5d. Do not ship, but add a deployment-phase check that a user-supplied ComputeDomain exists and fail closed when NVLS is expected — pairs naturally with D5a and reuses the advisory machinery from D4.

D6. Scope of decision 4 (recipe as single source of truth)

The rule as written forbids bundle-time-only input — but acceleratedNodeSelector is exactly that, from a dedicated CLI flag, endorsed in registry.yaml, rendering into this manifest.

• D6a. Restate as "the recipe must remain auditable and sufficient," list the four tiers that exist (values default → snapshot-into-recipe → --dynamic / cluster-values.yaml → --set-json), and reject a new flag on redundancy grounds.
• D6b. Keep the absolute rule and explain why acceleratedNodeSelector is an accepted exception.

D7. Rows classified — two confirmed, two conditional on cluster evidence

• rtx-pro-6000-eks — CONDITIONALLY affected. EFA support is per shape, not per family: the family is 2xlarge | 4xlarge | 8xlarge | 12xlarge | 24xlarge | 48xlarge, and only .2xlarge and .4xlarge lack EFA. (Note there is no g7e.16xlarge — the family skips 12 to 24, unlike g6e, which matters for any shape-matching logic.) The overlay criteria carry no instance-type dimension, so the recipe cannot distinguish them. Enable only on EFA-capable shapes with EFA provisioned and positive node-allocatable evidence. The in-repo contradiction is a defect: rtx-pro-6000-eks-inference.yaml:59-60 says "EFA is unused" while -training.yaml:82-83 says g7e nodes have "an inter-node fabric, which is exactly what this check targets." Vendor docs say the training overlay is right. File the inference-comment correction separately.
• a100-eks — CONDITIONALLY affected. P4d/P4de are EFA-capable, so the hardware conclusion is solid — but capability is not evidence that a node advertises vpc.amazonaws.com/efa: EFA interfaces must be attached at provisioning and the device plugin must register the resource. Enable only on node-allocatable evidence. Programmatic check: aws ec2 describe-instance-types --filters Name=network-info.efa-supported,Values=true per region, plus node allocatable for vpc.amazonaws.com/efa. This is also the strongest case for D2e — "is this shape EFA-attached" is exactly a snapshot-observed profile selection that fails closed on unknown. p4d/p4de are EFA-capable under Nitro v3 (EFA v1): RDMA read yes, RDMA write no, 4 network cards, 400 Gbps. AWS also states EFA traffic between P4d/P4de and other instance types is unsupported. Wiring it is correct; expecting H100-class behaviour is not. Note EFA count is 4 here versus 32 on p5 — direct support for making the count node-derived rather than a static default.
• a100-oke — AFFECTED and technically reachable; exclusion is a product decision, not a proof. Vendor docs settle reachability (BM.GPU.A100-v2.8 is a compute-cluster shape; OKE compute clusters enable the RDMA host plugins). Exclude it from the generic runtime, but decide explicitly whether an opt-in OKE hostNetwork or SR-IOV attachment profile is supported when its prerequisite is detected — this epic accepts externally provisioned prerequisites elsewhere. It is also a fifth wiring shape no child owns. OCI BM.GPU.A100-v2.8 has RoCE v2. Pod consumption is documented two ways: SR-IOV VF (nvidia.com/rdma-vf) or privileged hostNetwork. The resource name differs from AKS's rdma/hca_shared_devices_a, and AICR deploys no enabler on OKE. Neither EKS: shipped runtime has no EFA wiring — training falls back to TCP #2275 nor AKS: shipped runtime has no RDMA/InfiniBand wiring — training falls back to TCP #2276 covers it.
• OKE RDMA generally — EXCLUDE on design grounds, not "unverified". Both A100 and GB200 require privileged hostNetwork: true in the documented path; the SR-IOV alternative needs multi-network annotations and a network operator AICR does not deploy on OKE. A cluster-scoped runtime must not impose privileged hostNetwork on every TrainJob. That is a far better justification than the current "until OCI-specific pod RDMA exposure is verified" in gb200-oke-training.yaml.

Not a concern after checking: rdma/hca_shared_devices_a is a config-chosen name, but AICR chooses it itself (nic-cluster-policy-aks.yaml:97 sets "resourceName": "hca_shared_devices_a" with a compatibility comment), so #2276's use of the literal is correct for AICR-deployed clusters.

D8. GB200 / EKS composition — RESOLVED: #2287 carries both

An earlier revision marked this dissolved on the grounds that EFA left runtime scope. That was wrong, and backwards: excluding EFA from the runtime moved it onto the exact mechanism that erases NVLS.

The contradiction is between two decisions this epic states, on a combination it lists as in scope:

• D-A concludes EFA is workload-expressible, so the sanctioned path is the user setting resourcesPerNode carrying vpc.amazonaws.com/efa.
• D-C, NVLS row, requires the workload to omit resourcesPerNode entirely, or the container claim is dropped.

A user on gb200-eks cannot satisfy both. Following the EFA guidance destroys the NVLS attachment; following the NVLS contract forgoes EFA. This is stronger than the validator's NCCL_NET_PLUGIN=none exclusivity, which is a configuration choice — these two contracts are mutually unsatisfiable.

Decision: #2287 carries EFA and the NVLS claim together, so a user on gb200-eks sets nothing and gets both. Declaring the combination NVLS-only was the alternative; it silently forgoes inter-node bandwidth on a combination AICR validates a floor for, which is the wrong trade on a fabric-attached runtime that is full-node anyway.

This folds D8 into #2287 rather than dissolving it. #2287's scope now explicitly includes EFA on GB200/EKS.

D9. Delivery vehicle: one conditional runtime, or separate named runtimes

Not yet considered, and it sidesteps several other decisions at once.

• D9a. One runtime, conditionally wired (everything above assumes this). The generic torch-distributed gains fabric blocks that render when conditions are met.
• D9b. Separate fabric-specific runtimes. Keep torch-distributed byte-identical to today, and ship e.g. torch-distributed-tcpxo alongside it. Users opt in by name in runtimeRef. This removes the generic-runtime default blast radius — nothing changes for anyone who stays on torch-distributed. It does not remove: the blast radius for single-node jobs that select the fabric runtime; parameter resolution for counts, resource names, networks and node cohorts; or D1 replacement. "Fabric-specific" is also too coarse a key — EFA device-plugin versus future DRA attachment, OCI hostNetwork versus SR-IOV, and GKE multi-network versus managed DRANET are distinct attachment profiles that share a transport. Prefer provisioning-profile-specific runtimes. Costs: users must know the runtime exists, which is the same discoverability problem as default-off; and it does not escape D1, since replacement erases container resources in any runtime.
• D9c. Both — generic runtime unchanged, fabric runtimes shipped, and the advisory from D4 points users at the right one when it detects an unwired deployment on a fabric-capable cluster.

D10. Whether wiring is the right fix at all

If D1 resolves to replacement, the runtime demonstrably cannot deliver EFA or RDMA wiring, and the epic's premise is partly wrong for those fabrics.

• D10a. Ship wiring where it survives, document where it does not.
• D10b. Documentation and detection only — do not wire anything; make the gap visible via the D4 advisory plus an explicit per-fabric user contract. Smallest change, and given D1g's rejection this is a defensible interim posture for EFA, RDMA and NVLS until D1d or a post-materialization solution exists. It closes more of Gap 2 than first stated: the divergence half is closable by assertion alone, and the materialization half is closable by Performance validation: assert the realized Pod carries the declared attachment #2278's realized-Pod gate applied to the documented contract — submit exactly that TrainJob and assert the realized Pod carries the profile. What it cannot do is prove an arbitrary user's job is wired; only that the documented contract produces a wired Pod.
• D10c. Treat it as an upstream problem and drive D1d, accepting that the epic stalls until that lands.

Relationship to the Excalibur migration — not a v1 dependency

A separate effort proposes replacing AICR's NCCL performance check with Excalibur (Cluster Readiness Engine). Its ETA is unknown. Nothing in this epic assumes it is available for the v1 release, and no child should be sequenced behind it.

Recorded only so the interaction is not rediscovered later:

• The fabric-wiring children (Ship the named full-node TCPXO runtime: artifact, plumbing, and GKE wiring #2274-GB200 NVLS/IMEX: pod-level resourceClaims in the named runtime, EFA combined on EKS #2277) are unaffected. They concern the artifact AICR ships to user workloads, not how AICR benchmarks. No dependency in either direction.
• Performance validation: assert the realized Pod carries the declared attachment #2278 becomes more important, not less. Excalibur's per-fabric configuration lives in its own workload catalog, which is further from AICR's shipped runtime than today's testdata. An assertion tying the certified configuration to the shipped one would be the only link between them.
• Shared plumbing: lift GKE fabric helpers to pkg/, values schema, render + divergence tests #2273's divergence guard has a shelf life; its helper relocation does not. The guard compares against validators/performance/testdata/h100/gke/runtime.yaml, which the migration's retirement step deletes. Moving gkenet into pkg/ survives regardless.
• Independent confirmation of this epic's premise. The Excalibur PoC measured 3 GB/s over TCP versus 489 GB/s over EFA on the same nodes minutes apart — a well-formed number certifying a configuration that was not in use. Same failure shape as this epic, found in a different subsystem. Its stated conclusion, that the transport assertion must stay in AICR, is the same gap recorded in Ship the named full-node TCPXO runtime: artifact, plumbing, and GKE wiring #2274: on GKE/H100 no transport assertion runs at all today.
• Possible reuse, with caveats. Excalibur's internal/catalog/entries/_lib/nccl/ carries per-fabric env as data (aws-h100-efa-env.yaml, aws-gb200-efa-env.yaml, gcp-h100-fastrak-env.yaml). Two caveats: no hca_shared reference appears anywhere in that repository, so there is no AKS/InfiniBand entry to borrow despite AKS being listed as ready; and the catalog encodes what a benchmark image needs, which is not identical to what an arbitrary user training image needs.

Out of scope

Managed DRANET — out of scope by explicit decision, not because it is A4X-only. GKE documents managed DRANET for A4 a4-highgpu-8g with B200, not only A4X. It uses mrdma.google.com and requires both pod resourceClaims and container resources.claims, putting it squarely under the D1 claim-erasure problem. GKE also states managed DRANET and Device-type multi-networking must not manage the same NICs. AICR's B200 overlay permits Kubernetes >= 1.32 while managed DRANET requires Standard 1.34.1-gke.1829001+ or Autopilot 1.35.2-gke.1842000+, and GKE states explicitly: "Don't use both the GKE DRANET driver and the GKE multi-network API with a Network resource of Device type in the same cluster… both APIs attempt to manage the same set of NICs" — so B200/GKE has at least two mutually exclusive attachment profiles, a version and provisioning split rather than a replacement. Whichever is chosen is a product decision to record, not an inherited constraint.

Per-node-pool runtime variants. The NRI device list and the interface annotation derive from GPUs-per-node, so one cluster-scoped runtime cannot express two pool shapes. Children assume a homogeneous accelerated pool and should fail loudly otherwise.

Image-side requirements. The runtime can supply annotations, mounts, sidecars, env, and resource requests. It cannot supply the NCCL transport plugin itself — the validator mounts the plugin libraries the installer DaemonSet stages on the host, rather than running a pre-baked image; the user-image contract is ABI/glibc compatibility plus the mount and loader path. Children document the image contract; they cannot enforce it.

Context

Related: #2216 (health check and docs for the Network / GKENetworkParamSet prerequisite, shipped as the gke-gpu-nic-networks deployment check) and #2265 (that check verifies the objects exist, not that the fabric is usable). This gap survives a complete fix to both: a cluster can have all 8 networks bound, pass every validation phase, and still run production training over TCP. Creating and binding the Network / GKENetworkParamSet objects belongs to cluster provisioning, not AICR.

[yuanchen8911]

Code-reference update: #2248 (merged as 78b97cd4c) relocated the discovery helper this issue cites.

discoverGKEGPUNICNetworks is no longer in validators/performance/nccl_gke_utils.go. It moved to validators/internal/gkenet and is now exported as DiscoverGPUNICNetworks, shared by the performance validator and the new gke-gpu-nic-networks deployment check. buildGKENetworkInterfacesAnnotation and buildNRIDeviceAnnotation stayed in nccl_gke_utils.go.

This bears on one of the design questions above — how does the runtime learn cluster-specific network names? Discovery now lives in an importable package rather than package main, so a runtime-side or bundler-side consumer can call it directly instead of duplicating the logic. The annotation builders are still validator-local, so those would need lifting too.

Nothing about the gap itself changes: the shipped kubeflow-trainer runtime still carries no multi-NIC binding, and that is still distinct from the prerequisite detection #2248 added.

[mchmarny]

Triaged: P2→P1. Performance validation can pass while the shipped Kubeflow runtime runs single-NIC and silently loses GPUDirect-TCPXO; prioritize closing validator/runtime parity.