Ship the named full-node TCPXO runtime: artifact, plumbing, and GKE wiring

[yuanchen8911]

Consolidated — this issue now covers what were #2287, #2273 and #2274. They were three issues that could only ship together: an empty named runtime is not a deliverable, and the schema and builders exist solely to render this manifest. Full discussion history remains on the closed issues.

Part 1 — the named runtime artifact (was #2287). A ClusterTrainingRuntime whose declared property is full-node, single-shape, fabric-attached, in the single supported namespace (kubeflow, per D-E). Users reach it by name in runtimeRef.

Naming — DECIDED, and children inherit the rule, not just the list:

Runtime name	Profile
torch-distributed-tcpxo	GKE A3 Mega — TCPXO, 8 NIC (this issue)
torch-distributed-tcpx	GKE A3 High/Edge — TCPX, 4 NIC (#2290)
torch-distributed-nvls	GB200 OKE — NVLS/IMEX (#2277)
torch-distributed-nvls-efa	GB200 EKS — NVLS + EFA combined (#2277)

The rule: the suffix names the attachment mechanism. Where a transport has one mechanism, the transport name is the suffix; where it has more than one, the suffix disambiguates by mechanism, not transport. B200/GKE (#2291) and OCI A100 are the known cases where a transport-derived name would fail — see D9b in #2217.

Part 2 — shared plumbing (was #2273). Lift validators/internal/gkenet to pkg/gkenet (Go's internal rule blocks importing it from pkg/), add indentation-free builders, define the values schema this manifest renders from, and add the divergence guard: a test asserting the shipped manifest and validators/performance/testdata/h100/gke/runtime.yaml agree on the fabric-bearing fields. That guard is the recurrence prevention for this entire epic — runtime-versus-validator divergence is the originating defect.

Part 3 — the TCPXO wiring (below).

Part of #2217. Multi-NIC binding exists only inside the validator, which discovers networks and injects annotations at apply time. The shipped runtime has none of it — so a correctly provisioned cluster passes validation while real training jobs still run single-NIC.

Prerequisite, not scope: the Network / GKENetworkParamSet CRs are an infrastructure concern AICR does not manage. This issue assumes they exist and are bound.

Answers D-A for GKE TCPXO, the one fabric whose wiring survives the Trainer API constraint. Design is settled; open items here are implementation choices.

Goal

A TrainJob referencing the named full-node TCPXO runtime (torch-distributed-tcpxo, Part 1) on a GKE H100 a3-megagpu-8g cluster uses GPUDirect-TCPXO instead of falling back to TCP. The generic torch-distributed is unchanged.

The must-have: the sidecar

A workload cannot add a container. ContainerPatch.Name states "Runtime must have this container", so no TrainJob configuration can introduce the tcpxo-daemon. TCPXO does not function without it. That is the irreducible reason this fabric needs runtime support, and it is the only one.

Everything else TCPXO needs — pod annotations (via PodTemplatePatch.Metadata), hostPath volumes, volume mounts, capabilities, env — is workload-expressible. The runtime should still carry them, because splitting one mechanism across two owners would force every user to hand-write annotations naming a container they did not add. But they are shipped for coherence, not necessity.

Scope accordingly: the sidecar and the wiring that makes it functional. Do not extend this issue to anything a workload could supply on its own.

What is missing

Porting from validators/performance/testdata/h100/gke/runtime.yaml:

• 3 pod annotations — devices.gke.io/container.tcpxo-daemon (NRI device list), networking.gke.io/default-interface: eth0, networking.gke.io/interfaces (JSON array binding eth1..eth8 to the GPU NIC networks)
• tcpxo-daemon native sidecar — initContainers entry with restartPolicy: Always, image tcpgpudmarxd-dev:<version matched to the shipped plugin — see Supply chain>, running /fts/entrypoint_rxdm_container.sh --num_hops=2 (--num_hops=2 only — --num_nics is inert from v1.0.20), capabilities NET_ADMIN + NET_BIND_SERVICE (deliberately not privileged)
• 4 hostPath volumes — /home/kubernetes/bin/nvidia (mounted /usr/local/nvidia), /sys, /proc/sys, /dev/aperture_devices; plus dshm emptyDir
• Worker wiring — IPC_LOCK capability, the TCPXO variable set delivered as direct version-coupled env (not by sourcing nccl-env-profile.sh — see Decisions), and NCCL_FASTRAK_LLCM_DEVICE_DIRECTORY / NCCL_DYNAMIC_CHUNK_SIZE / CUDA_DEVICE_MAX_CONNECTIONS

The NRI injector does not create the sidecar — it injects devices into a container named by the annotation key (devices.gke.io/container.tcpxo-daemon), so the pod must declare that container itself.

Approach

RESOLVED: this ships a named full-node runtime, not a conditional generic one.

An earlier revision proposed replacing the generic torch-distributed manifest and gating it with an enable flag. That conflicted with the Part 1 decision, which says TCPX(O) makes the generic cluster-scoped runtime a poor host. The conflict is settled in favour of the named runtime, for a reason that is decisive rather than stylistic:

Google requires a GPUDirect-TCPX(O) workload to use all GPUs and all secondary NICs on the node. So a TCPXO-attached workload is full-node by the fabric's own rule, not by AICR's choice. A cluster-scoped runtime that every job inherits cannot carry a full-node contract without imposing it on jobs that never asked for it.

Consequences:

• The generic torch-distributed is unchanged — no conditional block, no enable flag, no default question.
• This issue creates a named, full-node, TCPXO runtime, whose contract is stated in its name. It is the artifact Part 1 defines; this issue supplies its GKE/TCPXO content.
• Users reach it by naming it in runtimeRef. Discoverability is then partly served by Workload fabric contract for EFA/RDMA: document it, and advertise cluster capability #2288's capability advisory, which can report that the cluster exposes a fabric and name the applicable runtime. It cannot know a workload is unattached — that needs a workload as subject, which deployment phase does not have. So the advisory narrows the gap; it does not close it.
• Artifact selection is Observe fabric attachment and record it in the recipe #2279's, not per-overlay manifestFiles. The GKE overlay has no machine-type axis, so overlay selection cannot distinguish A3 Mega from A3 High/Edge from smaller shapes. Observe fabric attachment and record it in the recipe #2279 must establish observed-mode selection before this issue attaches its artifacts.

Supply chain

Ship the sidecar image as a digest-pinned literal in the manifest, at the daemon version matching the shipped plugin.

RESOLVED — Google's matrix is explicit, and AICR currently runs a skewed pair.

From Google's gpudirect-tcpxo release notes:

Release	Plugin installer	TCPXO daemon
Jan 9, 2026	v1.0.15	v1.0.21
Nov 19, 2025	v1.0.14	v1.0.20

AICR stages plugin v1.0.15 but the validator runs daemon v1.0.20 — the daemon from the previous release. That is version skew against a documented one-to-one mapping.

--num_nics was retired in the v1.0.20 release, and the release note is the key to why the skew went unnoticed: "Specifying this flag will not cause an error, but it no longer has any effect." So the validator's --num_nics ${GPU_COUNT_PER_NODE} is a no-op today. The committed ~338-340 GB/s evidence run therefore shows the combination happened to operate, not that it is supported — a passing benchmark does not overturn a vendor support matrix.

Two earlier revisions of this section were wrong, in opposite directions: one asserted v1.0.20 was "known wrong" without a source; the next reversed that on the evidence-run argument. Google's published matrix settles it.

Decision:

• Keeping plugin v1.0.15 requires daemon v1.0.21, invoked with --num_hops=2 only.
• Treat the existing v1.0.15 + v1.0.20 validator configuration as skew requiring correction and revalidation — not as a supported option to preserve.
• Bumping both sides to Google's current pair is the alternative; either way v1.0.15 + v1.0.20 must not remain.

The installer/sidecar version pairing is a documented hard requirement, not a caution: "The installer and sidecar versions have a one-to-one mapping and must correspond." Bumping either means bumping both as a matched release.

Retired daemon arguments: --num_nics was retired in the v1.0.20 release — "Specifying this flag will not cause an error, but it no longer has any effect." So it is inert at every daemon version AICR would ship. Invoke with --num_hops=2 only. NIC count drives the interface annotation and bindings. It does not affect the daemon command at any version AICR would ship, since the flag is inert from v1.0.20 onward.

Decisions recorded

Sidecar image: digest-pinned literal, daemon v1.0.21 for the shipped plugin v1.0.15, invoked with --num_hops=2 only. Settled against Google's published matrix — see Supply chain.

Activation is settled, not blocked. Google publishes the TCPXO variable set as data and presents direct environment entries as the primary supported form, with the script as an optional shortcut. Deliver the variables as env on the runtime, version-coupled to the matched installer/daemon release, with a test asserting the shipped set matches the pin. No command-independent mechanism needs inventing.

Opt-in is by name, so there is no default question. The earlier revision of this section argued default-on versus default-off at length. That apparatus belonged to the abandoned design where the generic runtime carried conditional wiring. With a named full-node runtime the user opts in by referencing it in runtimeRef; there is nothing to enable or disable, no flip criterion, and no flag.

What survives from that argument, and matters more now: discoverability. A runtime users must name by hand reaches only users who know it exists. #2288 narrows this but cannot close it — it is a capability advisory scoped to the workload-owned EFA/RDMA overlays, with no TrainJob or realized Pod as subject at deployment phase. It can report that the cluster exposes a fabric and name the applicable runtime (D9c); it cannot observe that a workload is unattached. Criteria-only runtime recommendation and wrong-runtimeRef detection remain unowned.

Capability gating still applies, but to the runtime's contents, not to a default. The named runtime must not reference networks the cluster does not publish. #2265 case 3 supplies that check; under D-B the names come from observation, so a mismatch is a discovery failure rather than a wrong default.

Verification

Render tests are Part 2 of this issue. End to end: submit a bare two-node TrainJob against the shipped runtime and assert the transport.

Known blocker. verifyTransportFromLogs (nccl_all_reduce_bw_constraint.go:2027) returns nil immediately for variantDefault, and GKE/H100 is registered as variantDefault (:235). The GKE runtime also sets NCCL_DEBUG=WARN deliberately (testdata/h100/gke/runtime.yaml:170, per #1712 / #1713 log rotation), so the NCCL INFO Using network <plugin> banner is never emitted. Obtaining it requires promoting GKE/H100 to variantNET or adding a short dedicated NCCL_DEBUG=INFO run. Budget for this — it is not a free assertion.

Acceptance criteria

• GKE TCPXO manifest added, selected by observed attachment mode (Observe fabric attachment and record it in the recipe #2279), not by overlay membership
• Non-GKE bundles render byte-identical to before
• BOM regenerated and committed; mirror path verified for the new registry
• The named runtime is documented in docs/user/, including its full-node contract
• aicr validate reports that the cluster exposes a runtime-owned fabric and points at the named runtime to reference (Workload fabric contract for EFA/RDMA: document it, and advertise cluster capability #2288's capability advisory)
• Not claimed: omission detection. Reporting that a workload is unattached needs a subject that does not exist at deployment time — a concrete TrainJob, a realized Pod, or an admission path. Workload fabric contract for EFA/RDMA: document it, and advertise cluster capability #2288 explicitly disclaims it. Until that has an owner, wrong-runtimeRef is a documented limitation, not a detected condition.
• Homogeneity coherence check present, named and registered in pkg/bundler/validations alongside CheckDriverOwnershipCoherence, with a failure message naming both counts
• Transport verified as a real NET plugin (not Socket) on a GKE H100 TCPXO cluster

Sequencing

Wave 2 — blocked on Part 1 of this issue and Part 2 below (values schema + lifted gkenet builders). #2279 is a soft dependency.

#2274 needs values from #2279, not its selection machinery — and under profile-named runtimes (Part 1) the name is the selection, so the machine-type axis argument no longer gates this issue. #2274 can render the interface annotation from values and fail closed when they are absent (what D-B requires anyway); #2279 then populates them by observation, upgrading the experience from "needs --set-json" to automatic.

Verified safe: this issue changes only kubeflow-trainer's manifest content, not any component's enabled state. No validator gates on manifest content, the TCPXO deployment checks key on the separate gke-nccl-tcpxo component, and hasYAMLObjects drops an empty render cleanly. The F1 applicability hole is real but belongs to #2279.

Blocks #2278 and #2292, which need a runtime carrying wiring to assert against. #2290 lands after this so the TCPXO pattern is established first — though the two are not near-identical; see #2290.