EKS: shipped runtime has no EFA wiring — training falls back to TCP

[yuanchen8911]

Part of #2217.

Answers D-A for EFA. Blocked at the design level: EFA is a container resource, which the Trainer API erases. Until D-A's constraint changes upstream, the posture for this fabric is document-and-detect, not ship.

v1 milestone caveat. This issue is in v1, but its viability depends on D1 on #2217 — whether Kubeflow Trainer v2.2.0 replaces or merges the runtime container's Resources when a TrainJob sets trainer.resourcesPerNode. If it replaces, the resource request this issue adds is erased by AICR's own documented TrainJob (demos/cuj1-training.md:175-204) and this approach does not work. Do not plan v1 around this landing as scoped until D1 is settled.

Goal

A TrainJob submitted against the shipped torch-distributed runtime on an EKS cluster uses EFA instead of falling back to TCP.

Applicability is per overlay and must be evidence-backed — not "every EKS overlay". aws-efa is declared unconditionally in recipes/overlays/eks.yaml:66-68, but a deployed device plugin is not evidence that nodes advertise the resource.

Overlay	EFA wiring applies?	Evidence
h100-eks	yes	validated NCCL floor over EFA
gb200-eks	yes	validated NCCL floor; also needs #2277
a100-eks	conditional	p4d/p4de are EFA-capable (Nitro v3 / EFA v1): RDMA read yes, write no, 4 cards, 400 Gbps. AWS states EFA traffic between P4d/P4de and other instance types is unsupported — wire it, but do not expect H100-class behaviour
rtx-pro-6000-eks	conditional	EFA support is per shape: g7e.2xlarge/.4xlarge have none; .8xlarge+ do. The overlay criteria carry no instance-type dimension, so the recipe cannot tell them apart — gate on node-allocatable evidence. The overlays contradict each other in-repo — -inference.yaml:59-60 says "EFA is unused" while -training.yaml:82-83 says g7e has "an inter-node fabric". Vendor docs side with training; file the inference-comment fix separately

EFA count varies by instance family — 4 on p4d, 32 on p5 — which is direct support for making the count node-derived rather than a static default.

H200 has no *-training-kubeflow overlay, so it ships no runtime; it is covered the day one is added.

What is missing

The shipped runtime has no vpc.amazonaws.com/efa resource request, so no EFA device is injected into the pod at all. Porting from validators/performance/testdata/h100/eks/runtime.yaml:

• vpc.amazonaws.com/efa: N in container resources requests and limits
• FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, FI_EFA_FORK_SAFE=1
• LD_LIBRARY_PATH including /opt/amazon/ofi-nccl/lib/<arch>, /opt/amazon/efa/lib, /opt/amazon/openmpi/lib
• PATH including /opt/amazon/efa/bin
• NCCL_SOCKET_IFNAME=eth0, NCCL_IGNORE_DISABLED_P2P=1, IPC_LOCK capability

Substantially cheaper than the GKE case: no sidecar, no hostPath volumes, no new container images, and therefore no BOM / mirror / air-gap questions.

Approach

EKS overlays attach kubeflow-trainer through the shared platform-kubeflow mixin, so unlike the GKE case this edits a manifest that every non-GKE platform renders. The wiring must be conditional and inert by default elsewhere. Reuse the values-schema pattern from #2273.

Values follow the recipe-as-single-source-of-truth precedence documented in #2279; do not introduce a bundle-time-only input.

EFA count is node-derived and varies by instance family, so the default needs care. uniformFabricResourceCount (validators/performance/nccl_eks_utils.go:87) already reads allocatable vpc.amazonaws.com/efa across the node set and fails closed on a mixed rollout; the same homogeneity assumption applies here.

Enablement gate

#2217 establishes that fabric wiring ships disabled by default, flipping to default-on per fabric only once that fabric's capability gate exists. EFA needs its own such gate before adopting that default.

The failure shape differs from GKE. A missing GKE Network makes pods fail admission; a missing EFA resource makes pods stay Pending indefinitely, because nothing schedulable advertises vpc.amazonaws.com/efa. Loud either way, but it is a hard failure for clusters that previously ran (slowly) over TCP.

No such gate exists today. recipes/checks/aws-efa/health-check.yaml asserts that the aws-efa-k8s-device-plugin DaemonSet rolled out (desiredNumberScheduled > 0 and numberReady == desiredNumberScheduled) and runs pod-health assertions — but never checks that nodes advertise the extended resource. That is the same defect class as #2265: the check verifies the provider is present, not that the capability is. A pool containing non-EFA instance types passes it.

Recommended: add the gate here; ship disabled and flip once it exists. The logic already exists and there is a precedent for exactly this move.

uniformFabricResourceCount (validators/performance/nccl_eks_utils.go:87) is already parameterized by resource name and already serves both EFA and the AKS RDMA pool:

func uniformFabricResourceCount(nodes []v1.Node, resource v1.ResourceName) (int, error)

It reads allocatable across the full target node set and fails closed on a mixed or partial rollout. So the work is to lift it into a package importable from validators/deployment/ — e.g. validators/internal/fabricres/, mirroring how #2248 placed GKE discovery in validators/internal/gkenet — and add one deployment check parameterized by the resource name.

Note this destination differs from #2273's. #2273 moves gkenet out of validators/internal/ into pkg/ because the bundler needs it. uniformFabricResourceCount has no pkg/ consumer — its only new caller is validators/deployment/, which can import validators/internal/. Do not over-promote it.

Correction: the AKS gate already exists and this child must build the EFA one. rdmaFabricProbeCoverage (validators/deployment/expected_resources.go:1137) is already a deployment-phase, node-allocatable, fail-closed, uniform-count RDMA check. That same file contains zero vpc.amazonaws.com and zero EFA references, so no EFA equivalent exists. The prior art is on the AKS side — generalize rdmaFabricProbeCoverage rather than writing something new, and note #2276 therefore shrinks rather than depending on this child.

Precedent: PR #2248 did this for GKE — lifted DiscoverGPUNICNetworks out of the performance validator into validators/internal/gkenet, then added validators/deployment/gke_gpu_nic_networks.go. validators/deployment/ is an established directory (operator_health.go, gpu_operator_version.go, nvidia_smi.go, expected_resources.go).

Note the check must assert node allocatable, not merely that the device plugin rolled out — that distinction is the whole defect (#2265's framing: verifying the provider is present, not that the capability is).

Caveat to document

The runtime can supply the resource request and env, but not the aws-ofi-nccl plugin itself — that lives in the container image. The validator sidesteps this by running public.ecr.aws/hpc-cloud/nccl-tests with the plugin pre-baked; a real user brings their own training image. Document the image contract; it cannot be enforced from the runtime.

Acceptance criteria

• EFA wiring enabled only where target nodes actually advertise vpc.amazonaws.com/efa (node-allocatable evidence, not vendor capability), and inert on other platforms
• Wiring ships disabled by default (per [Epic]: No fabric-wired runtime ships, so validation can only measure a fixture #2217); capability gate present as a deployment-phase check asserting nodes advertise vpc.amazonaws.com/efa
• Flip criterion evaluated and recorded: gate exists, and enabling cannot break single-node jobs on a correctly provisioned EKS cluster
• GKE/AKS/OKE/Kind bundles render byte-identical to before
• Divergence guard from Shared plumbing: lift GKE fabric helpers to pkg/, values schema, render + divergence tests #2273 extended to cover the EKS runtime
• Image-side plugin requirement documented
• Transport verified as EFA (not Socket) on an EKS H100 cluster

[yuanchen8911]

Closing: this is not a runtime concern. A workload can express EFA/RDMA entirely on its own, so AICR shipping the wiring buys convenience rather than capability — and buys it at the cost of being silently erased whenever the user sets resourcesPerNode, which they must do for GPUs.

What a workload controls

Verified against Kubeflow Trainer v2.2.0. A TrainJob can set image, command, args, env, resourcesPerNode, and — via PodTemplatePatch — pod labels and annotations, volumes, volume mounts, securityContext, nodeSelector, tolerations, and affinity.

Everything this fabric needs is in that list:

• the extended-resource request (vpc.amazonaws.com/efa / rdma/hca_shared_devices_a) in resourcesPerNode, which the user sets anyway for GPUs
• NCCL and provider env via spec.trainer.env, which merges rather than replaces
• an image carrying the transport plugin — the user's choice, and something AICR cannot supply regardless

AICR's real contribution is deploying the device plugin so the resource is advertised, and that already happens unconditionally (aws-efa in recipes/overlays/eks.yaml, network-operator in aks.yaml).

Why this is the right boundary, not a workaround

Trainer's replace-not-merge behaviour on container resources is the API asserting that resources are workload-owned. Shipping them from the runtime fights that design and loses. By contrast a workload genuinely cannot add a container or set pod-level resourceClaims, which is why GKE TCPXO (#2274) and NVLS (#2277) remain in scope — those are must-have runtime support.

See D-A on #2217 for the full ownership analysis.

The real defect, which survives

Users are not told what to ask for, and nothing detects when they have not asked. That is a documentation and detection concern, and it is tracked separately rather than here. Closing this issue does not close that.