validate: fail-closed check for GKE device-plugin ownership conflict

[yuanchen8911]

Problem

On GKE, default GPU node-pool provisioning leaves the gke-no-default-nvidia-gpu-device-plugin label absent, so GKE's managed nvidia-gpu-device-plugin DaemonSet runs on every GPU node and advertises nvidia.com/gpu. AICR's stock GKE recipes ship gpu-operator with devicePlugin.enabled: true, so deploying a bundle onto an unlabeled (default-provisioned) cluster produces two device plugins registering the same nvidia.com/gpu resource on each GPU node.

Kubelet maintains one endpoint and one device inventory per resource name (device manager keys endpoints[resourceName]), so competing registrations and ListAndWatch updates replace each other — nondeterministic ownership, and potentially one plugin's device IDs (GKE: nvidia0-style names; NVIDIA: GPU UUIDs) passed to the other plugin's Allocate implementation. This can cause allocation and runtime failures; it is not a benign overlap.

AICR's stock profile intends the operator's plugin to be the sole advertiser, which requires GPU node pools to carry GKE's documented opt-out label gke-no-default-nvidia-gpu-device-plugin=true (see NVIDIA GPU Operator GKE guidance).

Current gap — nothing detects the conflict

• aicr bundle is offline by design (recipes define what, independent of cluster state) and cannot see node labels.
• aicr validate has no guard: the GKE overlays' only constraint is K8s.server.version, and the generic constraint syntax cannot express "every GPU node carries this label" — NodeTopology.label readings encode value|node1,node2,... (with possible truncation) while the evaluator compares scalar readings. There is no way to cross-reference the GPU-node set against the label's node list.
• Deployment-phase health checks pass (the operator's device-plugin DaemonSet is healthy); the conflict surfaces only as incidental, nondeterministic workload failures with no pointer to the cause.

So a user pointing AICR at a default-provisioned GKE cluster gets a clean bundle and a validation run with no deterministic guard, and ships a latent plugin-ownership conflict.

Proposed solution

Add a deterministic, fail-closed check that GPU nodes on GKE carry gke-no-default-nvidia-gpu-device-plugin=true, via either:

1. a dedicated GKE preflight/deployment validation check, or
2. an extension to topology constraints supporting node-set predicates (e.g. "all nodes matching have label X=Y"), which the GKE overlays would then use.

Requirements:

• Fail closed on missing, false, or mixed labels across GPU nodes (a spuriously passing check is the dangerous direction).
• Clear diagnostic, e.g.: "GKE's managed device plugin must be disabled on AICR-managed GPU node pools via the node label gke-no-default-nvidia-gpu-device-plugin=true; otherwise it conflicts with the GPU Operator's device plugin (devicePlugin.enabled: true). Because the recipe also sets driver.enabled: false, compatible driver provisioning remains a separate prerequisite."
• The label controls device-plugin ownership only; do not couple the check to driver provisioning.
• Optional follow-on: aicr recipe --snapshot (the only generation-side step with cluster state) could emit an advisory warning when the snapshot shows unlabeled GPU nodes on GKE.

Non-goal: inferring or auto-flipping devicePlugin.enabled from observed labels — advertiser ownership stays explicit in the recipe. A gke-managed variant (devicePlugin.enabled: false, label absent, validation recognizing GKE's advertiser) can be a separate explicit opt-in profile if demand appears.

Documentation

Document the label as a GKE cluster prerequisite (deployment prerequisites / component catalog GKE notes) in the same PR.

Relation to ADR-015 (recipe configuration profiles)

ADR-015 (PR #1762, implementation #1761) builds its first consumer — GKE gpuStack: [operator (default), csp-managed] — on this check, and depends on option 2 above (the reusable node-set constraint form, a new reading/evaluator capability), not a one-off preflight check:

• The operator profile value uses the positive predicate: every GPU node carries gke-no-default-nvidia-gpu-device-plugin=true (this issue's original scope).
• The csp-managed profile value additionally needs the negated form (no GPU node carries the opt-out label), so GKE's managed plugin is provably the sole advertiser.

Scope confirmation is tracked as ADR-015 Deferred Decision 2: confirm whether delivering the reusable node-set constraint form (including negation) lands under this issue, or whether the form splits into its own issue with this one consuming it.

Note: the gke-managed variant sketched above as a possible future opt-in ("if demand appears") is now planned work — it is exactly ADR-015's csp-managed profile value (adoption plan step 2). The non-goal stands unchanged: advertiser ownership stays explicit in the recipe; nothing infers or auto-flips devicePlugin.enabled from observed labels.

Acceptance requirements for the node-set constraint form (from ADR-015 cross-review)

1. Fail closed on truncated readings. --max-nodes-per-entry is a supported snapshot option and the topology collector truncates label membership (formatNodeList, pkg/collector/topology/topology.go — value|node1,node2,... with a (+N more) tail). A set evaluator seeing only the visible nodes can falsely pass both predicate directions. Any truncated reading participating in a node-set constraint must fail as "reading unavailable — regenerate the snapshot without truncation", never evaluate; cover the positive and the negated predicate.
2. Define the authoritative, non-empty GPU-node universe. The proposal's <GPU selector> must name its signal. Existing detection keys on post-NFD/GFD nvidia.com/gpu.* labels (hasGPUNodesInTopology, pkg/snapshotter/snapshot.go), which do not exist on a pre-Operator GKE cluster — the validation this check exists for. GKE natively labels GPU nodes with cloud.google.com/gke-accelerator; the chosen selector must yield a complete and non-empty GPU set, and an empty set must fail closed (otherwise the negated predicate passes vacuously). Cover multiple accelerator values and pre-Operator snapshots.

[yuanchen8911]

Triaged: P2→P1, Status stays Ready. Direct unblocker for a P1: #1761 (Implement ADR-015) states that item 2, the reusable node-set predicates tracked here, gates the GKE rollout in its PR 3. Prioritized to match the work it blocks.

[yuanchen8911]

Triaged: confirmed at P1 / Ready. Verified every factual claim in the description against origin/main before confirming the priority; all of them hold.

• GKE recipes do ship devicePlugin.enabled: true. It is set explicitly in recipes/components/gpu-operator/values.yaml, and values-gke-cos.yaml overrides only the env block, never enabled. driver.enabled: false is confirmed there too.
• The GKE overlays carry only K8s.server.version constraints: gke-cos >= 1.28, gke-cos-training and gke-cos-inference >= 1.30. Nothing else.
• Label readings do encode value|node1,node2,... and do truncate. encodeLabels and formatNodeList, pkg/collector/topology/topology.go:169-191, truncate when maxNodes > 0 and len(nodes) > maxNodes. --max-nodes-per-entry is a real snapshot flag (pkg/cli/snapshot.go:375), so acceptance requirement 1 is guarding a reachable state, not a hypothetical one.
• GPU-node detection does key on post-NFD labels. hasGPUNodesInTopology, pkg/snapshotter/snapshot.go:293-311, matches the nvidia.com/gpu.* prefix, and that file's own log line states that clusters without those labels are not detected. Acceptance requirement 2 is well founded.

One finding is worse than the description states. The label gke-no-default-nvidia-gpu-device-plugin appears in exactly one file in the whole repository, the ADR-015 design document. There are zero hits across docs/user, docs/integrator, and recipes. So the prerequisite is not merely unenforced, it is undocumented for users.

Two things worth acting on separately from the main work.

First, split the documentation out. The full fix is a new constraint form and is real work, but a prerequisite note in docs/user is a doc-only change that removes most of the practical risk immediately and does not depend on any of it. The Documentation section currently asks for this in the same PR; it does not need to be.

Second, resolve ADR-015 Deferred Decision 2 before implementation starts. This issue began as a GKE preflight check and now also covers a reusable node-set constraint form with negation, truncation fail-closed semantics, and an authoritative GPU-node universe. DD2 already tracks whether that form should split into its own issue with this one consuming it. Since this gates PR 3 of #1761, having a P1 whose scope is still open is the part worth closing first.

One honest caveat on severity: the mechanism is argued from the kubelet device manager keying endpoints by resource name, and no observed allocation failure is cited. The reachability is what carries the priority here, since GKE's opt-out label is opt-in and a default-provisioned GPU node pool therefore hits this path with a clean bundle and a passing validation run.

Still unassigned.