validate: fail-closed GKE device-plugin ownership check

[yuanchen8911]

Summary

Adds the fail-closed GKE device-plugin ownership check from #1755: a new node-set constraint form (NodeTopology.gpu-nodes.label) evaluated at aicr validate readiness, wired into the GKE-COS overlays so an unlabeled GPU node pool fails validation closed (exit 2) with an actionable diagnostic before any phase runs.

Motivation / Context

AICR's GKE recipes ship the GPU Operator with devicePlugin.enabled: true, while a default-provisioned GKE GPU node pool also runs GKE's managed device plugin — two advertisers of nvidia.com/gpu per node, with nondeterministic ownership. Until now nothing detected the conflict: aicr bundle is offline, and the generic constraint syntax compares scalar readings, so "every GPU node carries gke-no-default-nvidia-gpu-device-plugin=true" was inexpressible. PR #1998 documented the prerequisite; this PR enforces it.

Scope position (ADR-015 Deferred Decision 2 — marked resolved in the ADR by this PR). The reusable node-set constraint form — including the negated direction — lands here under #1755, exactly as the ADR reads it. The follow-up GKE gpuStack profile recipes (#1761 adoption step) will consume the form unchanged when they land: operator uses the positive predicate, csp-managed the negated one. Nothing in this PR touches the profile mechanism, the Constraint schema, the OpenAPI propertyNames allowlists, or the v1alpha2/v1alpha3 version gate.

Merge order / dependency: resolved — #1998 merged first (2026-08-04) and this PR is rebased onto that main. The component-catalog paragraph now describes the readiness gate (NodeTopology.gpu-nodes.label, fail-closed, exit 2) instead of claiming no deterministic check exists, closing the loop the readiness remediation text points into.

Review-round fixes folded in at head: deprecated apimachinery label validators replaced with content.IsLabelKey/content.IsLabelValue (the vendored v0.36.3 marks the old aliases deprecated); a ValidatePhases readiness-preflight test pinning the default client path fail-closed (the plural entry point previously had zero coverage on that branch); an encode→decode round-trip test running real topology.Collector output (plain, disambiguated, truncated shapes) through the evaluator so the cross-package wire format cannot silently diverge before #2003; parse-validation table rows for a valid key with an invalid label value; key= (empty value) accepted as a valid positive form — Kubernetes permits empty label values, the collector already encodes them (|<nodes>), and the disambiguated map key <key>. cannot collide with a real label key (keys may not end in a dot) — pinned by table rows for the uniform and mixed shapes plus a collector round-trip leg; and a comment documenting checkReadiness's deliberate error-code flattening.

Fixes: #1755
Related: #1761, #1998 (merge first — see above)

Type of Change

• New feature (non-breaking change that adds functionality)
• Bug fix (non-breaking change that fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter) — exported truncation predicate only
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: recipes/overlays/gke-cos.yaml, tests/uat/gcp/cluster-config.yaml, pkg/constraints

Implementation Notes

The form is name-dispatched, not schema-extended. constraints.Evaluate dispatches on the exact name NodeTopology.gpu-nodes.label before the scalar path (precedent: the nccl-benchmark-* sentinel names). The constraint stays name + value, so both existing call sites (readiness, overlay/profile evaluation) get the form with no API or artifact-version change.

Value grammar is <label-key>=<value> (every GPU node carries it) or !<label-key> (no GPU node carries the key), validated with the vendored Kubernetes label validators (content.IsLabelKey/content.IsLabelValue) — so the scalar operator grammar (>= x, != x) cannot be misread as a key/value predicate, and a key no node can legally carry (e.g. a double slash) is rejected instead of making the negated predicate pass vacuously.

GPU-node universe = nodes under the snapshot's cloud.google.com/gke-accelerator* label readings — GKE's native signal, present from pool creation before the Operator or NFD run (NFD's nvidia.com/gpu.* labels don't exist on the pre-deployment cluster this check validates). Per the issue's acceptance requirements, both predicate directions fail closed on: truncated node lists (--max-nodes-per-entry snapshots), an empty universe (no vacuous pass), missing NodeTopology.label readings, and mixed/disambiguated label values (key.value encoding). Disambiguated-shape decoding enforces encodeLabels' invariants — plain and disambiguated forms never coexist, and genuine disambiguation always yields at least two entries — so a distinct dotted label whose value equals its own suffix (e.g. a literal <key>.true=true label) cannot satisfy the predicate; an accepted disambiguated set must additionally partition its nodes — overlapping node sets prove the encoder's <key>.<value> map key collided with a real label of that literal name (one reading silently overwritten, #2003) and fail closed as an ambiguous reading. A single disambiguated-shape entry without the plain key (a possible collision remnant) and structurally malformed readings (missing or extra | separators, empty node lists, node tokens that are not canonical RFC-1123 node names) also fail closed — both would otherwise let the negated predicate pass vacuously. The residual ambiguity (collisions with identical node sets, or every disambiguated entry overwritten) is what the lossy encoding cannot express; the lossless-encoding root-cause fix is tracked in #2003, structured truncation metadata in #2002. The truncation detector is owned by the collector (topology.IsTruncatedNodeList, next to formatNodeList, with a round-trip test) so the format and its detector cannot drift apart.

Both validator entry points are gated. checkReadiness runs in ValidatePhases and in the per-phase ValidatePhase (the exported SDK path Client.ValidateState documentation directs per-phase callers to) — a single-phase caller cannot bypass the recipe's readiness constraints, and declared readiness constraints with a nil snapshot fail closed instead of silently skipping. Both pinned by tests.

Wiring: readiness, not spec.constraints. A top-level constraint would exclude the GKE overlays during snapshot-based generation on exactly the unlabeled cluster the diagnostic exists to fix. The check lives in the GKE-COS base overlay's validation.readiness.constraints; checkReadiness now evaluates readiness-phase constraints alongside the top-level set (they were declared, merged, and carried into ValidationInput but never consumed) and appends the constraint's remediation to the failure message — that's where the issue's required device-plugin diagnostic lives.

UAT cluster config. The GCP UAT GPU pool (tests/uat/gcp/cluster-config.yaml) predates the documented prerequisite and does not carry the label; without fixing it this PR would fail the GKE UAT lane. The label is added to the pool's nodeConfig.labels. Note the config actuates at cluster provision time — the nightly lifecycle (provision→CUJ→teardown) picks it up on the next run; a held daytime cluster would need re-provisioning or a manual gcloud container node-pools update.

Sequencing with #1998. #1998 merges first; this PR rebases on it and refreshes the component-catalog paragraph (added there) that states "AICR has no deterministic check for a violation today" — kept out of this PR until the rebase to avoid cross-PR conflicts.

Testing

make qualify   # PASS (exit 0) through the collision-guard revision; on the final head the local run hit
               # pre-existing tests/releasepolicy deadline timeouts (reproduced identically on clean
               # origin/main — machine-local flake); affected packages re-verified with -race and CI is green
GOFLAGS="-mod=vendor" go test -race ./pkg/constraints/... ./pkg/validator/ ./pkg/recipe/...   # PASS
golangci-lint run -c .golangci.yaml ./pkg/constraints/... ./pkg/validator/...   # 0 issues

Coverage (per-package, current vs origin/main baseline):

• pkg/constraints: 97.9% → 97.9% (flat)
• pkg/validator: 48.5% → 50.9% (+2.4%)
• pkg/collector/topology: 84.2% → 84.4% (+0.2%)

CLI end-to-end (branch-built binary, no cluster): generated the GKE H100 training recipe with aicr recipe — the hydrated artifact carries the readiness constraint with its remediation — then ran aicr validate --no-cluster against four crafted snapshots: labeled pool → exit 0 ("all 2 GPU node(s) carry …"); unlabeled pool → exit 2 naming the offending nodes plus the full device-plugin remediation; truncated snapshot → exit 2 with the regenerate-without---max-nodes-per-entry instruction; no GPU nodes → exit 2 with the empty-universe diagnostic. Live-cluster validation deliberately skipped: the gate is inline (no cluster I/O), and the first post-merge nightly UAT provisions from this PR's cluster-config and runs aicr validate through the gate, providing the live confirmation automatically.

New tests: table-driven coverage of both predicate directions × {pass, fail, mixed values via disambiguated keys, multiple accelerator types, non-GPU nodes carrying the label, truncated universe reading, truncated target reading (both directions), empty universe (both directions), missing label subtype/measurement/nil snapshot, prefix-collision non-misattribution, scalar-grammar rejection, malformed values}; validator-level test that readiness-phase constraints are evaluated and carry remediation into the failure.

New exported symbols: the GPUNodesLabelConstraintName const and topology.IsTruncatedNodeList (covered by the round-trip test). make qualify was run unsandboxed after verifying the sandbox-only failures (cleanup_test.sh stubs, httptest port binds) reproduce identically on a clean origin/main checkout.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert
• Medium — Touches multiple components or has broader impact
• High — Breaking change, affects critical paths, or complex rollout

Rollout notes: A pre-change aicr binary evaluates only top-level constraints and silently ignores validation.readiness.constraints in a recipe generated at this commit (version skew is advisory-only) — validate recipes with a matching or newer binary. aicr validate against a GKE cluster whose GPU pools lack gke-no-default-nvidia-gpu-device-plugin=true now fails readiness closed (exit 2) — intended and documented, but visible to existing users of unlabeled clusters; the failure message carries the exact remediation. Snapshots captured with --max-nodes-per-entry also fail this constraint closed with a regenerate instruction. Recipe generation and bundling are unaffected. The UAT GKE pool config is updated in this PR; held daytime UAT clusters need one re-provision (or a manual pool update) before their next validate run.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S) — GPG signing info

[github-actions]

Recipe evidence check

Protected recipes

Recipes with committed evidence (recipes/evidence/<slug>/<source>/<digest>.yaml) that this PR affects: 1

Recipe	Source	Pointer	Verify	Digest match
h100-gke-cos-training	7c4c0edc8c765a95a0f3afdb3bbb8e91	sha256-be4680f26ad9ebeb57145f1953f18311ca00e81a4edb37773e0ec1060c6bd261	❌ invalid — registry-forbidden (HTTP 401): registry not accessible (make the fork's aicr-evidence package public, or provide registry credentials)	⚠️ skipped (no signed digest)
h100-gke-cos-training	7c4c0edc8c765a95a0f3afdb3bbb8e91	sha256-f2573e7f2496cc895e6a780604645f7c24ed4d7e0edf4c4845c0d341a3a6326e	❌ invalid — registry-forbidden (HTTP 401): registry not accessible (make the fork's aicr-evidence package public, or provide registry credentials)	⚠️ skipped (no signed digest)

Other affected recipes without evidence yet: 10

These recipes are affected by this PR but carry no committed evidence pointer, so there is
nothing to verify. This is expected — evidence is hardware-gated and added over time.

• a100-gke-cos-training-kubeflow
• a100-gke-cos-training
• b200-gke-cos-inference-dynamo
• b200-gke-cos-inference
• b200-gke-cos-training-kubeflow
• b200-gke-cos-training
• h100-gke-cos-inference-dynamo
• h100-gke-cos-inference
• h100-gke-cos-training-kubeflow
• h100-gke-cos-training-slurm

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
# Profiled families (AKS gpuStack): capture the pool projection and
# hydrate the recipe with the pointer's recorded 'profile:' selection
# first — validating the raw overlay resolves only the declaration
# default, and 'aicr validate' has no --profile flag:
#   az aks nodepool list -g <rg> --cluster-name <cluster> -o json > pools.json
#   aicr snapshot --aks-gpu-pools pools.json -o snapshot.yaml
#   aicr recipe -s snapshot.yaml --intent <intent> [--platform <platform>] \
#     --profile <name>=<value> -o recipe.yaml
# State the target leaf's intent/platform explicitly (the snapshot
# fingerprint supplies service/accelerator/OS but intent and platform
# default to 'any') and pass -r recipe.yaml below instead of the raw
# overlay.
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
# Copy to the per-source path printed in the emit 'copyTo' hint:
#   recipes/evidence/<slug>/<source>/<bundle-digest>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds the NodeTopology.gpu-nodes.label constraint with positive and negated predicates over GKE GPU nodes. The evaluator fails closed for missing, empty, mixed, or truncated node data. Readiness validation now evaluates readiness-specific constraints and includes remediation text in failures. The GKE-COS recipe and UAT configuration add the required device-plugin opt-out label. Documentation and tests cover the new constraint and readiness behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• NVIDIA/aicr#1998: Addresses the same GKE device-plugin ownership label.

Suggested reviewers: arangogutierrez

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: a fail-closed GKE device-plugin ownership check.
Description check	✅ Passed	The description directly explains the new constraint, readiness validation behavior, affected configurations, testing, and rollout impact.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/validator/validator_test.go`:
- Around line 355-410: Extend
TestCheckReadinessEvaluatesReadinessPhaseConstraints with a recipe containing
both a top-level Validation.Constraints entry and a
Validation.Readiness.Constraints entry; make one or both fail to verify both are
evaluated, confirm the resulting error includes the relevant remediation, and
snapshot validationInput.Constraints before checkReadiness to assert it remains
unchanged afterward, preserving the combined ordering and non-aliasing behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 508c292a-df5b-4790-8e79-48253edb795d

📥 Commits

Reviewing files that changed from the base of the PR and between 9c5062b and 7bac947.

📒 Files selected for processing (10)

• docs/integrator/recipe-development.md
• docs/user/validation.md
• pkg/constraints/doc.go
• pkg/constraints/evaluate.go
• pkg/constraints/gpu_nodes.go
• pkg/constraints/gpu_nodes_test.go
• pkg/validator/validator.go
• pkg/validator/validator_test.go
• recipes/overlays/gke-cos.yaml
• tests/uat/gcp/cluster-config.yaml

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/collector/topology/topology_test.go`:
- Around line 465-481: Refactor TestIsTruncatedNodeListRoundTrip into a
table-driven test containing the truncated, full, and exact-limit cases with
their expected results. Iterate over the cases using subtests, while preserving
the existing formatNodeList inputs and IsTruncatedNodeList assertions.

In `@pkg/constraints/gpu_nodes.go`:
- Line 22: Replace the deprecated validation import and validator calls in the
GPU node constraint logic with content.IsQualifiedName and content.IsLabelValue.
Remove the k8s.io/apimachinery/pkg/util/validation dependency while preserving
the existing validation behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 7863614d-d4f6-425b-b95e-81ca870595f2

📥 Commits

Reviewing files that changed from the base of the PR and between 7bac947 and f6cab51.

📒 Files selected for processing (14)

• docs/contributor/validator.md
• docs/integrator/recipe-development.md
• docs/user/validation.md
• pkg/collector/topology/topology.go
• pkg/collector/topology/topology_test.go
• pkg/constraints/doc.go
• pkg/constraints/evaluate.go
• pkg/constraints/gpu_nodes.go
• pkg/constraints/gpu_nodes_test.go
• pkg/validator/doc.go
• pkg/validator/validator.go
• pkg/validator/validator_test.go
• recipes/overlays/gke-cos.yaml
• tests/uat/gcp/cluster-config.yaml

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/constraints/gpu_nodes_test.go`:
- Around line 397-409: Refactor TestSummarizeNodesCapsList into a table-driven
test containing both the capped seven-node case and the uncapped single-node
case, then iterate through the table with named subtests while preserving the
existing expected outputs.

In `@pkg/constraints/gpu_nodes.go`:
- Around line 204-208: Update the topology label decoding around
cutLabelEncoding so readings missing the separator, with an empty node list, or
containing empty node members are rejected as malformed validation data.
Propagate the decoding error through evaluateNoGPUNodeHasKey so both positive
and negated checks fail closed instead of producing an empty labelNodeSet. Add
regression tests covering positive and negated evaluations for each malformed
form.
- Around line 134-140: Update the node-set constraint parsing validation around
strings.Cut to remove the want == "" rejection, allowing key= values to be
validated by validation.IsValidLabelValue. Add a test case covering a valid
empty label value and preserve rejection of malformed inputs through the
existing validation flow.

In `@pkg/validator/validator.go`:
- Around line 80-82: Update the remediation handling in the validator
error-message construction to trim c.Remediation before checking whether it is
non-empty, then append the trimmed value only when it contains content. Preserve
the existing newline formatting for valid remediation text, and add a test
covering whitespace-only Remediation to ensure it does not append an extra
newline.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: de16349f-e220-4240-ab15-89216983aa1d

📥 Commits

Reviewing files that changed from the base of the PR and between f6cab51 and a663fdd.

📒 Files selected for processing (15)

• docs/contributor/validator.md
• docs/integrator/recipe-development.md
• docs/user/validation.md
• pkg/collector/topology/topology.go
• pkg/collector/topology/topology_test.go
• pkg/constraints/doc.go
• pkg/constraints/evaluate.go
• pkg/constraints/extractor.go
• pkg/constraints/gpu_nodes.go
• pkg/constraints/gpu_nodes_test.go
• pkg/validator/doc.go
• pkg/validator/validator.go
• pkg/validator/validator_test.go
• recipes/overlays/gke-cos.yaml
• tests/uat/gcp/cluster-config.yaml

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/contributor/validator.md`:
- Around line 47-53: Update the later pre-flight section describing
checkReadiness to state that it evaluates both top-level validation.constraints
and validation.readiness.constraints, matching the contract documented near the
phase checks. Remove any wording that limits pre-flight evaluation to top-level
constraints only.

In `@pkg/validator/validator_test.go`:
- Around line 416-439: Update the aliasing coverage in the checkReadiness test
around the topLevel setup and post-check assertions: construct topLevel with
spare capacity and initialize its unused tail with a sentinel value, then verify
after checkReadiness that both its length and unused tail remain unchanged. Keep
the existing error assertions and ensure the test would fail if checkReadiness
appends readiness constraints into the aliased backing array.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: a168baec-79c7-4640-a371-4a6cb526e2a1

📥 Commits

Reviewing files that changed from the base of the PR and between a663fdd and 26e0748.

📒 Files selected for processing (15)

• docs/contributor/validator.md
• docs/integrator/recipe-development.md
• docs/user/validation.md
• pkg/collector/topology/topology.go
• pkg/collector/topology/topology_test.go
• pkg/constraints/doc.go
• pkg/constraints/evaluate.go
• pkg/constraints/extractor.go
• pkg/constraints/gpu_nodes.go
• pkg/constraints/gpu_nodes_test.go
• pkg/validator/doc.go
• pkg/validator/validator.go
• pkg/validator/validator_test.go
• recipes/overlays/gke-cos.yaml
• tests/uat/gcp/cluster-config.yaml