recipes: GKE COS assumes a preinstalled NVIDIA driver — add opt-in gcp-nvidia-driver component

[yuanchen8911]

Problem

The GKE COS recipe chain assumes a preinstalled NVIDIA driver and provides no way to install one:

• recipes/overlays/gke-cos.yaml — "GKE preinstalls NVIDIA drivers on COS; disable driver installation"
• recipes/components/gpu-operator/values-gke-cos.yaml — driver.enabled: false, toolkit pointed at GKE's install path (/home/kubernetes/bin/nvidia)
• No driver-installer component exists anywhere in the chain (gke-cos → gke-cos-training → leaf overlays), and the GPU Operator cannot take over: it does not support driver installation on COS.

This assumption matches vanilla GKE, where GKE-managed driver installation (nodepool gpu-driver-version) is the platform default. It breaks on clusters provisioned outside that default — e.g. DGXC-provisioned GKE, where GPU nodes carry gke-no-default-nvidia-gpu-device-plugin=true and GKE's native install path does not run. On such clusters nothing installs the driver, and the GPU Operator operands fail against a driverless node. Today DGXC/AOR compensates by running its own nvidia-driver-installer DaemonSet (cos-gpu-installer) targeting the labeled nodes — so AICR bundles only work there because an external system pre-satisfies the assumption.

There are two ways a driver lands on GKE COS: GKE-managed install configured at nodepool creation (gpu-driver-version), or running the cos-gpu-installer DaemonSet in-cluster (what AOR does). AICR supports consuming either but can produce neither.

Proposal

Add a gcp-nvidia-driver component — Google's cos-gpu-installer DaemonSet (container-engine-accelerators nvidia-driver-installer/cos/daemonset-preloaded.yaml, pinned) — as a declarative manifest/Kustomize component, exposed as an explicit opt-in mixin (recipes/mixins/gcp-nvidia-driver.yaml):

• Public GKE leaf overlays keep the vanilla-GKE preinstalled-driver default (no behavior change).
• Recipes for non-default clusters (e.g. internal/NKX) opt in by including the mixin — mirroring what AOR runs today, but owned by the recipe.
• gpu-operator gains a dependencyRefs on the installer when present, so the driver lands before operator operands need it.
• Optional hardening: a preflight validation constraint with an actionable message when no driver is present and no installer component is in the recipe ("preinstall the driver (GKE default) or add the gcp-nvidia-driver mixin"), instead of a late operand crash.
• Docs: integrator page for the non-default GKE profile, mirroring docs/integrator/aks-gpu-setup.md's two ownership modes.

Explicit opt-in rather than label autodetection, per the #1327 contract (configuration selects, inspection verifies).

Naming: the component is deliberately named gcp-nvidia-driver (not the earlier working name gke-driver-installer) so it stays distinct from the driver-installer gpuStack profile value introduced by #2098. The profile value names the cluster shape where Google's standalone DaemonSet supplies the driver as an external prerequisite; this component is the AICR-owned recipe artifact that produces that DaemonSet.

Context

Surfaced by NKX bring-up on DGXC GKE (Slack thread in #aicr, 2026-07-10): AICR GCP images are not usable by NKX until the driver gap is closed by one of: AOR continuing to install the driver (works today, documented prerequisite), or this component.

Related

• GPU allocation policy: device-plugin production default with recipe-configured DRA opt-in #1327 (GPU allocation policy contract — the adjacent but distinct device-plugin/label discussion from the same thread)
• refactor(recipe): rename GKE gpuStack profile values #2098 (renames the GKE gpuStack values to gke-default / driver-installer — the value names used below)
• docs/integrator/aks-gpu-setup.md (precedent: documenting per-CSP driver ownership modes)

Scope update (2026-08-06): driver-installer gpuStack qualification folds in here

ADR-015 PR 3 (#2044, merged 2026-08-05) converted the GKE family to the gpuStack profile. Its driver-installer value (renamed from operator-managed by #2098) requires precisely the cluster shape this issue is about: every GPU node pool carries gke-no-default-nvidia-gpu-device-plugin=true, pools are created with gpu-driver-version=disabled, and Google's standalone nvidia-driver-installer DaemonSet supplies the driver — today an external prerequisite (what AOR runs), not something an AICR bundle produces.

That leaves two coupled deliverables, and this issue now tracks both:

• The gcp-nvidia-driver opt-in component/mixin as proposed above, so an AICR recipe can produce the driver instead of assuming it.
• Live validation of the driver-installer profile value plus its per-value v2 evidence signing — the follow-up named in feat(recipe): convert GKE family to the gpuStack profile (ADR-015 PR 3) #2044's rollout notes and Implement ADR-015: recipe-declared configuration profiles #1761's closing comment. Nightly UAT exercises only the gke-default default, so this needs a dedicated cluster window with the non-default pool shape; qualifying the value on a cluster where the recipe itself installs the driver (via the new component) closes both gaps in one pass.

Sequencing: land the component first, then qualify driver-installer on top of it — otherwise the qualification depends on externally pre-satisfied cluster state and proves less than it should.

[hiSandog]

The opt-in component should distinguish clusters with no driver from clusters where a compatible driver is already managed by GKE, so installation remains idempotent. A recipe validation case for mixed GPU and non-GPU node pools would be useful because a blanket daemon configuration can behave differently across taints and architectures. The docs could also state the supported COS and driver version combinations rather than implying the latest pair is always compatible.