Skip to content

constraints: NodeTopology.gpu-nodes.label GPU-node universe is GKE-only #2359

Description

@yuanchen8911

pkg/constraints/gpu_nodes.go defines the authoritative GPU-node universe as a single hardcoded label:

// gpuNodeUniverseLabel defines the authoritative GPU-node universe: nodes
// carrying GKE's native accelerator label.
const gpuNodeUniverseLabel = "cloud.google.com/gke-accelerator"

On any non-GKE service the universe comes back empty and the evaluator fails closed by design, so the NodeTopology.gpu-nodes.label constraint form (#1755) is unusable on OKE, AKS, and EKS today.

This is not an accidental hardcode. The comment states the rationale: the GKE label is present from pool creation, before the GPU Operator or NFD run, which is precisely the pre-deployment cluster this form validates. NFD's nvidia.com/gpu.* labels do not exist on such a cluster, so they cannot serve as the universe.

Why it matters now

The ADR-015 gpuStack profile work is expanding beyond GKE, and the label form is the natural way to express pool-shape preconditions. The OKE profile work has already had to route around it — its analysis notes the universe "is hardcoded to GKE's cloud.google.com/gke-accelerator; it can't run on OKE without evaluator work," and that OKE's own disable-gpu-device-plugin label isn't set by the provisioning Terraform.

Reachability today is GKE-only, so this is design debt rather than a cross-provider blocker. No current consumer needs the form on another service:

  • AKS reaches pool shape by a different route — the snapshot-side --aks-gpu-pools provider projection — so it does not depend on this form.
  • OKE would be only partly served, and only conditionally. Generalizing the universe enables the node-label disablement route — but that label (oci.oraclecloud.com/disable-gpu-device-plugin) is not set by the provisioning Terraform today (NKX-9804), and it does not cover cluster-level add-on removal, which leaves no node marker at all. If OKE's supported disablement path turns out to be add-on removal, the profile needs an add-on-state projection instead and this issue is not on its path. Whether it is on the path is decided by the OKE profile design discussion, not here.
  • EKS has no consumer for it at all today.

Nothing in flight is blocked: #2347 is service-agnostic and does not touch the evaluator, and the GKE work is on the supported path. Filing so the constraint is not rediscovered a third time, and so a future non-GKE profile does not assume the form is available.

What's needed

A per-service definition of the pre-NFD GPU-node universe, keyed off the service recomputed from the snapshot's measurements (pkg/fingerprint.FromMeasurements, sourced from k8s.node.provider) rather than a single constant. Open questions:

  • AKS, EKS, OKE: which label — or instance-type prefix — is reliably present at pool creation, before any operator runs?
  • Where does the mapping live: a Go table keyed by service, or declarative data alongside the registry?
  • What is the failure mode for a service with no such marker? Today's fail-closed empty universe is defensible, but it should be explicit and diagnosable rather than looking like a snapshot problem.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/validatortheme/validationConstraint evaluation, health checks, and conformance evidence

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions