feat: CDI-based GPU passthrough into gVisor actor containers - #1
Open
eliranw wants to merge 181 commits into
Open
feat: CDI-based GPU passthrough into gVisor actor containers#1eliranw wants to merge 181 commits into
eliranw wants to merge 181 commits into
Conversation
This enables parsing of flags like `-test.v` to pass into the Go test logic.
…gent-substrate#464) ## Summary Bumps `github.com/google/go-containerregistry` v0.21.5 → v0.21.7. Phase 0 of agent-substrate#463. v0.21.6 fixed the memory spike in `mutate.Extract` described in agent-substrate#120 (upstream fix: google/go-containerregistry#2190, merged as google/go-containerregistry@38d6e4087c): extraction is now refactored into a per-layer `extractLayer` function, so each layer's decompressor and HTTP response body are closed as soon as that layer is processed, instead of all being held open via deferred `Close` calls until the entire (multi-GB) extraction completes. atelet's pull path goes through `memorypullcache.Fetch` → `mutate.Extract`, so this directly caps its extract-time memory. Note this is only interim relief: agent-substrate#437's unbounded cache retention is untouched and is addressed by the redesign in agent-substrate#463. ## Changes - `go-containerregistry` v0.21.5 → v0.21.7, plus required transitive bumps: `docker/cli`, `klauspost/compress`, `golang.org/x/sync`, `golang.org/x/sys` - `go mod tidy` + `go mod vendor`; the `containerd/stargz-snapshotter/estargz` vendor tree drops out (upstream removed the estargz integration; nothing in substrate used it), along with `vbatts/tar-split` and `mitchellh/go-homedir` ## Testing - `go build ./...` passes - Full `go test ./...` passes - Verified the layer-close fix is present in the vendored `pkg/v1/mutate/mutate.go` Fixes agent-substrate#120 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…-substrate#412) This PR sets consistent, namespaced actor-identity set on the spans we already emit: - `ate.atespace`, `ate.actor.id`, `ate.actor.template.{name,namespace}`, `ate.actor.version` - ateapi: the RPC server span for create/resume/suspend/pause/delete We do this, so platform traces are queryable by actor, atespace, and template. Cross-hop correlation and per-tenant/per-template filtering, all on traces (not TSDB labels, as discussed in agent-substrate#174). This is a general, workload-neutral telemetry identity plumbing. We could add gen_ai specific stuff on top of this later. This is a resume: <img width="1913" height="1010" alt="image" src="https://github.com/user-attachments/assets/876cf689-a8ec-4f37-b035-3b9b1de87d59" /> **Notes:** - Left as it was on purpose: metric attribute names and the stdout log labels (`ate.dev/*`), but we should probably do this as well. Maybe in this PR? - Bikeshed is welcomed on the `ate.*` spelling before merge, renaming span attrs later is painful. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…mand` and `args` (agent-substrate#462) * Add `args` field in addition to `command`. * Implement logic in atelet to resolve both argv from the actor template fields and the OCI image config (`entrypoint` and `cmd` fields). * Update demos to not hardcode command as now we honor the image default, which makes it not necessary. Fixes agent-substrate#456
…nt-substrate#474) When the worker pod disappeared before Checkpoint was called, the suspend workflow skipped the atelet call but still let finalize promote the reserved InProgressSnapshot URI to LatestSnapshotInfo, overwriting the last good snapshot with one that was never written. This could break the next resume call of this actor. Now the dangling-worker path crashes the actor via `crashActor` too, and crashActor clears any reserved snapshot URI so `FinalizeSuspendedStep` won't overwrite the last good snapshot. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…agent-substrate#374) Fixes agent-substrate#369 Add a CheckPrerequisite method to the WorkflowStep interface, called by RunWorkflow after IsComplete returns false and before Execute, so that each workflow validates its actor state-machine edge up front while retried (reentrant) workflows still fast-forward past completed steps.
…g picked and persisted worker assignment (agent-substrate#478) Fixed flakey integration test at head. Broken test in https://github.com/agent-substrate/substrate/actions/runs/29795626928/job/88526261507 was because, once a old version of worker has been picked and saved in state.worker, retrying no longer picked a new, usable worker. Updated the implementation to only update state.worker after having successfully picked and persisted the worker.
…or. (agent-substrate#475) This will free up the worker to which the crashed actor was assigned, so it can be picked up by other actors. Minor fix: * Added functional test to make sure a crashed actor can be deleted.
An initial version of code convention guide for human and agents. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
On macOS, bash default has Bash 3.2 compatibility. mapfile isn't available and the mapfile command is breaking the install-ate.sh. Use a while loop to implement the exact behavior.
eliranw
force-pushed
the
eliranw/gvisor-gpu-poc
branch
3 times, most recently
from
July 22, 2026 13:19
d6d5e88 to
ce4a279
Compare
…#358) The tracing best practices doc points to cmd/ateapi/ateapi.go:initTracing() as the example, but that file/function does not exist. Servers now call internal/serverboot.InitTracing(), used in cmd/ateapi/main.go. Verified internal/serverboot.InitTracing exists and is called from cmd/ateapi/main.go:80.
The JWKS EC branch only ever returned an error, so an issuer publishing an EC key failed verification entirely (even for its RS256 tokens). Parse P-256/384/521 and add the package's first tests. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…ck/tools/ko (agent-substrate#497) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.81.1 to 1.82.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/grpc/grpc-go/releases">google.golang.org/grpc's releases</a>.</em></p> <blockquote> <h2>Release 1.82.1</h2> <h1>Security</h1> <ul> <li>server: Stop reading from the connection when flooded by HTTP/2 frames. The default value for this limit is 100 frames, excluding DATA and HEADERS, and may be changed by setting environment variable <code>GRPC_GO_EXPERIMENTAL_CONTROL_BUFFER_THROTTLE_LIMIT</code>.</li> <li>xds/rbac: Support <code>Metadata</code> and <code>RequestedServerName</code> permissions matcher fields. If present in a DENY rule, previously these would be ignored and fail-open.</li> <li>xds/rbac: Fix panic when parsing unsupported fields in <code>NotRule</code>/<code>NotId</code> permissions.</li> <li>xds/rbac: Support the deprecated <code>source_ip</code> principal identifier by treating it as equivalent to <code>direct_remote_ip</code>.</li> </ul> <h2>Release 1.82.0</h2> <h1>Behavior Changes</h1> <ul> <li>server: Remove support for <code>GRPC_GO_EXPERIMENTAL_DISABLE_STRICT_PATH_CHECKING</code> environment varibale. Strict incoming RPC path validation (which has been the default since <code>v1.79.3</code>) can no longer be disabled. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9112">#9112</a>)</li> <li>transport: Add environment variable to change the default max header list size from <code>16MB</code> to <code>8KB</code>. This may be enabled by setting <code>GRPC_GO_EXPERIMENTAL_ENABLE_8KB_DEFAULT_HEADER_LIST_SIZE=true</code>. This will be enabled by default in a subsequent release. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9019">#9019</a>)</li> <li>balancer: Load Balancing policy registry is now case-sensitive. Set <code>GRPC_GO_EXPERIMENTAL_CASE_SENSITIVE_BALANCER_REGISTRIES=false</code> (and file an issue) to revert to case-insensitive behavior. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9017">#9017</a>)</li> </ul> <h1>New Features</h1> <ul> <li>experimental/stats: Expose a new API, <code>NewContextWithLabelCallback</code>, to register a callback that is invoked when telemetry labels are added. (<a href="https://redirect.github.com/grpc/grpc-go/issues/8877">#8877</a>) <ul> <li>Special Thanks: <a href="https://github.com/seth-epps"><code>@seth-epps</code></a></li> </ul> </li> <li>client: Return a portion of the response body in the error message, when the client receives an unexpected non-gRPC HTTP response, to make debugging easier. (<a href="https://redirect.github.com/grpc/grpc-go/issues/8929">#8929</a>) <ul> <li>Special Thanks: <a href="https://github.com/chengxilo"><code>@chengxilo</code></a></li> </ul> </li> <li>server: Add environment variable <code>GRPC_GO_SERVER_GOROUTINE_LABELS</code> that controls setting <code>runtime/pprof.Labels</code> on goroutines spawned by the server. Set <code>GRPC_GO_SERVER_GOROUTINE_LABELS=grpc.method=true</code> to add the <code>grpc.method</code> label on goroutines spawned to handle incoming requests. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9082">#9082</a>) <ul> <li>Special Thanks: <a href="https://github.com/dfinkel"><code>@dfinkel</code></a></li> </ul> </li> </ul> <h1>Bug Fixes</h1> <ul> <li>xds/server: Fix a memory leak of HTTP filter instances occurring when route configurations are updated in-place during a Route Discovery Service (RDS) update. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9138">#9138</a>)</li> <li>grpc: In the deprecated <code>gzip</code> Compressor (used via the deprecated <code>WithCompressor</code> dial option), enforce the <code>MaxRecvMsgSize</code> limit on the decompressed message buffer, preventing excessive memory allocation from highly compressed payloads. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9114">#9114</a>) <ul> <li>Special Thanks: <a href="https://github.com/evilgensec"><code>@evilgensec</code></a></li> </ul> </li> <li>stats/opentelemetry: Record retry attempts, <code>grpc.previous-rpc-attempts</code>, at the call level and not the attempt level. (<a href="https://redirect.github.com/grpc/grpc-go/issues/8923">#8923</a>)</li> <li>encoding: Ensure <code>Close()</code> is always called on readers returned from <code>Compressor.Decompress</code> if possible. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9135">#9135</a>)</li> <li>channelz: Fix the <code>LastMessageSentTimestamp</code> and <code>LastMessageReceivedTimestamp</code> fields in <code>SocketMetrics</code> to ensure they contain correct timestamp values. (<a href="https://redirect.github.com/grpc/grpc-go/issues/9109">#9109</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/grpc/grpc-go/commit/ebd8f06a09426fbece97157c95c3917abff28f4e"><code>ebd8f06</code></a> Change version to 1.82.1 (<a href="https://redirect.github.com/grpc/grpc-go/issues/9238">#9238</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/4ea465d4ab98013f72a142fe0fc89c19770b2935"><code>4ea465d</code></a> Cherry-pick commits (<a href="https://redirect.github.com/grpc/grpc-go/issues/9236">#9236</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/9494a2cf32a0ec9d35420af401445ef3c9f66f05"><code>9494a2c</code></a> Change version to 1.82.1-dev (<a href="https://redirect.github.com/grpc/grpc-go/issues/9171">#9171</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/bd239854f0ab7f1ee63457d47f7c1d2675e1f736"><code>bd23985</code></a> Change version to 1.82.0 (<a href="https://redirect.github.com/grpc/grpc-go/issues/9170">#9170</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/0f3086db7a755b6af83a90809471dd7f645b345a"><code>0f3086d</code></a> Fix minor issues not covered by PR <a href="https://redirect.github.com/grpc/grpc-go/issues/9137">#9137</a> (<a href="https://redirect.github.com/grpc/grpc-go/issues/9147">#9147</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/fef07fbb2b94b668e8daca1f6b70433dcd36c1c8"><code>fef07fb</code></a> internal: Split v3procservicepb import into pb and grpc for extproc (<a href="https://redirect.github.com/grpc/grpc-go/issues/9163">#9163</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/91dd64f4b83cb5134e279d1126ebb1ccf47d4d31"><code>91dd64f</code></a> transport: surface subsequent data when receiving non-gRPC header (<a href="https://redirect.github.com/grpc/grpc-go/issues/8929">#8929</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/adc97de9521a9f377dab5e911039842dc4de23e5"><code>adc97de</code></a> test/kokoro: add config for regional-td test (<a href="https://redirect.github.com/grpc/grpc-go/issues/9158">#9158</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/57c9ff14e05b535ee6995ba49bc882b287a175de"><code>57c9ff1</code></a> xds: ensure full-string matching for RBAC Filter rules (<a href="https://redirect.github.com/grpc/grpc-go/issues/9148">#9148</a>)</li> <li><a href="https://github.com/grpc/grpc-go/commit/b58f32d9ff07c612d64e677bd826bcbec88af9bd"><code>b58f32d</code></a> server: Set a pprof label on new stream goroutines (<a href="https://redirect.github.com/grpc/grpc-go/issues/9082">#9082</a>)</li> <li>Additional commits viewable in <a href="https://github.com/grpc/grpc-go/compare/v1.81.1...v1.82.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/agent-substrate/substrate/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Balance large-PR breakdown against small/bulk-PR consolidation. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…llcache Content-addressed pool of unpacked image layers, shared by every actor on the node; actor rootfs becomes an overlayfs mount (cached layers as read- only lowers, bundle-local upper) instead of a full re-untar per run. atelet (no capabilities) pulls and unpacks; the privileged ateoms finalize whiteouts and mount. Tag refs resolve via one HEAD and become cacheable; pull memory is O(stream buffers); the cache survives restarts. Phase 1 of agent-substrate#463. Fixes agent-substrate#437, agent-substrate#166, agent-substrate#228. Validated: kind + GKE counter demos (gvisor and microvm), suspend/resume (oci_unpack ~3ms vs ~15-20s), 411 SWE-bench-scale images pulled and unpacked with 0 failures, root-gated unit tests for the privileged paths. Known gaps: no GC yet (Phase 2, see internal/imagecache/README.md); upgrade ordering — deploy new ateoms before/with the new atelet.
The fixed 30-minute idle guard made nothing evictable while a fast corpus filled a small disk (32-VM hicard sweep: local SSDs filled in ~12 minutes, then every remaining image failed ENOSPC). Expose it as --evict-idle.
Unmount runs before cleanupActorNetworkOrExit, which exits the process on failure and would otherwise skip the overlay detach.
…ayer cap mount(2) copies its option string through a single page, which caps digest-derived lowerdir chains (~114 bytes per layer path) at roughly 34 layers and fails with a bare EINVAL beyond that. Appending lowerdirs one fsconfig(2) call at a time removes the aggregate limit structurally, and failed mounts now carry the kernel's fs-context error log instead of an opaque errno. Adds a 64-layer regression test that asserts the joined paths exceed one page before mounting. Minimum supported kernel: Linux 6.5 (overlayfs "lowerdir+"). All current GKE channels meet it: Stable runs COS 121 LTS (kernel 6.6), Regular and Rapid run COS 125/129 (kernel 6.12).
Startup recovery swept layer unpack temp dirs but not writeRecord's ".<digest>.json.tmp-*" files left by a crash between create and rename. Addresses PR agent-substrate#467 review feedback.
Layer tars omit parents that exist in lower layers; unpack fabricates them (root:root 0755) and overlayfs takes merged dir attrs from the top-most layer containing the dir — so a fabricated parent shadowed real lower-layer metadata (/tmp's 1777, /root's 0700). Record implicit dirs in the layer metadata at unpack, and at compose repair the merged view from the top-most non-implicit provider in the image's chain; the chown/chmod copy-ups land in the bundle's private upper, never in the shared pool. Residual gaps (mtimes, xattrs, implicit-everywhere dirs) documented in the README. Addresses PR agent-substrate#467 review feedback (phantom parent dirs shadowing lower- layer directory metadata).
eliranw
force-pushed
the
eliranw/gvisor-gpu-poc
branch
4 times, most recently
from
July 23, 2026 14:17
b510ec9 to
2693528
Compare
…on labels (agent-substrate#642) Fixes agent-substrate#564 (Part 2) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Description This PR implements the second part of agent-substrate#564 by adding telemetry counter instrumentation for `ate.actor.crashes` in `ateapi` and establishing bounded label validation for operations and crash failure causes. ## Key Changes: - Updated `maybeCrashActor` and `crashActor` to reference `ateattr.OperationName*` constants and normalize `opName`. - Passed explicit failure reasons (`ReasonCorruptedAssignment`, `ReasonWorkerPodGone`, `ReasonWorkerReassigned`) across `workflow_resume.go`, `workflow_suspend.go`, and `workflow_pause.go`. - Instrumented `WorkerPoolSyncer.releaseActorOnDeadWorker` in `syncer.go` to record `recordActorCrash` when background pod deletion crashes an actor. - Instrumented `FinalizePausedStep` in `workflow_pause.go` to record `recordActorCrash` when `nodeName` is missing during pause finalization. - Added `ate_actor_crashes` to `PlatformMetricPrefixes` in `collector_metrics.go`. - Added pre-creation resource cleanup in `metrics_test.go` to prevent `AlreadyExists` errors on test reruns. - Updated `metrics_test.go` to trigger an actor crash via `UpdateActor` and `ResumeActor`, asserting `ate_actor_crashes` emission and label presence in OTel collector scrape outputs. ## Testing * `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...` * `go test -buildvcs=false ./cmd/atenet/internal/router/...` * `make test` ## E2E Test * `./hack/create-kind-cluster.sh` * `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter` * `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
Removes unnecessary locking in `UpdateActorSnapshotTag`
agent-substrate#714) > Split out of agent-substrate#487, which bundled three unrelated changes. ## Summary Envoy's end-to-end timeout on the workload route is hardcoded at `10s` in `buildRoutes`. An actor that legitimately holds a request open longer gets cut off: a harness relaying an LLM completion keeps the request open for the whole generation, and the client sees a **504 mid-turn**. Adds `--route-timeout` on atenet-router, and pairs it with a route-level `idle_timeout` so the ceiling is actually reachable. **The default is 10s, so behavior is unchanged** unless an operator passes the flag. ## Why the route timeout alone was not enough Raised in review by @LiorLieberman and @yan-vlasov, and they were right — the first version of this PR did not do what it claimed. We never set `stream_idle_timeout` on the HTTP connection manager, so Envoy applies its default of **5 minutes**. Per the HCM proto, that default is "overridable by the route-level `idle_timeout`", and when it fires "the stream is terminated with a 408 Request Timeout error code if no upstream response header has been received, otherwise a stream reset occurs." That is exactly this PR's case. A turn relaying a non-streaming completion sends no bytes at all while the actor is thinking, and a request parked across a suspend/resume is idle by the same measure. Both are progressing; Envoy cannot tell. So `--route-timeout=30m` would still have been cut at 5 minutes with a 408 — the knob would have looked like it worked and silently not. `routeIdleTimeout()` therefore resolves the accompanying idle timeout as `max(routeTimeout, 5m)`. Taking the larger keeps the operator's ceiling honest without ever making the idle timer *stricter* than it is today: below 5 minutes the route timeout fires first regardless, so at the 10s default this is a no-op. Route-level rather than HCM-level, so it stays scoped to workload traffic instead of every stream through the router. It is derived rather than exposed as a second `--route-idle-timeout` flag so the two cannot drift apart, with one silently defeating the other — happy to make it explicit if reviewers prefer. For naming: what this PR sets is the route-level `timeout`, which bounds upstream response time. Envoy's HCM `request_timeout` bounds how long the *request* takes to be received, which is not the limit in question here. ## Changes `cmd/atenet/internal/router/` — adds `XdsServer.routeTimeout` with a `SetRouteTimeout` setter and a `defaultRouteTimeout` const, wired from `routerConfig.RouteTimeout` / `--route-timeout`. Same shape as the adjacent `SetExtProcMessageTimeout` and `SetExtProcMaxRequests`, and a flag on the existing config struct rather than an env read, matching the convention the parked-request work established. Wired in `startEnvoyDataplane`. Adds `envoyDefaultStreamIdleTimeout` (5m) and `routeIdleTimeout()`, applied as the route's `IdleTimeout` in `buildRoutes`. A non-positive value leaves the default in place, since Envoy reads a zero route timeout as *no timeout at all*. The knob bounds the actor's own handling time only. The resume that may precede a request is covered by request parking and the ext_proc message timeout, both of which already derive from `--parked-request-budget`. `manifests/ate-install/atenet-router.yaml` documents it as a commented-out entry. ## Verification - `go build ./...`, `go vet ./...`, `go test ./...` — all pass. - `xds_test.go` reads the timeout back out of `buildRoutes`, where Envoy actually picks it up: default, setter override, and non-positive-keeps-default. The helper pins that route to `OriginalDstClusterName` — a change that moved actor traffic onto some other route would otherwise leave the test passing while the timeout governed a route nothing uses. - Two added subtests cover the pairing: `IdleTimeoutTracksLongerRouteTimeout` and `IdleTimeoutKeepsEnvoyDefaultWhenRouteTimeoutIsShorter`. - **On a live GKE cluster**, read back out of Envoy's own `/config_dump`. With the new image and no flag, the workload route reports `timeout: 10s`, so the default is genuinely unchanged. With `--route-timeout=5m` it reports `timeout: 300s`. Same binary, same manifest, only the flag differs. Caveat on that measurement: it was taken before `ingress: route actor ingress through the atunnel mTLS server` landed, so the route it read was the old `dynamic_forward_proxy` path to pod-IP:80. After rebasing, the timeout attaches to the `actor_original_dst` route that replaced it — which is now pinned by the test above rather than left to inspection. The `idle_timeout` pairing has test coverage only, not a live `/config_dump` read. - **Regression, resume with parking on the path:** a conversation actor that had been suspended for 4 days was resumed by an ordinary request through the router — HTTP 200 in 3.74s, exactly one parked request, `parking_wait_duration_seconds{outcome="served"} = 3.459s`, no shed and no `budget_exhausted`. ## Follow-up Per-ActorTemplate (or per-request) configurability, raised by @ronlv10: agreed it needs an API and is follow-up shaped rather than something to fold in here. The global flag remains useful as the cluster-wide ceiling. ## Relationship to agent-substrate#465 This is a stopgap for the connected-socket suspend/restore problem tracked in **agent-substrate#465 (suspend-safe actor networking)**. Once actor network traffic survives checkpoint/restore natively, much of the need to raise this ceiling should go away; this just makes the current behavior tunable in the meantime. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Maya Wang <mymaya@google.com>
#### What this PR does / why we need it: `atelet` previously had no `SIGTERM` handling — `main()` ended in a bare `svr.Serve(lis)`, so any pod eviction, `DaemonSet` rollout, or node drain killed the process instantly, aborting in-flight `Run`/`Checkpoint`/`Restore` RPCs mid-execution. This PR adopts the same graceful-shutdown pattern `ate-api` uses: on `SIGTERM`, mark not-ready, stop accepting new RPCs, let in-flight RPCs finish, and force-stop after a deadline. - `cmd/atelet/main.go`: - `signal.NotifyContext` (kept separate from the work ctx so in-flight RPCs aren't cancelled the moment `SIGTERM` arrives) + a local `drainOnShutdown` mirroring `ateapi`'s: `MarkNotReady` → sleep `--drain-delay` → `GracefulStop()` → force `Stop() `after `--drain-timeout`.` /readyz` and `/healthz` are wired into the metrics server (`serverboot.Readiness`), so readiness and liveness diverge correctly during the drain. - Flags: `--drain-delay` (default **`0s`** — `atelet` is dialed directly by pod IP, no route-drain window is needed) and `--drain-timeout` (default **`5m`** — `Checkpoint`/`Restore` stream multi-GiB snapshots to object storage and can take minutes; force-cancelling one mid-upload crashes the actor). - `manifests/ate-install/atelet.yaml`: `terminationGracePeriodSeconds: 330 `(drain-delay + drain-timeout + slack — the sum must fit inside the grace period or the kubelet `SIGKILLs` mid-drain), the drain flags, and `/readyz` readiness + `/healthz` liveness probes. **Test Scenarios Considered:** | In-flight at `SIGTERM` | Force-stop (past timeout) | |---|---| | Idle | n/a | | Checkpoint (suspend, external) | upload aborted → `CRASHED` (agent-substrate#362) | | Pause (local checkpoint) | aborted → crash | | Restore (resume) | RPC fails → ateapi retries | | Run (cold boot) | RPC fails → retried | | Multiple concurrent RPCs | one shared timeout; stragglers cut | | New RPC during drain | rejected `Unavailable` → ateapi retries | **Testing:** - Unit (`cmd/atelet/main_test.go`): a loopback gRPC server with a blocking handler holds an RPC in-flight across the drain. - `TestDrainOnShutdownInFlightFinishes` — in-flight RPC completes during `GracefulStop`; readiness flips to not-ready. - `TestDrainOnShutdownForceStopsAfterTimeout` — an RPC running past drain-timeout is force-cancelled by Stop(). - Live on a kind cluster (`hack/install-ate-kind.sh --deploy-atelet`, then `kubectl delete pod` to deliver `SIGTERM`, with `--drain-delay=25s` temporarily set to make the window observable): - Steady state: `/readyz=200`, `/healthz=200` (old build had no /readyz at all). - During drain:`/readyz=503` while `/healthz=200` for the whole window — `NotReady` but alive. - Log sequence in order, with the 25s drain-delay honored exactly: Shutdown signal received; draining → (+25s) Starting gRPC drain → Drain completed within deadline → Shutdown complete. #### Which issue(s) this PR is related to: Fixes agent-substrate#719 Required for System Upgrade flow (agent-substrate#473) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t. (agent-substrate#684) This new release requires multiple files and is released as a tarball. gVisor asset handling code is updated to automatically handle this release format and extract it as necessary. This build also contains the start of a series of upcoming startup/memory performance optimizations to make Substrate sandbox churn efficient. Benchmarks between the previous gVisor build and this one: ``` CI benchmark suite: │ old │ new │ │ sec/op │ sec/op vs base │ ResumeActor/glutton_baseline_1_user/p50 324.7m ± 3% 289.6m ± 2% -10.79% (p=0.000 n=11) ResumeActor/glutton_baseline_1_user/p95 373.4m ± 9% 328.6m ± 2% -12.00% (p=0.000 n=11) ResumeActor/glutton_baseline_5_users/p50 319.4m ± 1% 288.6m ± 1% -9.65% (p=0.000 n=9) ResumeActor/glutton_baseline_5_users/p95 367.6m ± 5% 341.7m ± 11% -7.05% (p=0.024 n=9) ResumeActor/glutton_baseline_10_users/p50 323.6m ± 1% 293.8m ± 2% -9.22% (p=0.000 n=9+10) ResumeActor/glutton_baseline_10_users/p95 396.2m ± 4% 372.1m ± 5% -6.07% (p=0.001 n=9+10) ResumeActor/glutton_oversubscribe_15_users/p50 330.8m ± 2% 304.7m ± 1% -7.89% (p=0.000 n=8+10) ResumeActor/glutton_oversubscribe_15_users/p95 402.7m ± 5% 390.4m ± 3% ~ (p=0.083 n=8+10) ResumeActorColdStart/glutton_baseline_1_user/p50 227.0m ± 10% 211.6m ± 19% -6.79% (p=0.003 n=11) ResumeActorColdStart/glutton_baseline_5_users/p50 231.5m ± 5% 216.3m ± 7% -6.54% (p=0.004 n=9) ResumeActorColdStart/glutton_baseline_5_users/p95 246.4m ± 3% 230.2m ± 6% -6.60% (p=0.000 n=9) ResumeActorColdStart/glutton_baseline_10_users/p50 217.4m ± 10% 193.5m ± 7% -10.99% (p=0.001 n=9+10) ResumeActorColdStart/glutton_baseline_10_users/p95 241.8m ± 7% 231.2m ± 9% -4.39% (p=0.022 n=9+10) ResumeActorColdStart/glutton_oversubscribe_15_users/p50 205.9m ± 6% 192.5m ± 28% ~ (p=0.122 n=8+10) ResumeActorColdStart/glutton_oversubscribe_15_users/p95 247.1m ± 4% 229.6m ± 102% -7.09% (p=0.043 n=8+10) SuspendActor/glutton_baseline_1_user/p50 313.3m ± 7% 294.1m ± 4% -6.14% (p=0.010 n=11) SuspendActor/glutton_baseline_5_users/p50 304.6m ± 3% 289.1m ± 2% -5.10% (p=0.000 n=9) SuspendActor/glutton_baseline_10_users/p50 309.7m ± 3% 296.9m ± 3% -4.13% (p=0.000 n=9+10) SuspendActor/glutton_oversubscribe_15_users/p50 320.6m ± 3% 305.3m ± 2% -4.75% (p=0.001 n=8+10) Manual benchmarks (GKE on a single c3-standard-88 node): - p50 sandbox lifecycle time (from issuing resume to checkpointed): ┌─────────┬──────────┬───────────┬────────┐ │ workers │ old │ new │ delta │ ├─────────┼──────────┼───────────┼────────┤ │ 1 │ 795 ms │ 565 ms │ −28.9% │ ├─────────┼──────────┼───────────┼────────┤ │ 2 │ 801 ms │ 590 ms │ −26.4% │ ├─────────┼──────────┼───────────┼────────┤ │ 8 │ 902 ms │ 759 ms │ −15.9% │ ├─────────┼──────────┼───────────┼────────┤ │ 64 │ 3.25 s │ 3.14 s │ −3.4% │ ├─────────┼──────────┼───────────┼────────┤ │ 88 │ 4.47 s │ 4.30 s │ −3.8% │ └─────────┴──────────┴───────────┴────────┘ - Sandbox starts per second: ┌─────────┬────────────┬────────────┬────────┐ │ workers │ old │ new │ delta │ ├─────────┼────────────┼────────────┼────────┤ │ 1 │ 1.256 ± 1% │ 1.755 ± 2% │ +39.7% │ ├─────────┼────────────┼────────────┼────────┤ │ 2 │ 2.482 ± 1% │ 3.393 ± 1% │ +36.7% │ ├─────────┼────────────┼────────────┼────────┤ │ 8 │ 8.81 ± 2% │ 10.50 ± 2% │ +19.2% │ ├─────────┼────────────┼────────────┼────────┤ │ 64 │ 19.85 ± 3% │ 20.45 ± 1% │ +3.0% │ ├─────────┼────────────┼────────────┼────────┤ │ 88 │ 19.70 ± 1% │ 20.58 ± 2% │ +4.5% │ └─────────┴────────────┴────────────┴────────┘ ```
agent-substrate#765) - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…trate#758) Fixes agent-substrate#732 * `UpdateActorSnapshot` now carries the resource itself + `update_mask` * `scope` is now applied via the update mask * Moved `update_mask` to a separate file, so it can be reused by other update RPCs. * Added `uid` and `version` as optional guards. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
atecontroller had no OTel at all. Dev-mode zap logger, and its controller-runtime metrics were only available on a :8080 that we don't scrape. After this PR, logs go through the shared slog handler (plus a `--log-level` flag to match the other binaries), and controller-runtime's Prometheus registry is bridged onto the OTLP reader so the reconcile/workqueue metrics actually reach the collector. Filtering these out is a pipeline responsibility. Also added otelgrpc to the ateapi client, which was untraced. This unblocks agent-substrate#564 the workperpool metrics, cc @Angelawork, @JeffLuoo: there's a working `MeterProvider` to use for `ate.workerpool.desired_workers`/`ready_workers`. Couple of things to mention for review: -`InitMetricsPushOnly`, not `InitMetrics`, even though we do serve :8080. That port is controller-runtime's own private registry, not the global one `serverboot.metricsMux` serves, so a pull reader there would collect into something we never expose. - Bridge is pinned to v0.68.0 to match otelgrpc. Wanted to go to v0.70.0, but that requires otel/sdk/metric 1.45.0 and pulls the whole SDK up with it (406 vendor files instead of 81). Happy to do that bump separately. - zap/zapr fall out of go.mod since atecontroller was the last importer. - I included an OTel collector image bump from the early 2024 (!) one to latest, which was breaking exposing native histograms - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Fixes agent-substrate#653 by adding `actor_uid` to `Assignment` alongside `actor` - [X] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…strate#787) - Update the gVisor release which has support for multiple durable-dirs. - Modify the tests to run with gVisor (which were disabled before).
## Summary atelet never deletes local pause snapshots, so they accumulate until the node disk fills (agent-substrate#668). The control plane only ever references the latest one (`LocalSnapshotInfo` is single-valued), so older directories are dead weight by design. Checkpoint now prunes every existing snapshot before writing the new one — each is superseded by the snapshot about to be written, and pruning first caps disk usage at a single snapshot. Pruning is best-effort: a failed delete is logged and retried by the next prune, never failing the checkpoint. Snapshot prefixes are now validated as single path segments at the Checkpoint/Restore RPC boundary (shared `ValidateLocalSnapshotPrefix`). Previously only checked non-empty: a nested prefix like `pause/2` would silently write nested directories, and `..` could escape the actor's directory entirely. Fixes agent-substrate#668 ## Test plan - Unit: prune remove-all/missing-dir cases; validator table (nested, traversal, absolute, backslash); Checkpoint + Restore request rejection cases - Live on kind: demo e2e suites pass (snapshot + durable-dir lifecycle: pause→resume→suspend→resume across pause-scope configs); atelet logs confirm each checkpoint prunes the prior snapshot before the write
…oken auth (agent-substrate#780) This PR introduces pluggable path configurations for the benchmark orchestrator and adds support for optional Kubernetes ServiceAccount token-based authentication in the benchmark runner.
…ram (agent-substrate#682) Part of agent-substrate#564 (Part 3) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Description This PR implements Part 3 of agent-substrate#564 by adding telemetry histogram instrumentation for `ate.scheduler.eligible_workers` in `ateapi`. It measures unassigned free worker capacity remaining after all scheduling constraint filters are applied, sampled at every scheduling decision. ## Key Changes: * Updated `Scheduler.Schedule()` to record eligible candidate workers per pool (`recordEligibleWorkers`). Defined `SchedulingConstraintKey` (`ate.scheduling.constraint`) and constraint classification values (`none`, `required_nodes`, `selector`) in `internal/ateattr/ateattr.go`. * Added unit tests in `scheduling_test.go` covering candidate counts, namespaced attributes, zero-capacity fleet states, empty fleets, sandbox class mismatches, draining workers, and constraint classifications. ## Testing * `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...` * `go test -buildvcs=false ./cmd/atenet/internal/router/...` * `make test` ## E2E Test * `./hack/create-kind-cluster.sh` * `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter` * `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...`
This PR implements the last two metrics from agent-substrate#433. Right now, we can see that a resume was slow but not where. `ate.actor.lifecycle.operation.duration` covers the whole ateapi operation, and `atenet.router.route.duration` covers the edge, but everything between ateapi-atelet-actors is one block that contains fetching the manifest, downloading the snapshot, unpack the OCI image, call to ateom. We have `rpc.server.call.duration` that gives us the atelet restore total time, but template, kind, and scope labels are missing, so today we cannot really pinpoint why/where we have a regressions in latency. In this PR I am adding per-phase histograms, here's an example of what we can know after these changes: ```console ateom_restore 522 ms ############################### download 8.9 ms # manifest_fetch 3.2 ms oci_unpack 2.6 ms --------------------------------------------------------- total 535 ms ``` It also fixes a gap agent-substrate#683 opened where a `data_on_golden` resume was labeled identically to a plain one on the lifecycle histogram. Things folks might want to argue with: - total as a phase value. Partly duplicates `rpc.server.call.duration`, but that one has no domain labels and gRPC-specifc. We can drop it, but it's an inferior operational UX, so I'd rather have it here. - Phases overlap, they are not a partition of total, because download runs concurrently with the asset fetch and unpack. I called this out in the metric description. Do not sum across phases. - New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind` value for `data_on_golden`. A new value would collapse local and external into one bucket, which is the biggest latency difference there is. This does add a label to the already shipped lifecycle histogram. Verified on kind, and all e2e suites pass, and the emitted series cover every kind (golden, latest, local) on both metrics with no unknown values. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Closes agent-substrate#706 ## Summary - broker short-lived actor certificates from atelet over a same-node mTLS Unix socket - keep the actor private key in atunnel and renew the certificate before expiry - authenticate egress CONNECT using the actor certificate instead of bearer tokens ## Testing - `make verify` - `go test -race ./internal/atunnel`
CONV=a6947013-0eab-49f2-9960-11d98ee43dbf
…#781) Use a fake clock and synctext to avoid waiting in real-time in the unit test. Addresses review comments left over from agent-substrate#221.
Currently we are unable to easily automate the benchmarking of micro VMs. This PR: * Refactors the microVM setup scripts to allow discrete installation of uVM sanbox configs separate from counter demo * Pipes a MicroVM container test config through the benchmarking system * Updates default atelet daemonset to tolerate all kinds of sandboxClass taints
…nt-substrate#792) `setup-envtest use` with no version re-resolves "latest" against the remote release index on every run, so the envtest-backed packages needed network even with fully cached binaries. This PR pins the control plane to `1.36.x`, and fold the three per-package `envtest` into `internal/testenv.Start`, which also honors `-short`: `go test -short ./...` skips the envtest-backed packages and is the guaranteed-offline path. Have Claude to help me fix it Fixes agent-substrate#789 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Generate a per-pod CDI spec with nvidia-ctk, parse it in-tree, and inject the device nodes (major/minor stat-resolved) of the "all" CDI device, the driver- library mounts, and env into each actor's OCI spec. Run the CDI createContainer hooks from the mounted toolkit except update-ldcache, whose ldconfig needs a private /proc mount; stage the SONAME symlinks it would create from each library's ELF DT_SONAME instead. That keeps the GPU worker on the same unprivileged posture as any other gVisor worker (no user namespace, no procMount:Unmasked, cgroup delegation intact). Enable runsc --nvproxy at sandbox creation. Detect GPUs by device-node glob so any assigned index works (multi-GPU). The distroless ateom cannot exec nvidia-ctk, so GPU pools must run a glibc ateom build (WorkerPool.spec.ateomImage). Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
GPU passthrough is implemented only for the gVisor runtime, but the pod template's resources are applied before the sandbox-class check, so a micro-VM pool requesting nvidia.com/gpu still got the request on its worker pod. The pod then scheduled onto a GPU node and held a device that no actor could use. Reject the combination at apply time with a CEL rule on WorkerPoolSpec, matching the cross-field rules ActorTemplate already carries. The rule keys off limits or requests, mirroring the pod-shaping check, and is written positively so a future sandbox class must opt in rather than silently inherit GPU support. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Kubernetes does not admit a pod that requests an extended resource without a
matching limit ("Limit must be set for non overcommitable resources"), but the
GPU pod shaping keyed off limits or requests. A pool that set only a request
was accepted, got the toolkit mount and driver-root env, and then had its
Deployment's pods refused — surfacing the error on the Deployment rather than
on the WorkerPool the user wrote.
Reject it with a CEL rule alongside the sandbox-class one, so both land on the
same object at apply time. The message names the Kubernetes rule behind it.
The controller still keys off limits or requests: with this rule the
requests-only branch is unreachable, but it should not assume CRD validation
is installed.
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
eliranw
force-pushed
the
eliranw/gvisor-gpu-poc
branch
from
August 7, 2026 19:24
393b4cc to
c8fb3e9
Compare
ateom-gvisor execs the NVIDIA container toolkit binaries it mounts from the host (nvidia-ctk to generate the CDI spec, nvidia-cdi-hook to run the createContainer hooks). Those are glibc-dynamic, so the distroless static default cannot load them and GPU injection fails at exec. Also stop quoting a runsc-version-specific error string in the docs: the message for checkpointing a live CUDA context changed between release-20260622 and release-20260803, though the failure is the same. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
GPU pools already get a glibc ateom through spec.ateomImage, which is what docs/api-guide.md documents and what the hardware testing used. Overriding the default base image would have widened every gVisor worker from distroless static to debian for a case that is opt-in and already served. The docs change from the reverted commit stands: the checkpoint error string for a live CUDA context differs between runsc releases. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
eliranw
pushed a commit
that referenced
this pull request
Aug 19, 2026
…-substrate#1002) Fixes agent-substrate#1001 `TestK8sResolverEndpointSliceUpdates` intermittently fails at `resolver_test.go:215` — 8 of the 30 most recent failed `pr-workflow` runs (~27%): ``` updated state.Addresses = [{Addr: "10.0.0.1:443", ServerName: "", }], want [{Addr: "10.0.0.1:443", ServerName: "", } {Addr: "10.0.0.2:443", ServerName: "", }] ``` `Build` starts a goroutine that calls `updateState` once `WaitForCacheSync` returns. That report is deliberate: a service with no EndpointSlices never fires `AddFunc`, so without it the resolver would stay silent instead of telling gRPC the answer is an empty set. But `WaitForCacheSync` polls at `syncedPollPeriod = 100ms`, so it fires roughly 100ms after Build — and if the test has not yet created the second slice by then, that update still carries only `10.0.0.1` and sits in the channel ahead of the real one. The second `select` took whatever came next, so it asserted against the stale update. Locally Build-to-Create is 0.6ms, well ahead of the timer, which is why this only shows up on loaded runners. The resolver is not at fault — it promises eventual convergence, not that the first update after a change is final, and a duplicate update costs gRPC nothing. So both waits now go through one `waitForAddrs` helper that consumes updates until the set matches, with a timeout so a genuinely broken resolver still fails and reports the last set it saw. Note this changes the first wait as well: it asserted the *first* update equals `[10.0.0.1]`, and now waits for that set instead. The same argument applies there — nothing promises the first update is final. ## Verification `-count=N` proves nothing here: the unfixed test passes locally at any count because the window is never hit. A/B with the Build→Create delay as the only variable: | delay | old assertion | new assertion | |---|---|---| | 0ms | 5/5 pass | 5/5 pass | | 150ms | **0/5 pass** | **5/5 pass** | Measured timeline with the 150ms stall in place: ``` [ 0.5ms] update #1: [10.0.0.1] <- AddFunc for slice1 first select takes it ... 150ms stall ... [101.1ms] update #2: [10.0.0.1] <- the WaitForCacheSync goroutine test creates slice2 [151.4ms] update agent-substrate#3: [10.0.0.1, 10.0.0.2] ``` The probe tests used for this are not included. Rebased over agent-substrate#1013. That fixes a different bug — concurrent `updateState` calls letting an older address set win — and does not close this one: the two updates here are ~100ms apart, so the queue has nothing to coalesce. Re-measured on top of it, unchanged: at a 150ms delay the old assertion is 0/5 and the new one 5/5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds CDI-based NVIDIA GPU passthrough into gVisor (runsc) actor containers, activated only on worker pods that have a GPU. Non-GPU actors are byte-for-byte unchanged.
When a WorkerPool template requests
nvidia.com/gpu:atecontrollermounts the host NVIDIA toolkit (/usr/local/nvidia/toolkit, read-only) into the worker pod.ateom-gvisor, at actor create/restore, detects/dev/nvidia0, enforces the cluster is in CDI mode (fails fast on a legacy/proc/driver/nvidiaovermount), generates a per-pod CDI spec once vianvidia-ctk(underreapLock), and injects it into each actor's OCIconfig.jsonvia the CDI library beforerunsc create.runsc's--nvproxyauto-enables from the injected device nodes.Why these choices
/var/run/cdi): NVML in the worker pod sees only the GPU the device-plugin assigned, so the spec is correctly scoped per assigned GPU on multi-GPU / multi-worker nodes — no "which device is mine" resolution.nvidia-ctkfrom the mounted toolkit (not vendornvcdi): the toolkit mount is needed anyway for the runtimenvidia-cdi-hookbinary, so this avoids the whole NVIDIA-toolkit Go dependency. Only the small CDI library (tags.cncf.io/container-device-interface) is vendored, for the inject step.update-ldcache) working under the privileged worker pod. The legacy runtime's/proc/driver/nvidiaovermount trips the kernel'smount_too_revealing()when runsc runsupdate-ldcachedeprivileged in the gofer; CDI mode never creates that overmount.Guarantees
maybeInjectGPUreturnsnilbefore any side effect when/dev/nvidia0is absent.Non-goals (out of scope, per design)
Non-privileged worker pods; rootless runsc; legacy-mode support; per-actor GPU API; MIG/vGPU/GDS.
Testing
gpuPresent,enforceCDIMode(legacy/missing-toolkit/ok),generateCDISpec(fakenvidia-ctk),injectGPUIntoBundle(fixture CDI + OCI spec),maybeInjectGPUno-op. All green;GOOS=linuxand native builds clean.cdi.WithAutoRefresh(false)).Manual e2e (GPU-gated — run before merge)
On a cluster with the GPU stack in CDI mode (
kubectl patch clusterpolicy cluster-policy --type merge -p '{"spec":{"cdi":{"enabled":true,"default":true}}}') and a WorkerPool whose template requestsnvidia.com/gpu:kubectl exec <worker> -- ls /usr/local/nvidia/toolkit/nvidia-ctk.nvidia-smi.nvidia-smiprints the GPU; the container kernel is*-gvisor(uname -r→ e.g.4.19.0-gvisor).