Per-container CPU and memory limits for micro-VM actors - #2
Open
eliranw wants to merge 111 commits into
Open
Conversation
eliranw
force-pushed
the
eliranw/microvm-container-resources
branch
from
August 11, 2026 16:36
d68e176 to
2f2f427
Compare
Commit da8414b made unmarshalSandboxRecord reject records without a pauseImage, fixing the test failure found in ``` --- FAIL: TestUploadLocalCheckpointDir (0.01s) --- FAIL: TestUploadLocalCheckpointDir/matching_scope_uploads_all_files (0.00s) --- FAIL: TestUploadLocalCheckpointDir/microvm_full_capture_uploads_durable_tar_alone_as_data (0.00s) --- FAIL: TestUploadLocalCheckpointDir/gvisor_full_capture_cannot_become_data_yet (0.00s) --- FAIL: TestUploadLocalCheckpointDir/microvm_full_capture_without_durable_tar_has_no_data (0.00s) --- FAIL: TestUploadLocalCheckpointDir/unknown_sandbox_class_cannot_convert (0.00s) --- FAIL: TestUploadLocalCheckpointDir/data_capture_cannot_become_full (0.00s) --- FAIL: TestUploadLocalCheckpointDir/manifest_without_scope_is_rejected (0.00s) --- PASS: TestUploadLocalCheckpointDir/gone_locally_but_already_uploaded_succeeds (0.00s) --- PASS: TestUploadLocalCheckpointDir/gone_locally_and_remotely_crashes_the_actor (0.00s) --- FAIL: TestUploadLocalCheckpointDir/upload_failure_is_a_plain_retryable_error (0.00s) ```
spec.pauseImage has moved and no longer exists in the ActorTemplate
…gent-substrate#804) stage-to-rustfs.sh required developers to install the `aws` CLI, and reached rustfs through a `kubectl port-forward`. The port-forward is a footgun: when it targets the wrong cluster, or runs before the control plane exists, the uploads fail against a dead localhost:9000 rather than saying rustfs isn't there. Run the S3 client in a throwaway container instead, using the same pinned amazon/aws-cli image as the rustfs-bucket-init Job that creates the bucket, and stream each asset in on stdin. The container joins the kind node's network namespace so rustfs's ClusterIP is routable -- a container merely attached to the `kind` docker network reaches the node's own IP but has no route to the service or pod CIDRs. Dropping the port-forward also fixes the macOS case, where a container cannot reach it at all (the container's localhost is the Docker VM, not the host). Also wait for the rustfs rollout and the bucket-init Job before uploading, so running this too early reports what it's waiting on instead of timing out on a connection. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR Surfaced in agent-substrate#743
…ers gauges (agent-substrate#826) Part of agent-substrate#564 (Part 4, Fixes agent-substrate#564) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Description This PR implements Part 4 of agent-substrate#564 by adding OpenTelemetry gauge instrumentation for `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` in `atecontroller`. ## Key Changes: * Added `ReadyReplicas int32` to `WorkerPoolStatus` with `+kubebuilder:printcolumn:name="Ready"` annotation in `pkg/api/v1alpha1/workerpool_types.go` and regenerated the CRD manifest. * Updated `WorkerPoolReconciler.syncStatus` to synchronize `dep.Status.ReadyReplicas` into `wp.Status.ReadyReplicas`. * Implemented `InitMetrics(meter)` registering `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers` as OpenTelemetry Observable UpDownCounters (`{worker}`) with an asynchronous observer callback that samples controller-runtime's local informer cache (`r.Client.List`). * Labeled metric datapoints using centralized attributes `ateattr.WorkerPoolNamespaceKey` and `ateattr.WorkerPoolNameKey`. ## Testing * `go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...` * `go test -buildvcs=false ./cmd/atenet/internal/router/...` * `make test` ## E2E Test * `./hack/create-kind-cluster.sh` * `./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter` * `./hack/run-e2e.sh ./internal/e2e/suites/metrics/...` --------- Co-authored-by: Jeff Luo <jeffluoo@google.com>
Adds a `CLASS` column (`gvisor` | `microvm`) and a `--sandbox-class` filter to `kubectl ate get workers` and `kubectl ate top workers`. ## Why Nothing user-facing distinguishes worker sandbox classes today, and `top workers` is where that silence actively misleads: its CPU/MEMORY columns come from metrics-server, i.e. the **host-side worker pod**. A gVisor worker's pod usage roughly tracks its workload, but a micro-VM worker's memory reflects the guest's RAM (demand-paged toward a fixed allocation) rather than the workload — on a mixed-class cluster the table invites comparing numbers that are not comparable. This is the same source-comparability caveat `ateompb.StatsSource` documents for workload-side samples. `ate.sandbox.class` is already one of the sanctioned low-cardinality metric-label keys in `internal/ateattr`, so this surfaces an established dimension rather than inventing one. ## What - `printer.WorkerTopItem` gains `Class` (`json:"class,omitempty"`); both table printers gain a `CLASS` column after `POOL`. - `top workers` populates it from `Worker.GetSandboxClass()` — the field is already on the `ListWorkers` response (`ateapi.proto` field 9), so there is **no server, proto, or RPC change**. - `--sandbox-class` filters both commands, alongside the existing `-n` / `-a` / `-l` filters (shared `filterWorkers`). - Column renders empty against a server that predates `sandbox_class`; pinned by the free-worker and metrics-unavailable test cases. ## Sample output Against a dev cluster running one gVisor pool and one micro-VM pool (nested-virt GKE node pool), with one actor resumed onto a micro-VM worker: ```console $ kubectl ate top workers NAME POOL CLASS STATUS ASSIGNED ACTOR CPU(CORES) MEMORY(bytes) counter-5f74698959-6qnqk counter gvisor FREE <none> 1m 6Mi counter-5f74698959-fcw6j counter gvisor FREE <none> 1m 7Mi counter-5f74698959-hd6px counter gvisor FREE <none> 1m 6Mi counter-5f74698959-ptlrt counter gvisor FREE <none> 1m 5Mi counter-5f74698959-pzrqs counter gvisor FREE <none> 1m 6Mi counter-microvm-7c5f74d879-6mdxs counter-microvm microvm FREE <none> 1m 6Mi counter-microvm-7c5f74d879-z9qds counter-microvm microvm ASSIGNED ate-demo-counter-microvm/counter-microvm/demo/my-counter-1 5m 45Mi $ kubectl ate top workers --sandbox-class=microvm NAME POOL CLASS STATUS ASSIGNED ACTOR CPU(CORES) MEMORY(bytes) counter-microvm-7c5f74d879-6mdxs counter-microvm microvm FREE <none> 1m 6Mi counter-microvm-7c5f74d879-z9qds counter-microvm microvm ASSIGNED ate-demo-counter-microvm/counter-microvm/demo/my-counter-1 4m 45Mi ``` (The FREE micro-VM workers sit at ~6Mi because the VM only exists once a workload boots; the assigned worker's memory grows with the guest's touched pages.) ## Testing - `go test ./cmd/kubectl-ate/...` — table expectations updated for both commands and the printer, including empty-class rendering and `--sandbox-class` filter cases for each command. - Output above is from a live GKE cluster.
Part of agent-substrate#23 Trap SIGTERM in the ateom, and forward SIGTERM to all application containers. Once all application containers exit, ateom will exit itself.
Instead of requiring custom code for each field, let the callers (i.e RPC handlers) declare which fields are mutable and implement a generic `Apply` function that copies fields over, using proto reflection.
Phase 2 of agent-substrate#463, continuing from agent-substrate#735 (the eviction engine): the watermark-driven loop in atelet that turns eviction on, plus the flags that govern it. After this: observability (metrics/spans), the ops-tool swap, and the e2e suite. ## What - A serialized, panic-recovered pass every `--image-cache-gc-period` (default 5m; `0` disables the loop entirely). Each tick: `statfs` the cache volume + sum the pool's recorded sizes, compute a byte target — down to `--image-cache-low-percent` when usage crossed `--image-cache-high-percent`, and/or down to `--image-cache-max-bytes` — and hand it to `Store.EvictUnused`. - **The target is capped at the pool's own size.** The cache is one tenant of a shared volume; uncapped kubelet-style watermark math on a pressured volume asks a tiny cache to fix pressure it didn't cause, evicting everything every tick. - **Gated passes are detected by contract, not inference**: the engine exports `ErrIncompleteEnumeration` (wrapped into every gate return), and `classifyGCPass` — a pure, table-tested function — maps each pass to skipped/shortfall/complete/quiet. A gated pass logs ERROR "pass skipped" and never feeds the shortfall backoff. - **Genuine shortfall warns with backoff** then decays to a periodic reminder: on a volume under foreign pressure, shortfall is the steady state and must not ERROR every tick. - atelet passes `WithActorsDir(ateompath.ActorsDir)` so the root set sees placed actors (the wiring agent-substrate#735's notes required); `ateompath` gains the `ActorsDir` constant. - The first pass runs immediately at boot (a node starting under disk pressure shouldn't wait a full period), and a `CacheSize` failure skips the pass explicitly rather than silently zeroing the target. - `--image-cache-gc-dry-run` computes and logs every decision while deleting nothing — the production soak mechanism. A cache dir outside `BasePath` logs a warning (its watermarks would measure a different volume than actor state). - README: the GC section's "loop lands next" intro replaced with the loop/flag documentation. ## For reviewers: the enabled-by-default question Defaults ship **enabled** (5m / 85% / 80%, mirroring kubelet so operator intuition transfers), with dry-run as the opt-out soak path. The alternative is dry-run-by-default for a release. The target cap removes the known pathological case; happy to flip the default if you'd rather soak first. ## Testing - `imagegc_test.go`: watermark/cap target-math table (including the capped-target cases), flag validation. - Loop + engine together, Kind: full gVisor e2e suite green under aggressive flags (15s period, 1-byte cap); planted orphan ignored by periodic passes and reclaimed at startup; shortfall backoff observed against an impossible cap; no enumeration-gate false positives. - GKE (6 nodes): idle pool evicted under aggressive watermarks, subsequent resumes cold-pulled cleanly with snapshot memory continuity; flags restored to defaults after.
This reverts the ActorTemplate valueFrom.secretKeyRef support added in agent-substrate#20 (issue agent-substrate#15). We don't want actors to have any access to secrets, they will be injected on the egress route instead. Removing this now so we don't need to copy secrets into substrate resources.
…-fast context checks (agent-substrate#863) This PR addresses the cold-start timeout issue reported in agent-substrate#811 by driving down the decompression/extraction cost of gVisor release archives on nodes, adding structured logging, and supporting context cancellation during extraction. * Ran unit tests locally: `go test -v ./cmd/atelet/... -run TestExtractTarArchive` (PASS). * Benchmarked extraction formats (Bzip2: 19.05s baseline vs Gzip: 1.50s vs Uncompressed Tar: 0.25s).
## Summary The comment on Hasher states the implementation is based on FNV-1a, but fnvhash used fnv.New64() which is FNV-1. Switch to fnv.New64a() to match the documented algorithm and get better distribution. ## Why this is safe The hash function is used solely for live work assignment across controller replicas via rendezvous hashing. No hash values are persisted anywhere.
…bstrate#847) ## What Make the micro-VM runtime choose how it restores guest RAM based on the cloud-hypervisor it is actually driving, so we can support v53. ## Why cloud-hypervisor 53.0.0 changed on-demand restore in two ways: the userfaultfd handler now background-prefaults every registered page (`#8150`), and a snapshot is refused while that runs (`#8556`). Together these make `memory_restore_mode=OnDemand` — what we use today — **unusable** on v53: the prefault storm starves the guest and its readiness probe never passes. The actor never comes up. cloud-hypervisor ships minor releases and rarely backports fixes to a patch branch, so pinning v52 means running an unmaintained hypervisor indefinitely. We need to be able to keep up. Restoring eagerly sidesteps both problems: it reads only the snapshot's populated extents and registers no userfaultfd, so nothing prefaults and nothing gates a later snapshot. But eager is the wrong choice on v52, where on-demand costs a tenth of the memory. So this is a per-VMM decision, not a configuration knob. ## How - `vmm.ping` already answers before every restore and its reply carries the version; we were discarding it. `Ping`/`WaitReady` now return it — no extra process, no extra round trip, nothing to cache. - Pick the mode from that version. The affected range is **bounded** (`prefaultingSince`/`prefaultingUntil`) rather than "53 and up forever", so when a release stops prefaulting unconditionally, on-demand's smaller footprint comes back by moving one constant. - An unreadable version restores eagerly and logs it. The two ways to be wrong are not equal: guessing on-demand on an affected version leaves the actor unable to start, while guessing eager only costs memory. - Eager makes the snapshot self-contained, so two things that follow from it: skip the merge (it would copy the whole resident set onto the restore source for nothing), and drop the staged memory image (nothing pages from it or merges against it afterwards). ## Measurements kind, arm64, 2 GiB guest, counter demo. Each config measured identically, no flags set. | | idle memfd | VMM RSS | pause (median) | per-actor disk | |---|---|---|---|---| | v52, before and after this PR | 16 MiB | 20–21 MiB | ~0.26 s | 158 MiB | | v53, before this PR | — | — | — | **actor never becomes ready** | | v53, with this PR | 158 MiB | 162 MiB | ~0.32 s | 158 MiB | On v52 the runtime detects `52.0.0`, selects on-demand, and reproduces the current shipped behaviour exactly — this is a no-op on what we run today. On v53 it detects `53.0.0`, selects eager, and the actor works. Within v53, the two follow-on changes are worth their own line: | | v53 pause (median) | v53 per-actor disk | |---|---|---| | mode selection only | ~0.69 s | 318 MiB | | + skip the merge | ~0.45 s | 318 MiB | | + drop the staged image | ~0.32 s | **158 MiB** | ## Testing - Unit tests for version parsing and selection, including the real `vmm.ping` payloads from both binaries we ship, and a test documenting how to retire the workaround. - e2e on kind against **both** binaries with no flags set: v52 → on-demand, v53 → eager, counter continuity across five pause/resume cycles each. - `go test -race ./cmd/ateom-microvm/...` green; builds for linux and darwin. ---- NOTE: This PR does not upgrade chv to v53+ by default, I'd like to close the remaining gaps first.
## Summary Adds `hack/metrics/ci-failure-analysis.py`, a script that samples recent failed `pr-workflow` CI runs, fetches their job logs, and classifies each failure by root cause rather than treating all CI failures as equivalent. **Failure categories:** - `named_test_fail` — a specific Go test emitted `--- FAIL: TestName` - `no_free_workers` — envtest/integration tests hit worker resource contention - `e2e_timeout` — e2e tests timed out waiting for actor/service responses (503s, connection resets) - `gcs_access` — GCS/S3 bucket access denied when fetching sandbox assets - `license_check` — `hack/verify/licenses.sh` detected uncommitted LICENSES changes - `verify_fail` — other `hack/verify-all.sh` step failed - `unknown` — could not classify from log output **Usage:** ```bash python hack/metrics/ci-failure-analysis.py \ --repo agent-substrate/substrate \ --fetch-limit 500 --sample 40 # Save for later comparison python hack/metrics/ci-failure-analysis.py \ --repo agent-substrate/substrate \ --fetch-limit 500 --sample 40 \ --save ci-breakdown.json ``` **Sample output (40 sampled runs, Aug 2026):** ``` Named Go test failure 50% ████████████████ No free workers 30% ██████████ E2e timeout / 503 5% █ Unclassified 15% █████ Named test failures: 6x TestDurableDirLifecycle 4x TestActorLifecycle 2x TestMultipleDurableDirLifecycle 2x TestSyncer_UpdateWorker_RetryOnVersionConflict 2x TestLoaderConcurrentHandshakes ``` This informed the timeout fixes in agent-substrate#800 and is intended to feed the planned scheduled CI remediation agent. ## Checklist - [x] Issue is linked above (see agent-substrate#799) - [x] Tests pass locally (`go test ./...`) - [x] Documentation updated if behavior changed (script is new, no existing docs affected) --------- Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>
…client (agent-substrate#820) (agent-substrate#876) Fixes agent-substrate#820. ## Problem The claude-code-multiplex demo UI dialed ateapi with a hand-rolled dialer (`dialAteAPI`) using `InsecureSkipVerify` TLS and **no bearer token**. Since ateapi now requires authorization, every `/api/actors` call fails with `missing bearer token`. ## Fix Per @maxsmythe's guidance, drop the demo's bespoke dialer and use the standard authorized client in `internal/ateclient`: - `server.go` now calls `ateclient.NewClient(ctx, "", "", ATEAPI_ADDR, false)` — the same path kubectl-ate uses: mints an ate-client ServiceAccount token and verifies ateapi's serving cert against the live ClusterTrustBundle before attaching the token. - Empty `ATEAPI_ADDR` → auto port-forwards to `svc/api` in `ate-system` (no manual `kubectl port-forward` needed); set `ATEAPI_ADDR` to target an already-forwarded endpoint (still authenticated). - Removed the now-dead `crypto/tls` + raw `grpc`/`credentials` imports and the `defaultAteapiAddr` constant; updated the package usage doc comment. ## Test - `go build ./demos/claude-code-multiplex/ui/` — clean. - `go vet ./demos/claude-code-multiplex/ui/` — clean. Net: `1 file changed, 31 insertions(+), 35 deletions(-)`. Signed-off-by: Alex Bulankou <alexbu@google.com>
Benchmarking needed updates to fix auth bitrot and to enable readint endpointslices. Also streamlined the ability to manually benchmark microvm and removed Python Locust workers by default to simplify self-serve benchmarking.
…gent-substrate#743) ### Why The README Quickstart covers getting a local Substrate cluster running with the default gVisor runtime, but there's no documentation for the micro-VM runtime, where the friction can't be scripted away: it requires `/dev/kvm` (bare metal, nested virtualization, or Lima on macOS), and on Apple Silicon the Lima configuration and asset assembly are non-obvious. New contributors have to reverse-engineer the `hack/` scripts. ### What Adds `docs/dev/microvm-local.md` — "Running the microVM runtime locally" — a focused guide covering only the microVM delta, based on notes from real onboarding runs. General setup is deferred to the README Quickstart (listed as the guide's prerequisite) rather than duplicated: - **Option A: Linux host with KVM** — verifying `/dev/kvm` and CPU virt support, the rootless-Docker caveat for the KVM probe, cluster creation, and the one-shot `run-microvm-demo-kind.sh` bring-up. - **Option B: Apple Silicon macOS via Lima** — nested virtualization with the guest image pinned to Ubuntu 25.10 (until the kernel issue in the default image is fixed), `vzNAT` networking, and assembling the arm64 assets inside the Lima guest (`assemble.sh` requires a Linux host of the target arch). - **Trying it out** — defers to the next steps the demo script prints and links the counter demo's micro-VM variant, instead of duplicating those commands. - **Troubleshooting** — symptom → root cause → fix entries actually hit during onboarding (rootless-Docker KVM probe failures, the `aws` CLI staging requirement until agent-substrate#804 lands, arm64 `virtiofsd` build deps, M1 lacking FEAT_NV2).
…-substrate#831) (agent-substrate#834) Add the `ate.imagecache.requests` counter. **`ate.imagecache.outcome`** (new key in `internal/ateattr`) | Value | Meaning | |---|---| | `hit` | the node holds a complete image record: each layer directory that the record names is present | | `miss` | the lookup must pull | | `error` | the lookup failed; the only outcome that carries `error.type` | | `cancelled`, `timeout` | the caller gave up, so the cache is not at fault | A failed lookup is neither a hit nor a miss, so it gets its own outcome, as `no_free_worker` does on `ate.scheduler.outcome`. `cancelled` and `timeout` are outcomes for the same reason they are on `ate.router.outcome`. The hit ratio is therefore `hit / (hit + miss)`, with failures and abandoned lookups out of the denominator. **`error.type`** — set only on the `error` outcome. | Value | Meaning | |---|---| | `404`, `401`, `429`, ... | the registry rejected the request; its own HTTP status, reported verbatim | | `_OTHER` | the failure carries no status of its own | Fixes agent-substrate#831 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
This is a very similar fix to agent-substrate#829 We're also removing the precondition as the storage layer function arguments and passing it as a wrapper around the closure functions. agent-substrate#763 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…d CRUD methods in redis package (agent-substrate#824) Part of agent-substrate#477 . All ATV fields are immutable. SandboxConifg will be frozen into ATV at creation time. Adds per-resource store methods and the ateredis package for the AT and ATV as global resources.
Switches the port-forwarding implementation in `internal/portforward` from SPDY to WebSockets using `portforward.NewSPDYOverWebsocketDialer`. In Kubernetes 1.31, by default kubectl now uses the WebSocket protocol instead of SPDY for streaming. Using WebSockets also allows `kubectl ate` to work with Connect Gateway.
agent-substrate#892) This was breaking `hack/install-ate.sh --delete-all` with `--delete-demo-autoscaled-workerpool is not supported on GKE` Also removed the custom usage message for --deploy-demo-autoscaled-workerpool. This was duplicate with the default message: Before: ``` Demo: demo-autoscaled-workerpool --deploy-demo-autoscaled-workerpool Deploy demo-autoscaled-workerpool --delete-demo-autoscaled-workerpool Delete demo-autoscaled-workerpool --deploy-demo-autoscaled-workerpool Deploy autoscaled-workerpool demo (HPA + prometheus-adapter + counter workload) ``` After: ``` Demo: demo-autoscaled-workerpool --deploy-demo-autoscaled-workerpool Deploy demo-autoscaled-workerpool --delete-demo-autoscaled-workerpool Delete demo-autoscaled-workerpool ``` - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…oop review follow-ups (agent-substrate#837) Phase 2 of agent-substrate#463, following agent-substrate#735 and agent-substrate#836: `validate-image-cache` drives eviction through `Store.EvictUnused`, deleting its pre-engine prototype. Also carries agent-substrate#836's post-merge review follow-ups. ## Tool - **Prototype evictor deleted** (mtime-sorted tree removal, then dropping every manifest record — no refcounts, no two-phase rename, no restore). The tool now asks the engine to reclaim the shortfall below `--min-free-gb`, so corpus runs exercise production semantics. `--evict-idle` maps to `WithMinAge`. - **`--evict-all`**: one-shot flush, no refs file or auth. Unlike `rm -rf`, running actors' images survive via bundle-spec rooting (`WithActorsDir`). It is not synchronized with a running atelet (the engine's locks are per-process): the pool can't be corrupted, but a layer reused mid-pass can be evicted from under an actor — a not-yet-mounted start fails once and heals; an already-mounted actor can take EIO. Documented, warned at runtime; the intended end state is an atelet-owned flush (RPC or trigger), not a second process in the pool. - **`--evict-idle` floors at 1m when an actors dir exists**: min-age is the only protection that applies across processes, so on a live node it can't be tuned away. Validation hosts (no actors dir) keep full freedom. - Gate-aware errors in both eviction paths (`ErrIncompleteEnumeration` = "did nothing, repair the named path" vs per-item errors = stats still print, exit 1); a 30s cooldown after fruitless passes so workers don't serialize full engine passes when nothing is evictable. ## Carried agent-substrate#836 follow-ups - Negative `--image-cache-gc-period` rejected (was silently disabling the loop); `noteOutcome` extracted with the backoff cadence unit-tested; engine per-pass line demoted to DEBUG (no-target ticks are now silent); single-line sentinel wraps; `filepath.Abs` + table test for the outside-BasePath warning; `runPass`/`Run` covered behind a `gcStore` seam; flag help and README made truthful (`period=0` still runs startup recovery; watermark reads ~5 pts above `df`; the total-size cap still evicts everything evictable under sustained foreign pressure — retention floor named as the extension). ## Testing - Unit: `-race` clean; new tests for backoff cadence, flag validation, path check, `runPass` skip/panic paths, and `Run` (immediate first pass pinned deterministically; ticking covered separately). - Kind: full e2e green; gated-pass arc verified live (one ERROR per tick naming the corrupt record; startup scan gates and recovers; planted orphan reclaimed at restart); `--evict-all` on a live node evicted 4 unrooted images while all rooted images survived. - GKE (6 nodes): zero image-cache log lines fleet-wide at default log level across ticks. - **Corpus sweep** (7.9 GB volume, always-low-water, `--evict-idle=10s` on a host with no actors dir, parallel pulls): 64/64 images validated with two eviction episodes mid-run (62 img / 103 layers / 5.2 GB, then 46 img / 65 layers / 2.7 GB) — continuous eviction under concurrent real pulls, zero races.
…bstrate#832) Third and last of the PRs for Phase 0 of agent-substrate#550, and what completes agent-substrate#594. agent-substrate#739 does the gVisor half; this one is independent of it and does not touch it, so the two can land in either order. Fills in the measurement half of `GetWorkloadStats` for the micro-VM runtime, so it returns real numbers instead of `Unimplemented`. ## Where the numbers come from Inside the guest, not from a host cgroup. The host cgroup here holds cloud-hypervisor, and its memory is the guest RAM allocation it took at boot — near-constant, and near-identical for an idle actor and a saturated one. The numbers that move with the workload are the ones the guest kernel keeps, and the kata-agent's `StatsContainer` is what reads them out. `AgentClient` grows that call; it is safe alongside the stdout/stderr forwarding, since ttrpc multiplexes the one connection, which is already what those goroutines rely on. What gets summed is the actor's overlay **workloads**, one per container. Their carriers are deliberately absent: a carrier is created and never started (see `CreateCarrier`), so it runs no process and its cgroup has nothing in it to add. Summing the containers is what turns per-container guest accounting into the one per-actor figure the proto reports. Summing the peaks is an upper bound on the peak of the sum rather than the figure itself — two containers need not have peaked at the same moment — and for the single-container actors this runtime mostly serves it is exact. Flagging it in case you'd rather report the largest single peak instead; I think the bound is the more useful of two imperfect answers. ## The conversion New `cmd/ateom-microvm/internal/agentstats`. Pure: it takes an already-fetched `CgroupStats` and never talks to a guest, which keeps it testable without a live micro-VM and — unlike the rest of the micro-VM ateom — without the `linux` build tag. It never fails. Every field the agent left out reads as zero, and nil stats (what the agent answers for a container it has no accounting for) is a zero sample rather than a panic on a path polled for the life of every workload. Working set subtracts the guest's reclaimable page cache, saturating at zero, and accepts both the v2 name (`inactive_file`) and the v1 hierarchical one (`total_inactive_file`). CPU time is divided by 1000: the agent reports nanoseconds, matching the runc stats struct its own is modeled on, and the proto wants microseconds. A container the agent cannot report contributes nothing instead of failing the sample, and for the common way that happens zero is the *correct* contribution rather than a fallback — a container that has exited took its guest cgroup with it and consumes nothing from here on. Failing an actor's telemetry because one sidecar is gone would be the wrong answer. The sample fails only when no container could be read at all, which is the guest as a whole not answering rather than one container being gone. ## Status codes A deliberate mirror of the gVisor handler, so the two runtimes answer a poller the same way. **`NOT_FOUND`** — available, or a UID mismatch. Each says the actor is not here, and the caller's worker-to-actor mapping wants re-resolving. **`FAILED_PRECONDITION`** — no guest to ask yet. A poll landing in the boot or the restore (attribution is retained from the moment the ateom accepts the actor), or one landing mid-teardown. A guest that answers nothing at all is this code and not `Internal`: the sandbox going away is a routine state here, and the next `CheckpointWorkload` turns it into the `NOT_FOUND` above. ## Locking The handler takes no lock, and that is a stronger constraint here than on the gVisor side, where the cgroup path is a constant. The agent client lives in `AteomService.running`, which `lock` guards, so reading it from the handler would be a data race whatever the read is for. Hence `AteomService.guestStats`: an atomic holding the agent client and the container ids, published once the containers are up and cleared by `teardownActor` before it closes anything. Clearing it there rather than alongside the attribution is what keeps a poll landing mid-teardown on the "no numbers right now" path instead of surfacing a closed connection as a failed read. `TestGetWorkloadStatsDoesNotTakeLock` pins the whole property: it holds `s.lock` across the call, so a handler that reached for it — or that looked the agent up in `running` — deadlocks. After the read the handler reloads `activeActor` and compares pointer identity, so a checkpoint plus a fresh run completing underneath it is `NOT_FOUND` rather than misattributed. ## Epoch semantics differ from the cgroup source `memory_peak_bytes` and `cpu_usage_usec` accumulate, and the epoch they accumulate over does not begin where the gVisor source's does. This is the source the proto's epoch note (landing in agent-substrate#739) is being careful about. There, a restore ends the epoch: `runsc delete` destroys the sandbox cgroup and its counters with it. Here the counters live in the guest kernel's own memory, so `restoreFullScope` — relaunch cloud-hypervisor with `--restore`, then resume — brings them back with the guest RAM rather than restarting them, and a **FULL** restore continues the epoch across what the caller sees as a gap. **DATA** has no guest to resume and cold-boots, so that scope does restart at zero. **DATA_ON_GOLDEN** is the case that defeats the obvious detection: it resumes the *template's* golden guest, so an actor's first sample can begin at whatever the golden had accumulated when it was snapshotted — an epoch beginning *above* the value last reported, with no decrease anywhere for a caller to notice. Nothing here can hide that. A caller wanting a lifetime figure has to accumulate one itself and treat a scope transition as a boundary. ## Proto Two comment-only edits, on different lines than agent-substrate#739 touches: - `STATS_SOURCE_GUEST_AGENT` now says what it counts and what it cannot see — the guest kernel, the agent, and the host VMM process are overhead outside the workload's own containers. The cgroup source is the other way round, since the sandbox's host process is one process and its runtime's overhead is charged along with the workload's. The two sources are not comparable figures and the enum should say so. - The message doc now explains *why* there is no per-container attribution rather than just stating it. This source could give it; the gVisor source cannot split one at all. A field only one runtime could ever fill would be worse than none. ## A gap worth knowing about A restore whose post-restore agent dial fails answers `FAILED_PRECONDITION` for the rest of that activation. Telemetry rides on the connection log forwarding already keeps open, and that dial is best-effort by design — a failed dial must not fail a restore whose actor is already running. A second dial of its own would not help: whatever kept the agent from answering a 15s retry loop would keep it from answering that one too. ## Testing - `agentstats`: a fifteen-case table over `FromCgroupStats` (v2 guest, v1 `total_inactive_file`, both keys present, reclaimable cache at and above usage, missing `memory.stat`, no peak, sub-microsecond truncation, memory-without-cpu and the converse, nil and empty stats), plus `TestSamplePlus` for the summation including saturation. - `GetWorkloadStats`: happy path, `TestGetWorkloadStatsSumsContainers`, `TestGetWorkloadStatsSkipsUnreadableContainer`, `TestGetWorkloadStatsCountsAnsweredContainer`, a seven-case error table, and the no-lock regression test. - The `cmd/ateom-microvm` tests are `//go:build linux` and were run rather than only compile-checked: `go test ./...` is green for the whole repo inside a `golang:1.26.3` container, and the `GetWorkloadStats` tests pass under `-race`. **Draft, because one thing is measured rather than exercised.** The field mapping is a property of the guest kernel and image rather than of this code, and a live guest answering `StatsContainer` needs a micro-VM worker, which needs a nested-virtualization node — neither the GKE dev cluster (`c3-standard-8` with `advancedMachineFeatures` absent) nor a local Mac provides one today. So it was measured against the pinned assets themselves — the kata-static 4.0.0 kernel and rootfs that `hack/microvm-assets/assemble.sh` stages and ateom fetches: - **Which cgroup version a container gets** is not a question that guest can answer two ways. Its kernel is 6.18.35, built `CONFIG_MEMCG=y` with `CONFIG_MEMCG_V1` unset, so a v1 memory controller cannot be mounted there at all. Its init is systemd 255.4, which reports `default-hierarchy=unified` when run from that rootfs, and `buildVMConfig` passes no hierarchy override on the cmdline. Containers get v2, and `inactive_file` is the spelling that appears. - **Whether a high-water mark is reported** resolves the same way. The agent links cgroups-rs 0.5.1, whose v2 path reads `memory.current` into `usage` and `memory.peak` into `max_usage`, and passes `memory.stat` through verbatim. `memory.peak` has existed since 5.19, so on a 6.18 guest the peak is a real figure rather than the zero this defends against. - **The unit conversion** falls out of the same source: v2 has no `cpuacct` controller, so the agent falls through to `cpu.stat` and multiplies `usage_usec` by 1000 — the nanosecond reading the division here turns back into microseconds. The v1 spellings stay anyway. The guest image is a `SandboxConfig` asset, so a cluster can be pointed at a different one, and a wrong guess there should cost a low working-set figure rather than a failed sample. Happy to take it out of draft as-is if reviewers are content with the assets being measured instead of a guest being run; otherwise it waits on a nested-virt node. Part of agent-substrate#594
…bstrate#640) Adds Postgres as an alternative persistence backend for ateapi as proposed in agent-substrate#731. Currently this is opt-in with flags in ateapi or use the `--store-backend=postgres` option in the install script. Slack discussion: https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1785183769252679 Performance benchmark results for Postgres vs Redis: https://docs.google.com/document/d/12-ko_BFHcBo_nJkx9f4B7zMbiiWKC2saGhMhZG3aQ-s/edit?usp=sharing Benchmarks are not included in this PR, but you can reproduce the results by following the doc above Limitations and questions: - This draft does not yet handle migrations, this might come as a follow up - It currently uses `testcontainers` for testing Postgres, but as a result it adds a lot of dependencies --------- Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>
E2E test WorkerPools (and thus pods) weren't cleaned up until all tests were finished running. This resulted in resource exhaustion and pods not being scheduled. If a test completes successfully delete the namespace immediately. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR
The atelet dial target concatenated the pod IP with ":8085", which net.Dial rejects for IPv6; grpc.NewClient dials lazily, so it surfaced as Unavailable on every snapshot and restore. Use net.JoinHostPort. Also lifts the 8085 literal into internal/atelet.DefaultPort, shared by atelet's --port flag and the dial target.
This is intended to be essentially a dead code cleanup, not a controversial change. There were two ways to deploy the same proxy component; I assume no one is customizing the deployment mechanic, and this follows how other components work (just deployed with YAML). This simplifies the setup so we only need to maintain one path. The default option is retained so there is no behavioral changes. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR Co-authored-by: Eitan Yarmush <eitan.yarmush@solo.io>
Actor logs used `ate.dev/actor_*` while spans and metrics use `ate.*` registry in internal/ateattr. This PR makes ateattr the single source of truth for everything telemetry-related. - Renamed the six actor log labels onto the registry: - `ate.atespace`, `ate.actor.name`, `ate.actor.uid`, `ate.template.namespace`, `ate.template.name`, `ate.actor.container.name` - Logs join traces now. Records set `trace_id`, `span_id` and `trace_flags`, so you can go from Actor restored to the resume that caused it. Our own lines only, not an actor's stdout: one goroutine forwards a whole container stream and can't know which request produced a given line. Per line correlation comes with agent-substrate#853. - Actors can't fake platform labels. They already couldn't overwrite ours, but they could invent new ones like `ate.tenant` that look platform issued downstream. Anything under `ate.` from an actor is now dropped. - Fixed the asymmetry that was actually left: actor supplied label values weren't stringified, and one non string value makes Cloud Logging discard the labels for that whole entry. - Note: the actor_uid bullet in the issue is stale, agent-substrate#841 fixed it earlier. Lifecycle records still set five labels rather than six, on purpose as they're about the actor, so no container produced them. Fixes agent-substrate#886 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: krisztianfekete <git@krisztianfekete.org>
Finishes up the vision from agent-substrate#715 to have atunnel serve CONNECT on the ingress path. This will give us the option to hit actors on other ports besides 80. I haven't wired up atenet router yet because it's nontrivial; we should do that in a second step so we can have a baseline for performance --------- Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Make workers a gobal resource, identified by a unique name. Worker APIs will be implemented in a follow up change. Part of agent-substrate#730 - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR
…nt-substrate#712) ## Summary Adds `docs/integration-repos.md`: where end-to-end integrations live, how their repositories are named, and how the fixes they need flow back into core. The convention in one line — trivial demos stay in the core repo, each non-trivial integration gets one dedicated repo under the `agent-substrate` org, and core gaps get closed by making core configurable with defaults unchanged rather than by patching it downstream. ## Why now We are about to create the first real, end-to-end integrations rather than counter-style demos: a code-execution sandbox, and an always-on agent. Both are large enough to need their own images, dependencies, and release cadence. Whichever repository gets created first will set the precedent for every one after it. This writes the convention down so that precedent is chosen deliberately instead of inherited by accident. ## What it covers - **Where code lives** — the core-repo/dedicated-repo split, the rough test for which side something falls on (API keys, external services, third-party accounts), and why this is a set of peer repos rather than a second org. - **Naming** — capability-named for general capabilities (`code-execution-sandbox`), integration-named for specific third-party products, named for the product rather than the vendor behind it. Plus what to avoid: over-broad names, names that clone a vendor's API or brand, and the redundant `-integration` suffix. - **Third-party names** — allowed descriptively, with a non-affiliation note in the repo README, and brand/policy edge cases cleared before the repo exists. - **Upstreaming** — the part with teeth for this repo. Integration repos that accumulate local patches against core bitrot, and the gap they work around stays invisible to everyone else. So: prefer making core behavior configurable with defaults unchanged. agent-substrate#487 and agent-substrate#465 are linked as illustrations of that pattern — this PR does not depend on either, and branches from `main`. - **Two worked examples** that validate the convention rather than just following it, including the third-party-name edge case. ## Review This was announced at the community meeting and circulated as a shared design doc with a 7-day review window, which has now closed. It synthesizes the `#integrations` thread discussion. Comment history: <https://docs.google.com/document/d/1Tb6u0b1XSvWrNpoyD4jdsQaJ58aAgDtQOM18uxujs-8/edit> This PR is the trimmed version: doc-review scaffolding — status block, reviewer list, self-link — is dropped, and only the durable convention is carried over. ## Left open Two questions are deliberately out of scope, called out in the doc rather than answered. Both are maintainer calls and neither blocks the first repositories: - Governance tiers — whether to distinguish "official" from "community" integrations with different review bars, as Home Assistant and Obsidian do. - Who creates integration repositories and grants per-integration maintainer access. ## Also in this PR - README gets an entry in the docs list, matching every other file in `docs/`. - `CONTRIBUTING.md` gets one sentence pointing there, since "where does my integration go?" is a question a contributor asks before opening a PR. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…substrate#961) Second piece of agent-substrate#896 (Phase 1 of agent-substrate#550): atelet polls every local ateom's `GetActiveWorkloadStats` and turns the samples into template-level metrics — the TSDB half of agent-substrate#174's cardinality split. After this PR, "how much CPU and memory is this template using" is a Cloud Monitoring query. ## The reader A `statsPoller` in atelet, driven by `--actor-stats-poll-interval` (default 1m; `0` disables; nonzero values are clamped to a 50s floor, the worst-case duration of sampling one micro-VM ateom — 25 containers × 2s per guest-agent call). - **Stateless discovery**: each tick lists `ateoms/*` (entry names are worker pod UIDs — the same sockets the lifecycle RPCs dial) and probes each with the parameterless discovery read. No worker-to-actor mapping, no control-plane dependency, nothing to recover after an atelet restart. Attribution comes solely from the identity echoed in each sample, per the RPC's contract. - **One tolerance rule**: any dial or call failure means "not a target this tick" — covering stale directories of deleted workers, ateoms that have not started listening, and teardowns mid-poll. `NO_WORKLOAD` / `NOT_MEASURABLE_YET` are skips by the RPC's own contract. - **Bounded concurrency**: distinct ateoms are probed 8 at a time (one probe per guest, so nothing the interval floor defends against is reintroduced); a node of stuck-but-accepting sockets degrades to ceil(n/8) timeouts instead of n sequential ones. - **WorkerPool enrichment**: one field+label-selected pod LIST per tick maps pod UID → owning pool (`ate.dev/worker-pool`, the label the pool controller stamps); unresolved pods group without pool labels rather than vanish. Chosen over an informer deliberately — negligible apiserver cost at this cadence, no cache-sync ordering, and the resolver sits behind a function seam if that trade ever changes. Needs one new Downward API env (`NODE_NAME`); the pods RBAC already existed. ## The metrics Labels on every series: `ate.template.namespace/name`, `ate.sandbox.class`, `ate.stats.source`, `ate.workerpool.namespace/name` — all bounded sets; actor and atespace identity never reach a metric label (they belong to the events channel, the next PR). - `ate.actor.stats.sampled_actors`, `…memory_current_bytes`, `…memory_working_set_bytes` — **observable gauges** over the latest tick's snapshot: each collection observes exactly the groups that currently exist, so a template whose actors leave a node disappears from the export. (Synchronous gauges would re-export their last value until process exit — stale memory for actors long gone.) - `ate.actor.stats.cpu_usage` — **Float64Counter in seconds** (cAdvisor / OTel `*.cpu.time` convention; the wire stays µs). The raw `cpu_usage_usec` is cumulative per-epoch per actor, so the poller accumulates per-sweep *increases* against per-actor baselines: first sight establishes a baseline and charges nothing (atelet cannot tell a new actor from its own restart — re-charging epochs the previous process counted would spike `rate()`), a decrease is an epoch reset charged from the new value, and baselines are pruned to the actors seen. Bounded imprecision (≤1 interval per actor across restarts; the pre-checkpoint tail) is documented on the instrument; per-actor precision arrives with the lifecycle events. ## Validated live on ate-dev Both source families, simultaneously, with one actor per class: <img width="2256" height="1180" alt="image" src="https://github.com/user-attachments/assets/86ed3390-99b9-47f4-bbd2-a39ff1fd8d45" /> The same counter application reads 5-6× larger through the cgroup source (whole sandbox: sentry heap, netstack, gofers) than through the guest agent (workload containers only) — the concrete case for the `ate.stats.source` label and its group-don't-sum rule. Also exercised live: pool labels resolved on every series, restart-without-spike on the CPU counter across a DaemonSet rollout, and OTLP delivery to the gke-managed-otel collector with zero export errors. ## Out of scope - Per-actor events + lifecycle first/final samples — next PR per agent-substrate#896. - `k8s.node.name` resource attribute (one manifest line, any time). - e2e coverage — the metrics e2e suite is the natural home once the events channel lands. Part of agent-substrate#896, toward agent-substrate#550.
Fixes agent-substrate#744 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ent-substrate#803) Part of agent-substrate#802 (first PR: the actorIdentity data source; does not close the issue). ## What changed and why Adds a systemInfo volume source to ActorTemplate — a read-only volume whose files are generated by atelet on every Run/Restore, analogous to Kubernetes projected volumes. The initial data source, actorIdentity, writes the actor's own name to a configurable relative path: ``` spec: volumes: - name: system-info systemInfo: dataSources: # Part 1 (this PR): own-metadata projection, downwardAPI-style - actorMetadata: items: - field: name # enum: name | atespace | uid path: actor-name - field: atespace path: atespace - field: uid path: actor-uid containers: - name: main image: app@sha256:... volumeMounts: - name: system-info mountPath: /run/ate ``` Because the files are regenerated before the sandbox starts, they carry the resumed actor's own values regardless of what checkpointed state it boots from — the property the old hardcoded /run/ate identity mount provided, now as an explicit, extensible API that future data sources (identity JWTs, certificates — see agent-substrate#802) can slot into. **Behavior change:** the automatic /run/ate/actor-id mount is removed; actors must opt in by declaring the volume (the e2e identity probe in this PR is the reference example). ### Reviewer notes: - Over half the diff is vendored + generated code (cmd/atelet/internal/third_party/atomicwriter/, atelet.pb.go, zz_generated.deepcopy.go, the CRD manifest). The hand-written surface is ~700 lines. - System-info volume roots live under a new ActorPath/system-info/ host dir, deliberately separate from durable-dir/: the micro-VM durable machinery snapshots everything under the durable-dir root, and generated identity files must never be captured into snapshots. - Supports microVM as well as gVisor. - The e2e identity suite exercises the new API end-to-end with unchanged probe binary and assertions. It runs in the kind-cluster CI job (not run locally). ## Checklist - [x] Issue is linked above - [x] Tests pass locally (go test ./...) - [x] Root-gated tests pass if applicable (N/A — no root-gated packages touched) - [x] Documentation updated if behavior changed (docs/api-guide.md: SystemInfo Volumes section with example) --------- Co-authored-by: Taahir Ahmed <taahm@google.com>
…gent-substrate#1061) Snapshot upload is the largest single item in a micro-VM bake or suspend — ~800 ms of a ~1.65 s server-side bake — and for the snapshots we actually upload it is round-trip bound rather than byte bound. The GCS client's resumable upload sends 16 MiB chunks one after another, each paying a round trip. An idle micro-VM golden snapshot is ~24 MiB compressed, so it spans two chunks and pays twice. Setting `ChunkSize` to 64 MiB puts a typical snapshot in one request. 16MiB is reasonable for clients on poor connections sending small files. It's not reasonable for data centers and large files. ### Measurements Uploads through this exact streaming path (non-seekable body into a `storage.Writer`), from a pod on a GKE worker node (c3-standard-4, us-central1-f) using atelet's service account and the snapshot bucket. Three runs at 24 MiB: | chunk size | 24 MiB upload | |---|---| | 16 MiB (client default) | 425 / 438 / 530 ms | | 32 MiB | 331 / 361 / 405 ms | | **64 MiB** | **258 / 313 / 314 ms** | | 128 MiB | 350 / 391 / 487 ms | About 35% off an idle actor's snapshot upload. The client buffers at most `min(object, ChunkSize)`, so a small object still costs only its own bytes. ### What this deliberately does not do Nothing for large snapshots. At 300 MiB a single stream measured 77–107 MiB/s for *every* chunk size from 16 to 128 MiB, because the transfer dominates: | config | 300 MiB | |---|---| | 16 / 32 / 64 / 128 MiB chunks | 82–107 MiB/s | | composite, 2 parts | 150–163 MiB/s | | composite, 4 parts | **233–257 MiB/s** | | composite, 8 parts | 224–231 MiB/s | Beating the single-stream ceiling needs parallel parts plus a compose, which is a format change (each part has to be independently produced and decodable), so it is left as a follow-up. For context on why this is not a compression problem: on the same node `zstd -1` compresses real guest memory at 549 MiB/s on two cores (167 ms of a ~700 ms upload), while the whole pipeline moves ~105 MiB/s. Raising the compression level trades CPU for fewer bytes on a wire that is not the constraint at these sizes. ### Testing - `go build`, `go vet`, `hack/verify/golangci-lint.sh` and `go test ./cmd/atelet/...` pass. - The chunk-size effect is measured against the real GCS backend from a real worker node, as above. I did not roll a patched atelet on the shared cluster, so the end-to-end effect on a live suspend (~700 → ~400 ms) is inferred from the benchmark rather than observed in situ. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed; the rationale and numbers live in the code comment)
Removes the non-functional token/JWT mode for in-cluster ateapi clients. Clients now always use mTLS certificates; related flags, install and benchmark plumbing, tests, and overlays are deleted. Validated with focused Go tests, shellcheck, and Kustomize renders. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
agent-substrate#954 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
An ActorTemplate container can declare limits so it cannot starve or kill its siblings in the same actor. Only limits are expressible: a request is a scheduling hint, and scheduling happens at the pool level, so per-container limits subdivide a budget that is already held. Gated to sandboxClass microvm. gVisor applies cgroup limits at the sandbox level, where one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). The gate lifts per runtime. Only cpu and memory are accepted and each must be greater than zero, so a template cannot declare a limit that is silently discarded downstream. Limits is a bounded named type because a MaxProperties marker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Scalar bytes and millis rather than a ResourceList map, so ate-api-server parses each resource.Quantity once and every consumer downstream compares numbers. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
toAteletResources is the only place a resource.Quantity is parsed. The wiring into the ateletpb.Container is covered by a test, verified by deleting the line and watching it fail: without it, removing the field left every test in the repository green while limits never left ate-api-server. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Carrying the limits as a standard OCI field means a runtime that gains per-container enforcement picks them up with no new plumbing. A container that declares none produces an unchanged spec. The CFS quota is clamped to the kernel's 1ms minimum, as kubelet's MilliCPUToQuota does: tg_set_cfs_bandwidth rejects anything smaller, so a cpu limit under 10m would otherwise produce a spec the guest refuses at container create, with an error naming a cgroup write rather than the template field. Quota is derived from the period constant rather than a second literal, and the test asserts quota/period against the declared milli-cores, so retuning the period cannot silently change every container's share. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Four changes, all on the path from the bundle's OCI spec to the guest cgroup. SpecToAgentPB converted only Devices and CPU.Shares out of Linux.Resources, so a memory limit reached config.json and was dropped on the way to the agent: in the guest memory.max read "max" and a container limited to 64Mi allocated 128MB without being killed. It now carries Memory and CPU quota, and defaults the period when a quota is set, because period is optional in OCI but a plain uint64 on the wire where an unset one is indistinguishable from zero. ensureKataCompatibleSpec applied kata's defaults only when Resources was nil, so the moment atelet set the field the device allowlist and CPU shares would have vanished. mergeKataResources now fills the gaps the defaults cover and leaves everything else alone, so a field added upstream reaches the guest rather than being silently dropped here. checkResourceEnvelope rejects limits the guest can never satisfy, summed across the actor's containers because they share one guest. Errors carry InvalidArgument: ActorTemplateSpec is immutable, so the failure is permanent and must not read as a server fault. CreateCarrier and StartOverlayWorkload are handed the same spec, so the limits were applied to the carrier as well as the workload. The carrier is created and never started, so a limit there bounds nothing, and one small enough to exclude its init fails the create with an error naming neither the limit nor the container. Verified on hardware: with the limit in place a container declaring 64Mi reads 67108864 in its guest cgroup and is OOM-killed allocating 128MB, while a sibling that declares nothing keeps running. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
A non-positive limit means "unlimited" in the OCI spec, so it is not a claim on the guest. Summing it let a negative offset a sibling's real limit and slip the total past the envelope check: 1536Mi + 1024Mi + (-1536Mi) reads as 1024Mi against a 2048Mi guest, so a 2560Mi overrun was accepted. cpuLimitMillis already skipped a non-positive quota; memory now matches. Also repoints the SandboxConfig anchor in api-guide.md, which went stale when agent-substrate#848 renamed the section. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
agent-substrate#679 sizes the sandbox from ActorTemplate.spec.resources and stamps that size onto every container, which overwrites the per-container limits PR agent-substrate#859 writes into the same OCI field. Records the composition model (per-container subdivides a declared actor envelope, no inheritance), the chosen fix (stop stamping user containers on the micro-VM path), and the evidence for keeping over-subscription validation at runtime: an admission-time CEL sum exceeds the cost budget by more than 100x and takes the existing rules down with it. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Six tasks from the rebase onto the merged agent-substrate#679 through to the PR update, each ending in a runnable test. Records that micro-VM tests are linux-only and must run in Docker, since a green run on macOS silently skips them. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
The actor-level size sizes the VM. Stamping it onto every container replaced each container's declared limit with the actor total, so a container that asked for 64Mi got the whole guest and could never be OOM-killed on its own. A container now gets a cgroup limit only when it declares one; an undeclared container is bounded by guest RAM, which is the real ceiling. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
Nothing applies the actor-level size to a container spec any more, so the size threaded from RunWorkload and RestoreWorkload through buildActorContainers into ensureKataCompatibleSpec had no reader. resolveGuestMemMiB still sizes the VM. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
These were working notes for composing per-container limits with actor-level sizing, not documentation the project publishes. The behaviour they describe is documented in docs/api-guide.md, where readers of the API will find it. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
… the ceiling When the actor declares its own size the guest is that limit minus the VMM reserve, so advising a larger SandboxConfig sends the user to a knob that has no effect. The error now names spec.resources.limits.memory and the reserve when the actor declared a limit, and keeps the SandboxConfig advice when it did not. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
The guide said a micro-VM guest is sized by the SandboxConfig and not by the actor, which is backwards now that the guest is sized from the declared actor limit. Also states where over-subscription is caught, so nobody assumes admission rejects it. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
An operator hitting a limits error or reading the sizing docs needs to be routed to the knob that actually governs the value they hit. A CPU-shortfall error named spec.resources.limits.memory, a field that cannot change the vCPU count, and cited a VMM reserve that applies only to memory. CPU shortfalls now name spec.resources.limits.cpu. Since only ateom-gvisor reaches sizing.ApplyToOCISpec, the sizing docs sent anyone tracing a micro-VM container's cgroup limit to a function the micro-VM binary never calls; they now point at spec.containers[].resources. The api-guide preamble made the same claim for both runtimes. The atelet quota floor comment stated a threshold of 100 milli-cores while the code, its test, the CRD doc and the api-guide all say 10. Records that a micro-VM container's rootfs upper is a guest tmpfs charged to its own cgroup, so writing a large file to the rootfs, /tmp or /run OOM-kills a container that would survive the same manifest on Kubernetes. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
A container's writable rootfs is an overlay upper on host disk, so its writes are reclaimable page cache and do not count against the container's memory limit. The guide described the guest-tmpfs upper that preceded it, which would send a reader chasing an OOM that cannot happen. Records on guestEnvelope why guest memory is reduced by the VMM reserve while CPU is not: the VMM's host processes draw on the same worker-pod CPU quota with nothing set aside for them, so a container gets less CPU than it declared, but CPU is compressible where memory is not. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
A non-positive quota means unlimited in the OCI spec, but it was sent as a literal zero, which the guest applies as no CPU at all. A non-nil zero period overwrote the CFS default set for a live quota, the unsatisfiable write the surrounding comment warns about. Both now match cpuLimitMillis, so the envelope check and the conversion read a spec the same way. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
internal/sizing is no longer shared: ateom-microvm sizes the VM and applies per-container limits through the kata-agent, so comments in runsc.go, spec.go and run.go described a topology that no longer exists. The per-container limits section also drops a redundant requests note, folds its bold blocks into the surrounding prose style, and documents the vCPU ceiling. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>
eliranw
force-pushed
the
eliranw/microvm-container-resources
branch
from
August 20, 2026 10:57
9934135 to
c292b22
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-container CPU and memory limits for micro-VM actors. First slice of agent-substrate#752.
An
ActorTemplatecontainer can cap its own cpu and memory so it cannot starve or kill its siblings in the same actor. A container that exceeds its memory limit is OOM-killed on its own; the rest of the actor is unaffected.Why micro-VM only
The two sandbox classes support opposite halves of agent-substrate#752. Micro-VM actors have a real guest kernel, so each container gets its own cgroup and the limits bind. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). Measured on a running actor, the workload container's cgroup reported
memory.current=0while all 20 sandbox processes sat in the pause leaf. A template that setsresourceswithsandboxClass: gvisoris rejected at admission.How a limit travels
ActorTemplate→ CEL validation →ate-api-serverresolves eachresource.Quantityonce →ateletpb→ atelet writes OCIlinux.resources→ateom-microvmmerges kata's defaults and checks the guest envelope →SpecToAgentPB→ kata agent → guest cgroup.The limit is carried as a standard OCI field rather than a substrate-private concept, so a runtime that gains per-container enforcement picks it up without new plumbing.
Relationship to agent-substrate#679
agent-substrate#679 adds actor-level sizing (
ActorTemplate.spec.resources), which sizes the sandbox itself: guest RAM and vCPUs for a micro-VM, or the sentry for gVisor. This PR adds per-container limits (spec.containers[].resources), which subdivide inside that sandbox. Different fields, complementary layers, no schema conflict.The two also explain each other's gVisor stance. Actor-level limits work under gVisor because the sandbox can be capped as a whole. Per-container limits cannot, because one sentry backs every container, which is why this PR rejects them there.
Either can merge first. Three follow-ups for whichever lands second:
checkResourceEnvelopeshould validate against the post-reserve guest size (the declared limit minus Right-size actor sandboxes to declared ActorTemplate resource limits agent-substrate/substrate#679's VMM reserve) rather than the kata config default.api-guide.mdshould end up with one section covering both layers rather than two.Verified on hardware
Same template, run twice, one commit apart on a micro-VM actor:
hog_ovl/memory.maxmax67108864(the declared 64Mi)bystander_ovl/memory.maxmaxmaxThe bug in between:
SpecToAgentPBconverted onlyDevicesandCPU.Sharesout ofLinux.Resources, so a memory limit reached the bundle'sconfig.jsonand was dropped on the way to the agent. Every unit test passed and the on-disk spec was correct while the feature did nothing.Notes for review
ContainerResourcesdeliberately does not reusecorev1.ResourceRequirements, which also carriesrequestsandclaims. There is no scheduler inside an actor to hint at, and for memory a soft request cannot express "must have this much to come back at all". Review on Right-size actor sandboxes to declared ActorTemplate resource limits agent-substrate/substrate#679 raised the same concern from the other direction: it takes a fullResourceRequirementsandrequestspersists into the immutable spec with no effect.Limitsis a bounded named type. AMaxPropertiesmarker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema.10mis raised to10m: the kernel rejects a CFS quota under 1ms.mergeKataResourcesfills the gaps kata's defaults cover rather than allowlisting known fields, so a field added upstream reaches the guest instead of being dropped.Known gaps
STATUS_RUNNINGand nothing records it. Drafted as a comment on Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity agent-substrate/substrate#550, where per-containermemory.eventsis the natural home.ateom-microvmtakes the oldif Resources == nilbranch once atelet starts emitting resources, and loses kata's device allowlist. Needs a capability check or an upgrade-order constraint.ActorTemplateSpecis immutable.