DO NOT MERGE: dual-stack, stack plus the egress DNS family fix - #6
Open
ygao-g wants to merge 82 commits into
Open
DO NOT MERGE: dual-stack, stack plus the egress DNS family fix#6ygao-g wants to merge 82 commits into
ygao-g wants to merge 82 commits into
Conversation
…rate#916) Since we only have ListWorkers so far, this is just a file rename. The rest of the handlers should be added to this file, instead of new handler-specific files, as per agent-substrate#891 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…strate#915) No declaration added, removed, or renamed, import sets unchanged, and every non-header line identical to what it replaced. agent-substrate#891 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ate#913) agent-substrate#891 No declaration added, removed, or renamed, import sets unchanged, and every non-header line is identical to what it replaced. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…g`. (agent-substrate#914) * Remove the ActorSnapshotRef message. * Add a `GetActorSnapshotTag` method for the `ActorSnapshotTag` resource. The request should (as all Get requests) have a single `ObjectRef` tag field. * Change `GetActorSnapshotRequest` to have a single `ObjectRef` snapshot field. * Add a new field to `Actor,` called `source_snapshot`, which the following schema: ``` message ActorSnapshotSource { // If specified, the actor will fork from this snapshot on the first resume. // Immutable. ObjectRef tag = 1; // A reference to the snapshot the tag resolved to at creation time. // Output only. ObjectRef snapshot = 2; // UID of the snapshot the tag resolved to at creation time. // Output only. string snapshot_uid = 3; } ``` See agent-substrate#912 for a more detailed description of the rationale for this change. Fixes agent-substrate#912
…rate#919) AIP-193 - we should return the details of ErrorInfo instead of a rebuilt, following how `InternalServerUnaryInterceptor` already did. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…#928) Verified in local kind cluster. Fix agent-substrate#925 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#927) Unifies Actor, ActorTemplate and ActorTemplateVersion resource refs
…#898) Fixes agent-substrate#897 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
First piece of agent-substrate#896 (Phase 1 of agent-substrate#550): `GetActiveWorkloadStats`, a parameterless sibling to `GetWorkloadStats` for a scraper that enumerates ateoms and holds no worker-to-actor mapping. ## Contract * An **available** ateom answers an empty `stats` list, not an error — an idle worker is a normal thing for a scraper to find. * An **executing** ateom answers the same sample the keyed read would give, wrapped in a one-element list. * `FAILED_PRECONDITION` keeps the meaning it has on `GetWorkloadStats`: executing, but no numbers yet (mid-boot, guest not answering) — skip this sample, take the next one. * Consumers MUST attribute each sample solely from the identity echoed inside it, never from a mapping they hold: with no asserted uid, the response is the only statement of who was measured. This rule is on the RPC's doc comment. ## Implementation Both runtimes share the measurement half of their existing `GetWorkloadStats`, extracted as `sampleSandbox` (gVisor) / `sampleGuest` (micro-VM) — the discovery handler is attribution-load, empty-if-nil, same helper, re-check. No change to what is measured or how. The one deliberate semantic split: a lifecycle transition underneath the lock-free read is `NOT_FOUND` on the keyed read (the caller asserted an actor that is now gone; its mapping wants re-resolving) but `FAILED_PRECONDITION` on the discovery read (there is no requested actor to disown — the numbers just cannot be attributed to any single actor this tick). Naming note: agent-substrate#896's sketch called this `GetCurrentActiveWorkloadStats`; "current" was redundant with "active", so it landed shorter. ## Testing * Both runtimes: available → empty list; executing mid-boot (no sandbox cgroup / no guest target) → `FAILED_PRECONDITION`; and a cross-check pinning that the discovery sample is identical to the keyed sample against the same fixture — one measurement, two addressing modes. * `make verify` clean through the proto checks (`go-generate.sh` confirms the regenerated `ateom.pb.go` / `ateom_grpc.pb.go` are canonical). Part of agent-substrate#896, toward agent-substrate#550.
…substrate#937) agent-substrate#914 got merged in between the time I sent agent-substrate#898 for review and its submission. agent-substrate#898 wasn't rebased, so tests broke.
…#929) AGENTS.md points to README instructions for a gVisor patch that no longer exist. Removes the stale pointer.
added configurable JWT authentication to ateapi.
…ods. (agent-substrate#947) * Align ActorSnapshot/ActorSnapshotTag store methods to other CRUD methods. * Add common struct types for List<Type> methods to carry parameters and returned pages. * Sort methods in store interface to group methods for each resource. * Remove pessimistic locking from ActorSnapshot/ActorSnapshotTag RPC handlers. * Align CreateActorSnapshotTag method to API guideline. It should take just the resource, no extra top level field.
…ate#955) RunContractTests is getting large and resource tests are not always grouped together. So, let's start with splitting them into separate functions. As they grow, we may consider moving them to their own files, like we did for functional tests in agent-substrate#918 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
added TestActorEgressHTTPS e2e test
…gent-substrate#679) Actors previously ran in sandboxes sized to the whole node; there was no way to declare how much CPU/memory a given actor should get. This adds an explicit, immutable sizing knob on the ActorTemplate and plumbs it into the sandbox's OCI spec for both the gVisor and micro-VM runtimes. **API** — `ActorTemplate.spec.resources` (`*corev1.ResourceRequirements`). The `limits` size the sandbox and are baked into the immutable spec; the CRD and generated code are regenerated accordingly. **internal/sizing** (shared by both runtimes) — new `SandboxSize` value (`FromLimits` / `VCPUs` / `ApplyToOCISpec`). `ApplyToOCISpec` writes CPU quota+period and the memory limit onto the OCI spec, and is a no-op when neither dimension is set, so 0 means "unconstrained". `VCPUs` rounds milliCPU up to whole vCPUs. **Plumbing** — ateapi reads the template limits (`actorResourceLimits` → `tmpl.Spec.Resources`) and supplies `CpuMilli`/`MemoryBytes` over the actor RPCs (ateapi → atelet → ateom); ateom applies them — gVisor via the cgroup leaf (`runsc --cpu-num-from-quota` provisions the sentry vCPU count), micro-VM via the guest VmConfig. The two fields are carried on the proto messages. **Scheduling** — worker capacity is taken from the WorkerPool's per-worker limits and advertised on the Worker; the scheduler only places an actor on a worker whose capacity >= the actor's declared limits. A missing worker or actor dimension is treated as unconstrained, so placement is never blocked by absent data. **Docs & demos** — document the model in `api-guide.md`; the counter, sandbox, and micro-VM demos declare actor limits, and their WorkerPool comments now describe the real model (worker limits size the worker pod + advertise scheduling capacity; the sandbox itself is sized by `ActorTemplate.spec.resources`). - [x] Tests pass (unit tests for sizing + scheduling; e2e suite in `internal/e2e/suites/sizing` resumes an actor and asserts, via the probe fixture's `/resources` endpoint, that the running sandbox observes the declared CPU/memory from the inside) - [x] Appropriate changes to documentation are included in the PR
…nt-substrate#809) ateom runs inside the worker pod that hosts the actor, and exported OTLP straight to the collector over the pod's network. This adds a node-local relay: ateom pushes OTLP/gRPC over a unix socket that atelet serves and forwards to the collector, so a worker pod needs no network path of its own to export spans and metrics. Four things motivate it: - Blast radius. The pod runs untrusted agent code, so allowing it egress to the collector makes the collector reachable to anything that escapes the sandbox. A unix socket cannot leave the node. - Connection count. Worker pods are heavily oversubscribed; N ateoms per node each held their own collector connection. They collapse into atelet's single per-node one. - Interference. ateom transparently redirects actor egress to its own atunnel listener, and its own outbound traffic has to stay clear of the rules it installs. A unix socket is not IP traffic. - Shutdown loss. Teardown frees the actor's network and then the pod goes away, which is when the spans describing teardown are still queued in the batch processor. atelet outlives the worker pod. The relay forwards the OTLP request verbatim rather than decoding and re-exporting, so each ateom's own resource (service.name, service.instance.id) survives instead of being absorbed into atelet's. It is best-effort: an ateom that finds no socket at startup logs it and exports directly to OTEL_EXPORTER_OTLP_ENDPOINT as before, so this is a no-op for a cluster running an older atelet. atelet likewise declines to serve a relay when no collector is configured, since it would accept spans only to drop them. Both halves stay off with --otlp-relay-socket="". The socket lives in ateompath.BasePath, the hostPath already mounted at the same path into atelet and into every ateom pod, so no new volume or controller change is needed. Also includes: - End-to-end tests covering the full serverboot-to-collector path. - Observability documentation updates for Jaeger tracing. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#921) `MergeSparseOverlay` ended with a rename onto an existing `outFile`. Renaming over an existing file makes [ext4](https://github.com/torvalds/linux/blob/v6.17/Documentation/admin-guide/ext4.rst#L317-L328) (`data=ordered`) write back the renamed file's dirty pages, and the scratch file carries the whole merged memory image, so the merge pays a full flush of it. `MergeDeltaIntoBase` (see [link](https://github.com/agent-substrate/substrate/blob/a4121a4ca438603481f1799c85928c78a4ba9a6a/cmd/ateom-microvm/internal/ch/merge.go#L150-L158)) already unlinks first for this reason. On the counter demo the merge goes from ~1140 ms to ~115 ms. This is a no-op today, `MergeSparseOverlay` only runs when base and delta land on different filesystems. It becomes the common path in the follow-up that routes hardlinked restores through it, and landing first keeps that change from carrying a 1.1 s suspend regression.
added CONNECT support for atenet ingress to support arbitrary actor ports
After agent-substrate#914 this is not needed any longer.
The step went in preemptively alongside the micro-VM e2e (agent-substrate#327), not in response to a run running out of space. Deleting `/usr/share/dotnet`, `/usr/local/lib/android`, `/opt/ghc` and the CodeQL toolcache took 24-88s per matrix leg across the last few main runs, which is pure wall-clock on every PR. Easy to put back if a run does hit ENOSPC.
EnableIPv4Forwarding now also writes /proc/sys/net/ipv6/conf/all/forwarding so actor IPv6 traffic (including DNS queries) is routed between the actor veth and pod eth0 instead of being dropped by ip6_forward() on dual-stack / IPv6-only clusters. Factor the sysctl write into writeSysctlIfUnset preserving the original read-only remount/restore behavior, and add unit coverage for its fast paths. Fixes: agent-substrate#945
IPv6 sysctls are absent on kernels with IPv6 disabled (e.g. some containers set net.ipv6.conf.* only when IPv6 is enabled). Treat a missing path as 'nothing to enable' instead of forcing a remount and failing, matching the documented behavior.
…-substrate#953) ate.actor.crashes, ate.actor.lifecycle.operation.duration and ate.scheduler.assignment.duration named a WorkerPool by name alone. A WorkerPool is namespaced, so same-named pools in different namespaces merged into one series, and the three could not join the instruments that already carry both keys. ateattr.WorkerPoolAttributes now builds the pair, and omits both keys when no pool is assigned, so a crash before the actor reached a worker no longer reports an empty-string pool. This changes the series identity of the three instruments. Fixes agent-substrate#951 > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#771) Fixes agent-substrate#608 `crashActor` collected the `releaseWorker` error but still marked the actor `CRASHED` and cleared its `WorkerAssignment`, stranding the still-assigned worker with no actor left to reclaim it. On a release failure, return instead so the actor stays intact and the workflow's retry reclaims the worker (`releaseWorker` is idempotent). - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
The egress gateway's Envoy config lives as one inline string in the atenet-egress ConfigMap, which Kustomize can replace but cannot patch into, so the sdsmint variant is a whole second manifest rather than an overlay. The flag picks between the two. deploy_ate_system and deploy_atenet apply whichever manifest the flag selects. delete_atenet deletes both, because teardown has to clean up an install made under the other setting and either file may declare resources the other does not. atenet-egress-with-sdsmint.yaml starts as a copy of atenet-egress.yaml, so the flag is a no-op until the two diverge.
…ate#772) Fixes agent-substrate#609 The startup stored-worker scan (`enqueueStoredWorkers`) returned on the first `ListWorkers` error, skipping orphan cleanup until the next restart. The `agent-substrate#674` workqueue only retries per-key reconciles, not this initial scan, so retry the scan with capped `wait.Backoff` until it succeeds or the context is cancelled. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
IP_FAMILY selects ipv4, ipv6 or dual and becomes networking.ipFamily, leaving kind's per-family subnet defaults alone. The script also recreates a pre-IPv6 "kind" Docker network, fails fast if the daemon has IPv6 off, sets proxy_ndp alongside proxy_arp for gVisor pod-to-pod traffic, and repoints an ipv6 kubeconfig from [::1] at localhost so a client outside the Docker host can still reach the apiserver. Tested on kind with all three families: node InternalIPs, Service ClusterIPs and pod IPs land in the requested families, pod-to-pod and CoreDNS work on them, and pods still pull through the local registry.
…nce of DurDir (agent-substrate#907) # Benchmark DurDir Fixes agent-substrate#673 Adds a `DurDirUser` load-generation workload that exercises the DurableDir suspend/resume loop end to end, verifies every served byte against a SHA-256 digest, and emits separable latency percentiles for each step. ## The loop per VU, first iteration: create -> resume -> WriteDisk -> ReadDisk+verify steady state: suspend -> resume -> ReadDisk+verify (cold) -> ReadDisk+verify (warm) -> WriteDisk (overwrite, TRUNCATE) Under `onCommit: Data` the container cold-boots from the OCI image and process memory is discarded, so a matching digest after resume can only have come from the restored DurableDir. That is the durability assertion. ## Results All six scenarios ran on GKE/gvisor at 1 VU for 1m each: `Data` vs `Full` snapshot scope, explicit vs implicit resume, and a 5/10/64 MiB size sweep. **Zero failures across every run**, so every served byte matched its digest in all six. **Snapshot growth over repeated overwrites:** `SuspendActor` latency stayed flat across consecutive overwrite-and-suspend cycles on the same DurableDir volume. The loop overwrites with `WRITE_MODE_TRUNCATE`, so the file is exactly X bytes after every write and the captured directory contents are the same size every cycle. Nothing accumulates across suspends. That is the growth question the issue asks about. Latency percentiles per step are in the run artifacts. ## Change surface - **glutton:** `WriteDisk` returns size + sha256; new `ReadDisk` with a `READ_MODE_DIGEST_ONLY` mode for measuring restore cost without paying wire transfer; disk RPCs exposed over HTTP mode. - **manifests:** two new ActorTemplates, `glutton-durdir-{data,full}`, with a `durableDir` volume and `onPause: Full` / `onCommit: {Data,Full}`. - **boomer:** shared actor-lifecycle plumbing extracted from the ping task, then a `DurDirUser` task on top of it; a general `resume_mode` knob. - **harnesses:** `--workload` selects the task at deploy time; `durdir.py` stub, typed dynconfig flags, six nightly scenarios. The two boomer commits above are incremental extractions made as the second workload landed. The final package layout for `internal/benchmarking/boomer` lands as a follow-up PR. ## Testing - `go build ./...`, `go test -race ./cmd/benchmarking/... ./internal/benchmarking/...` clean, no race warnings. - `hack/verify-all.sh`: all nine checks pass. Python protos regenerated, tree clean. - Both ActorTemplates reach `Ready`; golden snapshots confirmed in the bucket. - **Manual durability proof:** 200 MiB payload written, suspended, resumed, and re-read with matching sha256 across every read. Process memory discarded and container cold-booted in between. - **Scale validation:** DurableDir persistence verified up to 1 GiB with 0 failures. - **Regression gate:** `glutton_baseline_5_users` ran 919 requests with 0 failures and 7 ms ping latency post-rebase. No regression on the existing benchmark. ## Deliberately out of scope - **Image size over time:** `ActorSnapshot` has no size field, so there is nothing for a client to read. The issue permits deferring this. `SuspendActor` latency is the available proxy; `atelet.snapshot.size` is the server-side one. - **boomer package restructure:** landing as a follow-up so this PR's files stay reviewable in place. - **RAM-backed variant:** glutton already has `WriteRAM`; small follow-up.
…#1046) #### benchmarking: right-size actor memory, default 256Mi Benchmark `ActorTemplates` declared no resources, so microvm actors fell through to the 2 GiB kata default guest — and everything the guest kernel caches rides along in the memory snapshot, inflating snapshot size and suspend/resume latency, which the benchmarks then measure. Set `spec.resources.limits.memory` on the `glutton` and `sleep` templates, parameterized as `ACTOR_MEMORY` with a `256Mi` default (the smallest size microvm admits: 128Mi VMM reserve + 128Mi guest floor), and thread it through the deploy chain: - `--actor-memory` on `workloads/deploy.sh` and `deploy_locust.sh` - `--benchmark-actor-memory` on `install-ate.sh` - Optional per-test `actorMemory` field in the automation's `tests.yaml` for future `WriteRAM` stress suites.
The in-memory version counter restarted at 1 on every router boot; if Envoy reconnected still holding an identical version string the snapshot cache saw a match and skipped the push, stranding Envoy on pre-restart config. Versions now carry a per-process epoch (unix seconds plus a random suffix) so no incarnation repeats an earlier one's strings, even across clock jumps. Fixes agent-substrate#617.
Actor logs used `ate.dev/actor_*` while spans and metrics use `ate.*` registry in internal/ateattr. This PR makes ateattr the single source of truth for everything telemetry-related. - Renamed the six actor log labels onto the registry: - `ate.atespace`, `ate.actor.name`, `ate.actor.uid`, `ate.template.namespace`, `ate.template.name`, `ate.actor.container.name` - Logs join traces now. Records set `trace_id`, `span_id` and `trace_flags`, so you can go from Actor restored to the resume that caused it. Our own lines only, not an actor's stdout: one goroutine forwards a whole container stream and can't know which request produced a given line. Per line correlation comes with agent-substrate#853. - Actors can't fake platform labels. They already couldn't overwrite ours, but they could invent new ones like `ate.tenant` that look platform issued downstream. Anything under `ate.` from an actor is now dropped. - Fixed the asymmetry that was actually left: actor supplied label values weren't stringified, and one non string value makes Cloud Logging discard the labels for that whole entry. - Note: the actor_uid bullet in the issue is stale, agent-substrate#841 fixed it earlier. Lifecycle records still set five labels rather than six, on purpose as they're about the actor, so no container produced them. Fixes agent-substrate#886 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: krisztianfekete <git@krisztianfekete.org>
Finishes up the vision from agent-substrate#715 to have atunnel serve CONNECT on the ingress path. This will give us the option to hit actors on other ports besides 80. I haven't wired up atenet router yet because it's nontrivial; we should do that in a second step so we can have a baseline for performance --------- Signed-off-by: Keith Mattix II <keithmattix2@gmail.com>
Make workers a gobal resource, identified by a unique name. Worker APIs will be implemented in a follow up change. Part of agent-substrate#730 - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR
…nt-substrate#712) ## Summary Adds `docs/integration-repos.md`: where end-to-end integrations live, how their repositories are named, and how the fixes they need flow back into core. The convention in one line — trivial demos stay in the core repo, each non-trivial integration gets one dedicated repo under the `agent-substrate` org, and core gaps get closed by making core configurable with defaults unchanged rather than by patching it downstream. ## Why now We are about to create the first real, end-to-end integrations rather than counter-style demos: a code-execution sandbox, and an always-on agent. Both are large enough to need their own images, dependencies, and release cadence. Whichever repository gets created first will set the precedent for every one after it. This writes the convention down so that precedent is chosen deliberately instead of inherited by accident. ## What it covers - **Where code lives** — the core-repo/dedicated-repo split, the rough test for which side something falls on (API keys, external services, third-party accounts), and why this is a set of peer repos rather than a second org. - **Naming** — capability-named for general capabilities (`code-execution-sandbox`), integration-named for specific third-party products, named for the product rather than the vendor behind it. Plus what to avoid: over-broad names, names that clone a vendor's API or brand, and the redundant `-integration` suffix. - **Third-party names** — allowed descriptively, with a non-affiliation note in the repo README, and brand/policy edge cases cleared before the repo exists. - **Upstreaming** — the part with teeth for this repo. Integration repos that accumulate local patches against core bitrot, and the gap they work around stays invisible to everyone else. So: prefer making core behavior configurable with defaults unchanged. agent-substrate#487 and agent-substrate#465 are linked as illustrations of that pattern — this PR does not depend on either, and branches from `main`. - **Two worked examples** that validate the convention rather than just following it, including the third-party-name edge case. ## Review This was announced at the community meeting and circulated as a shared design doc with a 7-day review window, which has now closed. It synthesizes the `#integrations` thread discussion. Comment history: <https://docs.google.com/document/d/1Tb6u0b1XSvWrNpoyD4jdsQaJ58aAgDtQOM18uxujs-8/edit> This PR is the trimmed version: doc-review scaffolding — status block, reviewer list, self-link — is dropped, and only the durable convention is carried over. ## Left open Two questions are deliberately out of scope, called out in the doc rather than answered. Both are maintainer calls and neither blocks the first repositories: - Governance tiers — whether to distinguish "official" from "community" integrations with different review bars, as Home Assistant and Obsidian do. - Who creates integration repositories and grants per-integration maintainer access. ## Also in this PR - README gets an entry in the docs list, matching every other file in `docs/`. - `CONTRIBUTING.md` gets one sentence pointing there, since "where does my integration go?" is a question a contributor asks before opening a PR. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…substrate#961) Second piece of agent-substrate#896 (Phase 1 of agent-substrate#550): atelet polls every local ateom's `GetActiveWorkloadStats` and turns the samples into template-level metrics — the TSDB half of agent-substrate#174's cardinality split. After this PR, "how much CPU and memory is this template using" is a Cloud Monitoring query. ## The reader A `statsPoller` in atelet, driven by `--actor-stats-poll-interval` (default 1m; `0` disables; nonzero values are clamped to a 50s floor, the worst-case duration of sampling one micro-VM ateom — 25 containers × 2s per guest-agent call). - **Stateless discovery**: each tick lists `ateoms/*` (entry names are worker pod UIDs — the same sockets the lifecycle RPCs dial) and probes each with the parameterless discovery read. No worker-to-actor mapping, no control-plane dependency, nothing to recover after an atelet restart. Attribution comes solely from the identity echoed in each sample, per the RPC's contract. - **One tolerance rule**: any dial or call failure means "not a target this tick" — covering stale directories of deleted workers, ateoms that have not started listening, and teardowns mid-poll. `NO_WORKLOAD` / `NOT_MEASURABLE_YET` are skips by the RPC's own contract. - **Bounded concurrency**: distinct ateoms are probed 8 at a time (one probe per guest, so nothing the interval floor defends against is reintroduced); a node of stuck-but-accepting sockets degrades to ceil(n/8) timeouts instead of n sequential ones. - **WorkerPool enrichment**: one field+label-selected pod LIST per tick maps pod UID → owning pool (`ate.dev/worker-pool`, the label the pool controller stamps); unresolved pods group without pool labels rather than vanish. Chosen over an informer deliberately — negligible apiserver cost at this cadence, no cache-sync ordering, and the resolver sits behind a function seam if that trade ever changes. Needs one new Downward API env (`NODE_NAME`); the pods RBAC already existed. ## The metrics Labels on every series: `ate.template.namespace/name`, `ate.sandbox.class`, `ate.stats.source`, `ate.workerpool.namespace/name` — all bounded sets; actor and atespace identity never reach a metric label (they belong to the events channel, the next PR). - `ate.actor.stats.sampled_actors`, `…memory_current_bytes`, `…memory_working_set_bytes` — **observable gauges** over the latest tick's snapshot: each collection observes exactly the groups that currently exist, so a template whose actors leave a node disappears from the export. (Synchronous gauges would re-export their last value until process exit — stale memory for actors long gone.) - `ate.actor.stats.cpu_usage` — **Float64Counter in seconds** (cAdvisor / OTel `*.cpu.time` convention; the wire stays µs). The raw `cpu_usage_usec` is cumulative per-epoch per actor, so the poller accumulates per-sweep *increases* against per-actor baselines: first sight establishes a baseline and charges nothing (atelet cannot tell a new actor from its own restart — re-charging epochs the previous process counted would spike `rate()`), a decrease is an epoch reset charged from the new value, and baselines are pruned to the actors seen. Bounded imprecision (≤1 interval per actor across restarts; the pre-checkpoint tail) is documented on the instrument; per-actor precision arrives with the lifecycle events. ## Validated live on ate-dev Both source families, simultaneously, with one actor per class: <img width="2256" height="1180" alt="image" src="https://github.com/user-attachments/assets/86ed3390-99b9-47f4-bbd2-a39ff1fd8d45" /> The same counter application reads 5-6× larger through the cgroup source (whole sandbox: sentry heap, netstack, gofers) than through the guest agent (workload containers only) — the concrete case for the `ate.stats.source` label and its group-don't-sum rule. Also exercised live: pool labels resolved on every series, restart-without-spike on the CPU counter across a DaemonSet rollout, and OTLP delivery to the gke-managed-otel collector with zero export errors. ## Out of scope - Per-actor events + lifecycle first/final samples — next PR per agent-substrate#896. - `k8s.node.name` resource attribute (one manifest line, any time). - e2e coverage — the metrics e2e suite is the natural home once the events channel lands. Part of agent-substrate#896, toward agent-substrate#550.
Fixes agent-substrate#744 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ent-substrate#803) Part of agent-substrate#802 (first PR: the actorIdentity data source; does not close the issue). ## What changed and why Adds a systemInfo volume source to ActorTemplate — a read-only volume whose files are generated by atelet on every Run/Restore, analogous to Kubernetes projected volumes. The initial data source, actorIdentity, writes the actor's own name to a configurable relative path: ``` spec: volumes: - name: system-info systemInfo: dataSources: # Part 1 (this PR): own-metadata projection, downwardAPI-style - actorMetadata: items: - field: name # enum: name | atespace | uid path: actor-name - field: atespace path: atespace - field: uid path: actor-uid containers: - name: main image: app@sha256:... volumeMounts: - name: system-info mountPath: /run/ate ``` Because the files are regenerated before the sandbox starts, they carry the resumed actor's own values regardless of what checkpointed state it boots from — the property the old hardcoded /run/ate identity mount provided, now as an explicit, extensible API that future data sources (identity JWTs, certificates — see agent-substrate#802) can slot into. **Behavior change:** the automatic /run/ate/actor-id mount is removed; actors must opt in by declaring the volume (the e2e identity probe in this PR is the reference example). ### Reviewer notes: - Over half the diff is vendored + generated code (cmd/atelet/internal/third_party/atomicwriter/, atelet.pb.go, zz_generated.deepcopy.go, the CRD manifest). The hand-written surface is ~700 lines. - System-info volume roots live under a new ActorPath/system-info/ host dir, deliberately separate from durable-dir/: the micro-VM durable machinery snapshots everything under the durable-dir root, and generated identity files must never be captured into snapshots. - Supports microVM as well as gVisor. - The e2e identity suite exercises the new API end-to-end with unchanged probe binary and assertions. It runs in the kind-cluster CI job (not run locally). ## Checklist - [x] Issue is linked above - [x] Tests pass locally (go test ./...) - [x] Root-gated tests pass if applicable (N/A — no root-gated packages touched) - [x] Documentation updated if behavior changed (docs/api-guide.md: SystemInfo Volumes section with example) --------- Co-authored-by: Taahir Ahmed <taahm@google.com>
…gent-substrate#1061) Snapshot upload is the largest single item in a micro-VM bake or suspend — ~800 ms of a ~1.65 s server-side bake — and for the snapshots we actually upload it is round-trip bound rather than byte bound. The GCS client's resumable upload sends 16 MiB chunks one after another, each paying a round trip. An idle micro-VM golden snapshot is ~24 MiB compressed, so it spans two chunks and pays twice. Setting `ChunkSize` to 64 MiB puts a typical snapshot in one request. 16MiB is reasonable for clients on poor connections sending small files. It's not reasonable for data centers and large files. ### Measurements Uploads through this exact streaming path (non-seekable body into a `storage.Writer`), from a pod on a GKE worker node (c3-standard-4, us-central1-f) using atelet's service account and the snapshot bucket. Three runs at 24 MiB: | chunk size | 24 MiB upload | |---|---| | 16 MiB (client default) | 425 / 438 / 530 ms | | 32 MiB | 331 / 361 / 405 ms | | **64 MiB** | **258 / 313 / 314 ms** | | 128 MiB | 350 / 391 / 487 ms | About 35% off an idle actor's snapshot upload. The client buffers at most `min(object, ChunkSize)`, so a small object still costs only its own bytes. ### What this deliberately does not do Nothing for large snapshots. At 300 MiB a single stream measured 77–107 MiB/s for *every* chunk size from 16 to 128 MiB, because the transfer dominates: | config | 300 MiB | |---|---| | 16 / 32 / 64 / 128 MiB chunks | 82–107 MiB/s | | composite, 2 parts | 150–163 MiB/s | | composite, 4 parts | **233–257 MiB/s** | | composite, 8 parts | 224–231 MiB/s | Beating the single-stream ceiling needs parallel parts plus a compose, which is a format change (each part has to be independently produced and decodable), so it is left as a follow-up. For context on why this is not a compression problem: on the same node `zstd -1` compresses real guest memory at 549 MiB/s on two cores (167 ms of a ~700 ms upload), while the whole pipeline moves ~105 MiB/s. Raising the compression level trades CPU for fewer bytes on a wire that is not the constraint at these sizes. ### Testing - `go build`, `go vet`, `hack/verify/golangci-lint.sh` and `go test ./cmd/atelet/...` pass. - The chunk-size effect is measured against the real GCS backend from a real worker node, as above. I did not roll a patched atelet on the shared cluster, so the end-to-end effect on a live suspend (~700 → ~400 ms) is inferred from the benchmark rather than observed in situ. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed; the rationale and numbers live in the code comment)
Removes the non-functional token/JWT mode for in-cluster ateapi clients. Clients now always use mTLS certificates; related flags, install and benchmark plumbing, tests, and overlays are deleted. Validated with focused Go tests, shellcheck, and Kustomize renders. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
agent-substrate#954 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
) Part of agent-substrate#212 > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR ## Summary - add bounded, validated labels and annotations to `WorkerPoolPodTemplate` - propagate that metadata to the generated Deployment and worker pod template - reserve the controller-owned `ate.dev/worker-pool` label - regenerate the WorkerPool CRD and deepcopy code - add API validation, controller tests, and documentation I chose intentionally to avoid exposing a complete `PodTemplateSpec` as per the discussion in the agent-substrate#212.
…substrate#1075) Fixes agent-substrate#957 `ate.actor.lifecycle.operation.duration` carried the pool pair on suspend and pause only when they failed, so per-pool dashboards saw those two operations exclusively as failures. Both workflows record the histogram from a defer that reads the `actor` variable, and the happy path reassigns it to the finalized record. The finalize step commits the new state and the cleared `WorkerAssignment` in one update, so the defer found no assignment and dropped both keys. A failure returns the pre-finalize record, which still names the worker. Both now snapshot `lifecycleOpAttrs(...)` just before the finalize step — the same snapshot-before-clear crash.go does for the crash counter. Paths that end earlier keep the current computation. `delete` stays without a pool: it only runs from SUSPENDED or CRASHED, which already released the worker, so there is none to name. TestLifecycleOpPoolAttributesOnSuccess drives a real suspend and pause through the gRPC service; both subtests fail without the fix. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
… rules Add an fd00:169:254::/126 point-to-point pair alongside the existing IPv4 addresses, with a matching ::/0 default route, and move the actor nftables table from ip to inet so a single table carries both families. Each payload match now guards on NFPROTO to stay off the other family's packets, and teardown lists the inet family too: naming the wrong family there dumps empty, takes the "already clean" path, and silently leaks the table. Assign the IPv6 addresses with IFA_F_NODAD instead of writing the accept_dad sysctl. The ateom container is unprivileged, so containerd mounts /proc/sys read-only and the write failed with EROFS, taking SetupActorNetwork and every actor start down with it on both sandbox classes. A root-gated assertion pins the flag; the existing tests run as real root, where the sysctl is writable and the bug is invisible.
…bled An IPv4-only cluster leaves net.ipv6.conf.all.disable_ipv6=1 in the worker pod netns, which is the default on IPv4-only GKE, and netlink there rejects the veth's IPv6 address with EPERM. The assignment sits on the path of every SetupActorNetwork call site, so actor startup went from working to failing outright and the actor never left ResumeGoldenActor. Gate the IPv6 address and default route on a per-link disable_ipv6 read, leaving the interior IPv4-only on those clusters instead of failing. A root-gated test covers a netns with IPv6 disabled; reverting the gate reproduces the EPERM against it.
LinkIPv6Enabled answers whether the kernel will accept an IPv6 address on a link, not whether the cluster routes IPv6. IPv4-only kind leaves disable_ipv6 at 0, so the actor got fd00:169:254::2 and a ::/0 route it could not use, Go's destination sorting preferred the AAAA of any dual-stack host, and the egress fetch died mid-response -- the IPv4 e2e job has been red since. Pair the capability read with a check that the worker pod's own eth0 carries a global IPv6 address, and the actor stays IPv4-only wherever the pod is. Decide it once in the pod netns and pass it into ConfigureActorVeth. The interior namespace is created fresh, so its own sysctl always said IPv6 was available whatever the pod's families were. Root-gated tests cover a pod without IPv6 and a pod whose new links have IPv6 disabled per link; dropping either half of the check reproduces its failure.
The actor table moved from the ip family to inet so one table can carry both address families, but the teardown only ever listed inet. An nftables table name is unique per family, so an ip table left behind by an earlier ateom is invisible to every later cleanup and keeps redirecting alongside the inet table installed next to it. Teardown now sweeps both.
podHasGlobalIPv6 gates the actor's IPv6 on the worker pod having a global address of its own, and the doc comment reads as though that settles whether the actor can reach anything. It does not: IsGlobalUnicast is true for a ULA, and dual-stack kind hands pods a ULA with no path off the host.
Nothing runs a cluster carrying both address families, so two things go unexercised: code that reads one address where there are two, and IPv4 continuing to work once IPv6 does. Neither single-family lane can see either, because on each of them the one address present is the right one. Opt-in rather than a gate, two ways. A pull request labelled ci/dual-stack gets a run, forks included. A dispatch takes a list of pull requests and merges them onto main first, because the changes that make IPv6 work do nothing apart -- one publishes an AAAA, another gives the router an address to publish -- so only a stack of them shows the feature working. The stack lives for the length of the run, so there is no branch to rebase and no provenance to hand-maintain. Part of agent-substrate#246.
Egress DNS was pinned to V4_ONLY at six sites across two manifests, so an actor on an IPv6-only cluster could not resolve anything: the name came back empty and the connection failed before Envoy ever dialled. ALL rather than AUTO. Despite the name, AUTO is a legacy alias for "V6 preferred" and does not fall back, so a name published with both an A and an unroutable AAAA -- the shape a dual-stack kind cluster produces -- makes exactly one connection attempt, to the AAAA, and 503s when it times out. ALL resolves both families and lets Happy Eyeballs pick the one that works.
The egress dns_lookup_family is set at six sites across two manifests and no Go test reads either file, so a seventh site added without it, or one of the six reverted, would pass CI and only surface as an actor that cannot resolve anything on a single-family cluster. Fails on a V4_ONLY, V6_ONLY or AUTO value anywhere under manifests/ate-install, and on a dns_cache_config that leaves the family unset. Every manifest is scanned rather than the two that carry egress clusters today, so a new variant of atenet-egress.yaml is covered the day it is added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same stack as #4 (agent-substrate#979 + agent-substrate#1057 + agent-substrate#753) plus the two commits that move atenet-egress off dns_lookup_family: V4_ONLY.