ateom-microvm: measure the guest agent for GetWorkloadStats by baizhenyu · Pull Request #832 · agent-substrate/substrate

Tim Bai (baizhenyu) · 2026-08-10T19:19:04Z

Third and last of the PRs for Phase 0 of #550, and what completes #594. #739 does the gVisor half; this one is independent of it and does not touch it, so the two can land in either order.

Fills in the measurement half of GetWorkloadStats for the micro-VM runtime, so it returns real numbers instead of Unimplemented.

Where the numbers come from

Inside the guest, not from a host cgroup.

The host cgroup here holds cloud-hypervisor, and its memory is the guest RAM allocation it took at boot — near-constant, and near-identical for an idle actor and a saturated one. The numbers that move with the workload are the ones the guest kernel keeps, and the kata-agent's StatsContainer is what reads them out. AgentClient grows that call; it is safe alongside the stdout/stderr forwarding, since ttrpc multiplexes the one connection, which is already what those goroutines rely on.

What gets summed is the actor's overlay workloads, one per container. Their carriers are deliberately absent: a carrier is created and never started (see CreateCarrier), so it runs no process and its cgroup has nothing in it to add. Summing the containers is what turns per-container guest accounting into the one per-actor figure the proto reports.

Summing the peaks is an upper bound on the peak of the sum rather than the figure itself — two containers need not have peaked at the same moment — and for the single-container actors this runtime mostly serves it is exact. Flagging it in case you'd rather report the largest single peak instead; I think the bound is the more useful of two imperfect answers.

The conversion

New cmd/ateom-microvm/internal/agentstats. Pure: it takes an already-fetched CgroupStats and never talks to a guest, which keeps it testable without a live micro-VM and — unlike the rest of the micro-VM ateom — without the linux build tag.

It never fails. Every field the agent left out reads as zero, and nil stats (what the agent answers for a container it has no accounting for) is a zero sample rather than a panic on a path polled for the life of every workload.

Working set subtracts the guest's reclaimable page cache, saturating at zero, and accepts both the v2 name (inactive_file) and the v1 hierarchical one (total_inactive_file). CPU time is divided by 1000: the agent reports nanoseconds, matching the runc stats struct its own is modeled on, and the proto wants microseconds.

A container the agent cannot report contributes nothing instead of failing the sample, and for the common way that happens zero is the correct contribution rather than a fallback — a container that has exited took its guest cgroup with it and consumes nothing from here on. Failing an actor's telemetry because one sidecar is gone would be the wrong answer. The sample fails only when no container could be read at all, which is the guest as a whole not answering rather than one container being gone.

Status codes

A deliberate mirror of the gVisor handler, so the two runtimes answer a poller the same way.

NOT_FOUND — available, or a UID mismatch. Each says the actor is not here, and the caller's worker-to-actor mapping wants re-resolving.

FAILED_PRECONDITION — no guest to ask yet. A poll landing in the boot or the restore (attribution is retained from the moment the ateom accepts the actor), or one landing mid-teardown. A guest that answers nothing at all is this code and not Internal: the sandbox going away is a routine state here, and the next CheckpointWorkload turns it into the NOT_FOUND above.

Locking

The handler takes no lock, and that is a stronger constraint here than on the gVisor side, where the cgroup path is a constant. The agent client lives in AteomService.running, which lock guards, so reading it from the handler would be a data race whatever the read is for.

Hence AteomService.guestStats: an atomic holding the agent client and the container ids, published once the containers are up and cleared by teardownActor before it closes anything. Clearing it there rather than alongside the attribution is what keeps a poll landing mid-teardown on the "no numbers right now" path instead of surfacing a closed connection as a failed read.

TestGetWorkloadStatsDoesNotTakeLock pins the whole property: it holds s.lock across the call, so a handler that reached for it — or that looked the agent up in running — deadlocks. After the read the handler reloads activeActor and compares pointer identity, so a checkpoint plus a fresh run completing underneath it is NOT_FOUND rather than misattributed.

Epoch semantics differ from the cgroup source

memory_peak_bytes and cpu_usage_usec accumulate, and the epoch they accumulate over does not begin where the gVisor source's does. This is the source the proto's epoch note (landing in #739) is being careful about.

There, a restore ends the epoch: runsc delete destroys the sandbox cgroup and its counters with it. Here the counters live in the guest kernel's own memory, so restoreFullScope — relaunch cloud-hypervisor with --restore, then resume — brings them back with the guest RAM rather than restarting them, and a FULL restore continues the epoch across what the caller sees as a gap. DATA has no guest to resume and cold-boots, so that scope does restart at zero.

DATA_ON_GOLDEN is the case that defeats the obvious detection: it resumes the template's golden guest, so an actor's first sample can begin at whatever the golden had accumulated when it was snapshotted — an epoch beginning above the value last reported, with no decrease anywhere for a caller to notice. Nothing here can hide that. A caller wanting a lifetime figure has to accumulate one itself and treat a scope transition as a boundary.

Proto

Two comment-only edits, on different lines than #739 touches:

STATS_SOURCE_GUEST_AGENT now says what it counts and what it cannot see — the guest kernel, the agent, and the host VMM process are overhead outside the workload's own containers. The cgroup source is the other way round, since the sandbox's host process is one process and its runtime's overhead is charged along with the workload's. The two sources are not comparable figures and the enum should say so.
The message doc now explains why there is no per-container attribution rather than just stating it. This source could give it; the gVisor source cannot split one at all. A field only one runtime could ever fill would be worse than none.

A gap worth knowing about

A restore whose post-restore agent dial fails answers FAILED_PRECONDITION for the rest of that activation. Telemetry rides on the connection log forwarding already keeps open, and that dial is best-effort by design — a failed dial must not fail a restore whose actor is already running. A second dial of its own would not help: whatever kept the agent from answering a 15s retry loop would keep it from answering that one too.

Testing

agentstats: a fifteen-case table over FromCgroupStats (v2 guest, v1 total_inactive_file, both keys present, reclaimable cache at and above usage, missing memory.stat, no peak, sub-microsecond truncation, memory-without-cpu and the converse, nil and empty stats), plus TestSamplePlus for the summation including saturation.
GetWorkloadStats: happy path, TestGetWorkloadStatsSumsContainers, TestGetWorkloadStatsSkipsUnreadableContainer, TestGetWorkloadStatsCountsAnsweredContainer, a seven-case error table, and the no-lock regression test.
The cmd/ateom-microvm tests are //go:build linux and were run rather than only compile-checked: go test ./... is green for the whole repo inside a golang:1.26.3 container, and the GetWorkloadStats tests pass under -race.

Draft, because one thing is measured rather than exercised. The field mapping is a property of the guest kernel and image rather than of this code, and a live guest answering StatsContainer needs a micro-VM worker, which needs a nested-virtualization node — neither the GKE dev cluster (c3-standard-8 with advancedMachineFeatures absent) nor a local Mac provides one today. So it was measured against the pinned assets themselves — the kata-static 4.0.0 kernel and rootfs that hack/microvm-assets/assemble.sh stages and ateom fetches:

Which cgroup version a container gets is not a question that guest can answer two ways. Its kernel is 6.18.35, built CONFIG_MEMCG=y with CONFIG_MEMCG_V1 unset, so a v1 memory controller cannot be mounted there at all. Its init is systemd 255.4, which reports default-hierarchy=unified when run from that rootfs, and buildVMConfig passes no hierarchy override on the cmdline. Containers get v2, and inactive_file is the spelling that appears.
Whether a high-water mark is reported resolves the same way. The agent links cgroups-rs 0.5.1, whose v2 path reads memory.current into usage and memory.peak into max_usage, and passes memory.stat through verbatim. memory.peak has existed since 5.19, so on a 6.18 guest the peak is a real figure rather than the zero this defends against.
The unit conversion falls out of the same source: v2 has no cpuacct controller, so the agent falls through to cpu.stat and multiplies usage_usec by 1000 — the nanosecond reading the division here turns back into microseconds.

The v1 spellings stay anyway. The guest image is a SandboxConfig asset, so a cluster can be pointed at a different one, and a wrong guess there should cost a low working-set figure rather than a failed sample.

Happy to take it out of draft as-is if reviewers are content with the assets being measured instead of a guest being run; otherwise it waits on a nested-virt node.

Part of #594

Da Huang (git286) · 2026-08-11T19:51:27Z

+		lastErr error
+	)
+	for _, id := range target.workloadIDs {
+		callCtx, cancel := context.WithTimeout(ctx, statsCallTimeout)


Should we have a timeout for the whole loop?

The loop is already bounded end-to-end by ctx — it is the RPC's context, so the caller's deadline caps the sweep, with maxActorContainers × statsCallTimeout (50s) as the ceiling when none is set. Documented on the function, and it now bails early when the caller is gone instead of returning a partial sum as success (e41b274).

Da Huang (git286) · 2026-08-11T20:12:30Z

+			continue
+		}
+		read++
+		total = total.Plus(agentstats.FromCgroupStats(cs))


FYI, by looking at follow code in cmd/ateom-microvm/internal/kata/:

overlay_linux.go:277 — workloads: CgroupsPath = "/ateomchv/" + workloadID

overlay_linux.go:228 — carriers: "/ateomchv/" + cid + "-carrier"

spec.go:52 — default: "/ateomchv/" + id

It seems that every container of the actor are already grouped in the VM (through the request), but we don't have a way to read it from Kata now.

No action now but probably something we can influence upstream if that can give us more accurate data and simplify the implementation (no more client side add up).

Agreed — and this caught the Plus doc overclaiming: it said the guest does not track the actor as a unit, but the shared /ateomchv parent does exactly that under hierarchical accounting; only the agent read path is missing. Rewrote that paragraph as the breadcrumb for the upstream idea (e41b274).

Da Huang (git286) · 2026-08-11T20:45:37Z

+			continue
+		}
+		read++
+		total = total.Plus(agentstats.FromCgroupStats(cs))


In agentclient.go you said "A nil return with a nil error means the agent answered without cgroup stats, which callers should read as "no numbers", not zero."

but here when cs is nil, total will be zero, not "no numbers", right?

Right — per container, nil is "no numbers". The summing layer then decides what that means for the actor, and for the way it actually happens — the container exited and took its cgroup with it — zero is the true contribution, not a fallback (that's the paragraph on sumContainerStats). The guard sits one level up: a guest that answers for no container fails the sample (read == 0) rather than reporting confident zeros. The agentclient comment was prescribing behavior its only caller deliberately doesn't follow — reworded to describe the semantics and point at the summing decision (2c89c97).

Da Huang (git286) · 2026-08-11T20:47:08Z

+
+	// workloadIDs are the guest containers to sum: the overlay WORKLOADS, one
+	// per actor container. Their carriers are deliberately absent — a carrier is
+	// created and never started (see CreateCarrier), so it runs no process and


I don't understand this sentence

Fair — it leaned on the carrier/workload split without explaining it. Each actor container exists in the guest as two kata containers (the overlayWorkloadID comment in run.go is the authoritative explanation): a "carrier" that only makes the agent bind the read-only image rootfs at a fixed guest path, and the overlay workload that runs the actual process. Carriers are created but never started, so nothing ever runs in their cgroups — hence only workload ids are summed. Rewrote the comment to say that in place (2c89c97).

Fills in the measurement half of GetWorkloadStats for the micro-VM runtime, so it returns real numbers instead of Unimplemented. The gVisor runtime's half is an independent change; this one does not touch it. The sample comes from inside the guest, not from a host cgroup. The host cgroup here holds cloud-hypervisor, whose memory is the guest RAM allocation it took at boot: near-constant, and near-identical for an idle actor and a saturated one. The numbers that move with the workload are the ones the guest kernel keeps, and the kata-agent's StatsContainer is what can read them out. AgentClient grows that call; it is safe alongside the stdout/stderr forwarding, since ttrpc multiplexes the one connection, which is already what those goroutines rely on. What gets summed is the actor's overlay WORKLOADS, one per container. Their carriers are deliberately absent: a carrier is created and never started (see CreateCarrier), so it runs no process and its cgroup has nothing in it to add. Summing the containers is what turns per-container guest accounting into the one per-actor figure the proto reports. Summing the peaks is an upper bound on the peak of the sum rather than the figure itself -- two containers need not have peaked at the same moment -- and for the single-container actors this runtime mostly serves it is exact. The conversion lives in a new cmd/ateom-microvm/internal/agentstats. It is pure, taking an already-fetched CgroupStats and never talking to a guest, which keeps it testable without a live micro-VM and, unlike the rest of the micro-VM ateom, without the linux build tag. It never fails: every field the agent left out reads as zero, and nil stats -- what the agent answers for a container it has no accounting for -- is a zero sample rather than a panic on a path polled for the life of every workload. Working set subtracts the guest's reclaimable page cache, saturating at zero, and accepts both the v2 name (inactive_file) and the v1 hierarchical one (total_inactive_file). CPU time is divided by 1000: the agent reports nanoseconds, matching the runc stats struct its own is modeled on, and the proto wants microseconds. A container the agent cannot report contributes nothing instead of failing the sample, and for the common way that happens zero is the correct contribution rather than a fallback: a container that has exited took its guest cgroup with it and consumes nothing from here on. Failing an actor's telemetry because one sidecar is gone would be the wrong answer. The sample fails only when no container could be read at all, which is the guest as a whole not answering rather than one container being gone. Status codes follow the NOT_FOUND / FAILED_PRECONDITION split the proto documents, and the handler is a deliberate mirror of the gVisor one so the two runtimes answer a poller the same way. Available and a UID mismatch are both NOT_FOUND: each says the actor is not here, and the caller's worker-to-actor mapping wants re-resolving. No guest to ask yet is the transient FAILED_PRECONDITION -- a poll landing in the boot or the restore, since attribution is retained from the moment the ateom accepts the actor, or one landing mid-teardown. A guest that answers nothing at all is the same code and not Internal: the sandbox going away is a routine state here, and the next CheckpointWorkload turns it into the NOT_FOUND above. The handler takes no lock. That is a stronger constraint than on the gVisor side, where a cgroup path is a constant: the agent client lives in AteomService.running, which lock guards, so reading it from the handler would be a data race whatever the read is for. Hence AteomService.guestStats, an atomic holding the agent client and the container ids, published once the containers are up and cleared by teardownActor before it closes anything. Clearing there rather than alongside the attribution is what keeps a poll landing mid-teardown on the "no numbers right now" path instead of surfacing a closed connection as a failed read. TestGetWorkloadStatsDoesNotTakeLock pins the whole property: it holds s.lock across the call, so a handler that reached for it -- or that looked the agent up in running -- deadlocks. After the read the handler reloads activeActor and compares pointer identity, so a checkpoint plus a fresh run completing underneath it is NOT_FOUND rather than misattributed. One gap worth knowing about: a restore whose post-restore agent dial fails answers FAILED_PRECONDITION for the rest of that activation. Telemetry rides on the connection log forwarding already keeps open, and that dial is best-effort by design (a failed dial must not fail a restore whose actor is already running). A second dial of its own would not help -- whatever kept the agent from answering a 15s retry loop would keep it from answering that one too. The two accumulating fields, memory_peak_bytes and cpu_usage_usec, are scoped to an epoch that does not begin where the gVisor source's does. That is why the proto's epoch note, which lands with the gVisor change, is written per source rather than as one rule, and this is the source it is being careful about. There a restore ends the epoch: runsc delete destroys the sandbox cgroup and its counters with it. Here the counters live in the guest kernel's own memory, so restoreFullScope -- relaunch cloud-hypervisor with --restore, then resume -- brings them back with the guest RAM rather than restarting them, and a FULL restore continues the epoch across what the caller sees as a gap. DATA has no guest to resume and cold-boots, so that scope does restart at zero. DATA_ON_GOLDEN is the case that defeats the obvious detection: it resumes the template's golden guest, so an actor's first sample can begin at whatever the golden had accumulated when it was snapshotted -- an epoch beginning above the value last reported, with no decrease anywhere for a caller to notice. Nothing here can hide that. A caller wanting a lifetime figure has to accumulate one itself and treat a scope transition as a boundary rather than trusting the numbers to announce one. The field mapping is not covered by a test here, since it is a property of the guest kernel and image rather than of this code. It was instead measured against the pinned assets themselves -- the kata-static 4.0.0 kernel and rootfs that hack/microvm-assets/assemble.sh stages and ateom fetches. Which cgroup version the guest gives a container is not a question the guest can answer two ways. Its kernel is 6.18.35, built CONFIG_MEMCG=y with CONFIG_MEMCG_V1 unset, so a v1 memory controller cannot be mounted there at all; its init is systemd 255.4, which reports default-hierarchy=unified when run from that rootfs, and buildVMConfig passes no hierarchy override on the cmdline. Containers therefore get v2, and inactive_file is the spelling that appears. Whether a high-water mark is reported resolves the same way. The agent links cgroups-rs 0.5.1, whose v2 path reads memory.current into usage and memory.peak into max_usage, and passes memory.stat through verbatim. memory.peak has existed since 5.19, so on a 6.18 guest the peak is a real figure rather than the zero this defends against. The same source settles the unit conversion: v2 has no cpuacct controller, so the agent falls through to cpu.stat and multiplies usage_usec by 1000, which is the nanosecond reading the division here turns back into microseconds. The v1 spellings stay anyway. The guest image is a SandboxConfig asset, so a cluster can be pointed at a different one, and a wrong guess there should cost a low working-set figure rather than a failed sample. Still not covered: a live guest answering StatsContainer. That needs a micro-VM worker, which needs a nested-virtualization node. Part of agent-substrate#594

Cross-reference the summed-peak caveat from the MemoryPeakBytes field, correct the Plus doc (the guest does track the actor as a unit at the shared /ateomchv parent; only the agent read path is missing), and bail out of the container loop when the caller's ctx is already dead instead of returning a partial sum as success.

StatsContainer's doc prescribed reading a nil result as "no numbers, not zero" while its only caller deliberately folds nil into a zero contribution; describe the semantics and point at the summing decision instead. And make the workloadIDs comment self-contained: it leaned on the carrier/workload split without explaining it.

Tim Bai (baizhenyu) force-pushed the stats-microvm-guest-agent branch 3 times, most recently from bf3a015 to df90625 Compare August 11, 2026 14:01

Tim Bai (baizhenyu) marked this pull request as ready for review August 11, 2026 17:45

Tim Bai (baizhenyu) requested review from Jeff Luo (JeffLuoo) and Da Huang (git286) August 11, 2026 17:45

Da Huang (git286) reviewed Aug 11, 2026

View reviewed changes

Comment thread cmd/ateom-microvm/internal/agentstats/agentstats.go

Da Huang (git286) reviewed Aug 11, 2026

View reviewed changes

Tim Bai (baizhenyu) added 3 commits August 12, 2026 10:10

Tim Bai (baizhenyu) force-pushed the stats-microvm-guest-agent branch from 2c89c97 to 38fe7e9 Compare August 12, 2026 14:11

Da Huang (git286) approved these changes Aug 12, 2026

View reviewed changes

Tim Bai (baizhenyu) mentioned this pull request Aug 12, 2026

Actor resource telemetry Phase 1: atelet reader and OTel emission #896

Open

Da Huang (git286) merged commit 3c0f354 into agent-substrate:main Aug 12, 2026
11 checks passed

Tim Bai (baizhenyu) deleted the stats-microvm-guest-agent branch August 12, 2026 18:10

Tim Bai (baizhenyu) added the area/observability label Aug 14, 2026

Navigation Menu

Uh oh!

ateom-microvm: measure the guest agent for GetWorkloadStats - #832

ateom-microvm: measure the guest agent for GetWorkloadStats#832
Da Huang (git286) merged 3 commits into
agent-substrate:mainfrom
baizhenyu:stats-microvm-guest-agent

Tim Bai (baizhenyu) commented Aug 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Da Huang (git286) Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) Aug 11, 2026

Uh oh!

Da Huang (git286) Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) Aug 11, 2026

Uh oh!

Da Huang (git286) Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) Aug 11, 2026

Uh oh!

Da Huang (git286) Aug 11, 2026

Uh oh!

Tim Bai (baizhenyu) Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Tim Bai (baizhenyu) commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Where the numbers come from

The conversion

Status codes

Locking

Epoch semantics differ from the cgroup source

Proto

A gap worth knowing about

Testing

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Tim Bai (baizhenyu) commented Aug 10, 2026 •

edited

Loading