Per-container CPU and memory limits for micro-VM actors

[eliranw]

Per-container CPU and memory limits for micro-VM actors. First slice of #752.

An ActorTemplate container can cap its own cpu and memory so it cannot starve or kill its siblings in the same actor. A container that exceeds its memory limit is OOM-killed on its own; the rest of the actor is unaffected.

sandboxClass: microvm
containers:
  - name: trainer
    resources:
      limits: {memory: 1500Mi}
  - name: sidecar
    resources:
      limits: {memory: 256Mi, cpu: "0.2"}

#752 asks for per-container device selection as well. This PR does the cpu and memory half; devices are the remaining half, and when they land, GPU assignment should follow the same field rather than staying actor-wide as it is today.

Why micro-VM only

The two sandbox classes support opposite halves of #752. Micro-VM actors have a real guest kernel, so each container gets its own cgroup and the limits bind. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty (google/gvisor#190). Measured on a running actor, the workload container's cgroup reported memory.current=0 while all 20 sandbox processes sat in the pause leaf. A template that sets resources with sandboxClass: gvisor is rejected at admission.

How a limit travels

ActorTemplate → CEL validation → ate-api-server resolves each resource.Quantity once → ateletpb → atelet writes OCI linux.resources → ateom-microvm merges kata's defaults and checks the guest envelope → SpecToAgentPB → kata agent → guest cgroup.

The limit is carried as a standard OCI field rather than a substrate-private concept, so a runtime that gains per-container enforcement picks it up without new plumbing.

Composition with #679

#679 sizes the sandbox itself from ActorTemplate.spec.resources: guest RAM and vCPUs for a micro-VM, the sentry for gVisor. This PR subdivides that sandbox. The two are different fields and complementary layers.

They collided in one place. #679 applied the actor-level size to every container's OCI spec, overwriting the per-container limits atelet writes into the same field, so a container asking for 64Mi silently received the whole guest. This PR removes that call on the micro-VM path. A container now gets a cgroup limit only when it declares one; an undeclared container is bounded by guest RAM, which is the real ceiling. On micro-VM the stamp was close to a no-op regardless, since a cap equal to the whole guest can never bind.

gVisor is untouched. There the actor-level size is applied to the pause container, and since one sentry backs every container, that is the only cgroup that binds.

Two consequences worth calling out for reviewers of #679:

• The sizing.SandboxSize threaded into buildActorContainers and ensureKataCompatibleSpec had no remaining reader, so it and the guestSize helper are removed. resolveGuestMemMiB still sizes the VM and still rejects a declared limit too small to boot.
• internal/sizing's package doc claimed both runtimes shared ApplyToOCISpec for container cgroups. That is now gVisor-only, and the comments say so. No code in that file changed.

checkResourceEnvelope runs after resolveGuestMemMiB, so it validates the per-container sum against the post-reserve guest rather than the SandboxConfig default. Its error now names spec.resources.limits.memory (or .cpu) when the actor declared a size, and SandboxConfig when it did not.

internal/e2e/suites/sizing is unaffected: it asserts on num_cpu and mem_total_bytes, both of which come from VM sizing, and only logs the cgroup files.

Verified on hardware

Same template, run twice, one commit apart on a micro-VM actor:

	before	after
hog_ovl/memory.max	max	67108864 (the declared 64Mi)
bystander_ovl/memory.max	max	max
hog allocates 128MB	survived	OOM-killed
bystander	alive	alive

The bug in between: SpecToAgentPB converted only Devices and CPU.Shares out of Linux.Resources, so a memory limit reached the bundle's config.json and was dropped on the way to the agent. Every unit test passed and the on-disk spec was correct while the feature did nothing.

Notes for review

• ContainerResources deliberately does not reuse corev1.ResourceRequirements, which also carries requests and claims. There is no scheduler inside an actor to hint at, and for memory a soft request cannot express "must have this much to come back at all". Right-size actor sandboxes to declared ActorTemplate resource limits #679 reached the same conclusion from the other direction and now rejects spec.resources.requests and .claims at admission.
• Limits is a bounded named type. A MaxProperties marker on the field lands at the wrong schema level for a named map, and without a bound the CEL cost estimator rejects the whole schema. This CRD is close to its schema-wide CEL cost ceiling; a rule that iterates containers and parses quantities exceeds it by more than 100x and takes the existing rules down with it.
• A cpu limit below 10m is raised to 10m: the kernel rejects a CFS quota under 1ms.
• mergeKataResources fills the gaps kata's defaults cover rather than allowlisting known fields, so a field added upstream reaches the guest instead of being dropped.

Known gaps

• An OOM-killed container is not reported above the guest. The actor stays STATUS_RUNNING and nothing records which container died. Raised on Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity #550, where the guest cgroup's memory.events fits the per-container telemetry being designed there.
• A container's writable rootfs is a guest tmpfs, so filesystem writes are charged to its memory limit and a container writing more than its limit to /tmp is OOM-killed. Documented in api-guide.md; separating the two budgets needs an ephemeral-storage field.
• Over-subscription across containers is caught when the actor starts, not at apply time. An admission-time sum check is not implementable within the CEL cost budget.
• The atelet hop that attaches resources to the bundle spec has no test: replacing ctr.GetResources() with nil leaves the whole suite green. Closing it needs an imagecache fixture for prepareOCIDirectory.
• Restored actors inherit the golden's cgroups rather than applying their own spec. Correct today only because ActorTemplateSpec is immutable.
• The GPU section of api-guide.md still says ActorTemplate has no per-container resource fields, which this PR makes false. Correcting it is left to the devices half of Select devices per-container in the ActorTemplate #752, which is what will change the GPU behaviour that paragraph describes.

---

• Tests pass
• Appropriate changes to documentation are included in the PR

[gemini-code-assist]

Code Review

This pull request introduces per-container CPU and memory limits for micro-VM actors, propagating these limits from the ActorTemplate API down to the Kata agent's OCI runtime spec. It includes API schema validations, guest envelope checks, and extensive unit tests. The review feedback highlights a critical bug in the resource envelope validation where negative memory limits could bypass checks and offset other container limits, as well as a broken anchor link in the API documentation.

[Benjamin Elder (BenTheElder)]

Have you seen #679 ?

[eliranw]

Have you seen #679 ?

Yes, this PR overlaps, but that PR sizes the actor and applies limits to every container’s cgroup, while this is per-container. Happy to go second, or perhaps this should be folded into #679 if you’d rather it were one change.

[Benjamin Elder (BenTheElder)]

Yes, this PR overlaps, but that PR sizes the actor and applies limits to every container’s cgroup, while this is per-container. Happy to go second, or perhaps this should be folded into #679 if you’d rather it were one change.

Sorry I just saw this comment, yes let's do this as a follow-up? I will check back next week for rebase, given the timing, apologies.