Follow-up to #852. Fixing the worker pod vs guest sizing relationship needs the guest size to be something the control plane and operators can actually see and reason about. Today it is default_memory and default_vcpus inside the opaque kata-config asset, so it does not show up in kubectl get sandboxconfig, it is not validated at apply time, and nothing can reconcile it against the worker pod limit.
It is also the value that bounds what an actor can be given, so operators have to know it to size a pool correctly, and right now the only way to find it is to fetch the asset and read the TOML.
Two ways to fix this, with different tradeoffs.
Option A: make guest size an explicit field
SandboxConfigSpec is intentionally generic. It has sandboxClass, default, and an assets map, with per-class requirements enforced by a ValidatingAdmissionPolicy rather than by the schema. So this should not be a microvm{} sub-struct. Instead add a generic guestResources field (cpu and memory as a ResourceList) and let the VAP require it for microvm and reject it for gvisor. That matches how assets is already handled: generic schema, class rules in the policy.
The VAP can then validate at apply time: positive values, sane bounds, and rejecting vcpus: -1, which kata accepts to mean "all host CPUs" but ateom does not support.
This keeps guest size a property of the pool's sandbox class, so a given class always produces the same size guest.
Option B: derive guest size from the worker pod resources
Size the guest as worker pod memory limit - VMM overhead, and vCPUs from the pod's cpu limit.
This is appealing because it removes the whole class of problem in #852 by construction rather than by adding a check. The guest cannot exceed the pod, because it is defined as the pod minus overhead. It also stops stranding capacity, since today an actor on an 8Gi worker still gets a 2Gi guest. And it collapses a confusing three number model (pod resources, guest size, per container limits) down to the one number a Kubernetes user already expects, which is what gVisor effectively does today.
ateom does not need a new API to learn this. The worker already knows its own budget, either through the downward API or by reading its own pod cgroup, so this can be a small change in how the VM config is built.
The cost is snapshot portability. A memory snapshot is taken at a specific guest size and needs a restore target at least that large. If every worker sizes its guest to its own pod limit, snapshots become worker specific, which constrains cross worker resume and means a golden snapshot taken on a large worker cannot restore onto smaller ones. Today's fixed per pool size avoids that.
Getting it to ateom
For Option A, ateom should not read the CRD. ateapi/atelet already resolve the SandboxConfig and hand ateom the fetched asset paths, so the guest size can travel the same way, in the request proto alongside the asset paths. ateom then uses it when building the VM config instead of parsing the kata-config. The asset stays for kernel_params and the binaries.
Related: #852, #587, #212, #793.
Follow-up to #852. Fixing the worker pod vs guest sizing relationship needs the guest size to be something the control plane and operators can actually see and reason about. Today it is
default_memoryanddefault_vcpusinside the opaque kata-config asset, so it does not show up inkubectl get sandboxconfig, it is not validated at apply time, and nothing can reconcile it against the worker pod limit.It is also the value that bounds what an actor can be given, so operators have to know it to size a pool correctly, and right now the only way to find it is to fetch the asset and read the TOML.
Two ways to fix this, with different tradeoffs.
Option A: make guest size an explicit field
SandboxConfigSpecis intentionally generic. It hassandboxClass,default, and anassetsmap, with per-class requirements enforced by a ValidatingAdmissionPolicy rather than by the schema. So this should not be amicrovm{}sub-struct. Instead add a genericguestResourcesfield (cpu and memory as aResourceList) and let the VAP require it formicrovmand reject it forgvisor. That matches howassetsis already handled: generic schema, class rules in the policy.The VAP can then validate at apply time: positive values, sane bounds, and rejecting
vcpus: -1, which kata accepts to mean "all host CPUs" but ateom does not support.This keeps guest size a property of the pool's sandbox class, so a given class always produces the same size guest.
Option B: derive guest size from the worker pod resources
Size the guest as
worker pod memory limit - VMM overhead, and vCPUs from the pod's cpu limit.This is appealing because it removes the whole class of problem in #852 by construction rather than by adding a check. The guest cannot exceed the pod, because it is defined as the pod minus overhead. It also stops stranding capacity, since today an actor on an 8Gi worker still gets a 2Gi guest. And it collapses a confusing three number model (pod resources, guest size, per container limits) down to the one number a Kubernetes user already expects, which is what gVisor effectively does today.
ateom does not need a new API to learn this. The worker already knows its own budget, either through the downward API or by reading its own pod cgroup, so this can be a small change in how the VM config is built.
The cost is snapshot portability. A memory snapshot is taken at a specific guest size and needs a restore target at least that large. If every worker sizes its guest to its own pod limit, snapshots become worker specific, which constrains cross worker resume and means a golden snapshot taken on a large worker cannot restore onto smaller ones. Today's fixed per pool size avoids that.
Getting it to ateom
For Option A, ateom should not read the CRD. ateapi/atelet already resolve the SandboxConfig and hand ateom the fetched asset paths, so the guest size can travel the same way, in the request proto alongside the asset paths. ateom then uses it when building the VM config instead of parsing the kata-config. The asset stays for
kernel_paramsand the binaries.Related: #852, #587, #212, #793.