Skip to content

Resource accounting#51

Merged
sjmiller609 merged 17 commits intomainfrom
resources
Jan 5, 2026
Merged

Resource accounting#51
sjmiller609 merged 17 commits intomainfrom
resources

Conversation

@sjmiller609
Copy link
Collaborator

@sjmiller609 sjmiller609 commented Dec 26, 2025

Note

Introduces host resource accounting and per-VM rate limiting across CPU, memory, disk, and network.

  • Adds lib/resources (CPU/memory/disk/network discovery, oversubscription, capacity/allocations) and new GET /resources endpoint and types in oapi/openapi.yaml
  • Extends instance API/types to accept and report network.bandwidth_{download,upload} and disk_io_bps; stores in metadata and includes in responses
  • Computes default disk_io_bps and symmetric network limits from CPU share via ResourceManager
  • Implements bidirectional network shaping: TBF on TAP egress (download) and HTB classes on bridge (upload) with setup/cleanup; NetworkManager.SetupHTB, class derivation, orphan cleanup, and restore paths updated
  • Enables disk I/O throttling in hypervisors (Cloud Hypervisor/QEMU) and applies limits to all VM disks
  • Adds config for oversubscription ratios, burst multipliers, and hard capacity limits with validation; providers/wiring initialize ResourceManager
  • Images/volumes expose total sizes for disk accounting; broad test and API fixture updates; minor Makefile test verbosity flag

Written by Cursor Bugbot for commit 366e0f7. This will update automatically on new commits. Configure here.

@github-actions
Copy link

github-actions bot commented Dec 26, 2025

✱ Stainless preview builds

This PR will update the hypeman SDKs with the following commit message.

feat: Resource accounting
⚠️ hypeman-typescript studio · code

There was a regression in your SDK.
generate ⚠️build ✅lint ✅test ✅

npm install https://pkg.stainless.com/s/hypeman-typescript/57ff511aa73bbc08b67496d7b807418b9342dc87/dist.tar.gz
⚠️ hypeman-go studio · code

There was a regression in your SDK.
generate ⚠️lint ✅test ✅

go get github.com/stainless-sdks/hypeman-go@414128770e8137ed2a40d404f0f4ac06ea1a0731
⚠️ hypeman-cli studio · conflict

There was a regression in your SDK.


This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push.
Last updated: 2026-01-05 22:12:05 UTC

@sjmiller609 sjmiller609 requested a review from rgarcia January 5, 2026 15:24
- fq_codel leaf qdisc for low latency under load

**Default limits:**
- Proportional to CPU: `(vcpus / cpu_capacity) * network_capacity`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so if i have 1 vcpu vm on a 256 core machine, with 2.0 oversub and a 10Gbps NIC:

Limit = 10Gbps × 2.0 × (1/256) = ~78 Mbps download and upload

seems low? Maybe we just raise oversub

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and yes I agree / ran into same concept here. But it seems to me that it's sensible to give a low limit by default if the CPU is super high - presumably this user would be planning to deploy 100 VMs on the machine. Also, I think in practice we will not use the default, it seems possibly too high, because the NIC speed is going to be much higher than the actual bandwidth

@sjmiller609 sjmiller609 merged commit 58df3eb into main Jan 5, 2026
3 of 4 checks passed
@sjmiller609 sjmiller609 deleted the resources branch January 5, 2026 22:05
hash := h.Sum32()
// Use only 16 bits (tc class ID max is 0xFFFF)
return fmt.Sprintf("%04x", hash&0xFFFF)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HTB class ID hash collision can fail instance creation

The deriveClassID function uses a 16-bit hash (FNV-1a truncated to 0xFFFF) to generate HTB traffic control class IDs from TAP names. With only 65536 possible values, the birthday paradox makes collisions increasingly likely as VM count grows (~7% chance with 100 VMs, ~26% with 200 VMs). When two TAP names hash to the same class ID, tc class add in addVMClass fails with "File exists", causing instance creation to fail with a cryptic error. The user would need to create a new instance with a different ID to work around this.

Additional Locations (1)

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants