bug: rust-vm E2E intermittently fails with masked ext4 formatter error

[pimlock]

Agent Diagnostic

• Skills loaded: create-github-issue
• OpenShell version tested: GitHub Actions revisions from 5402551797585268356d84b2a3a1c99f1031f076 (the commit that added VM CI coverage) through cbdeb4d537ad8b4b8592596b7668b9d03464544c; the cited failure ran at cbdeb4d
• Latest release checked: v0.0.88
• Known fixes reviewed: v0.0.88 release notes and the merged CI changes in #2305 and #2311; none addresses this recurring formatter failure
• Possible duplicates reviewed: #2308 concerns e2fsprogs licensing/distribution, #1762 concerns the mke2fs -d large-file bug, and #2314 concerns separating provisioning from execution. None tracks this CI failure.
• Findings: From the VM CI job's introduction on 2026-07-16 through 2026-07-22 17:09 UTC, there were 9 failed e2e / E2E (rust-vm) job attempts (including failures hidden by reruns), or 8 others besides the cited example. Seven of the nine—including the example—failed while formatting the VM root disk and ended with /usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found; therefore there are 6 other matching failures. The other two failures were unrelated: a Rustls crypto-provider panic and a disk-full error while applying an image layer.
• Remaining reason for filing: the recurring formatter failure affects branch, dev-release, and tagged-release workflows, while the final error masks the first executable formatter failure and no existing issue tracks the CI instability.

Description

Actual behavior: e2e / E2E (rust-vm) intermittently fails in gateway_smoke while provisioning the first VM sandbox. The CLI reaches Formatting root disk, the sandbox enters ProvisioningFailed, and the reported cause ends with:

failed to create ext4 rootfs image ...:
/usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found.
Install e2fsprogs (mke2fs/mkfs.ext4) and retry

The reusable workflow's Ubuntu dependency step does not explicitly install e2fsprogs.

The error is also misleading: format_ext4_image_from_dir tries mke2fs and mkfs.ext4 from PATH before probing Homebrew-specific candidates, but each failed probe overwrites last_error. A real failure from a system formatter can therefore be replaced by the final nonexistent Homebrew path.

Expected behavior: The rust-vm E2E job should reliably provision the VM root disk on its supported ubuntu-24.04 runner. If formatting fails, logs should retain the executed formatter's status and stderr instead of replacing them with a later candidate's not found error.

Failure history

Matching root-disk formatting failures:

• Branch E2E run 29759361523 — 2026-07-20
• Branch E2E run 29762231843 — 2026-07-20
• Branch E2E run 29766134153 — 2026-07-20
• Branch E2E run 29862032502 — 2026-07-21
• Release Dev run 29919978693 — 2026-07-22 (reported example)
• Release Tag run 29929940153 — 2026-07-22
• Branch E2E run 29933058988 — 2026-07-22

The failures occurred on GitHub ubuntu-24.04 image revisions 20260714.240.1 and 20260720.247.2.

Reproduction Steps

1. Run a workflow that invokes .github/workflows/e2e-test.yml with the rust-vm job enabled on ubuntu-24.04.
2. Let mise run --no-deps --skip-deps e2e:vm pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest.
3. Observe the intermittent failure during gateway_smoke after Preparing rootfs / Formatting root disk.

Environment

• OS: GitHub-hosted Ubuntu 24.04.4
• Runner images: ubuntu24/20260714.240.1, ubuntu24/20260720.247.2
• Docker: not involved in the failing VM root-disk formatting path
• OpenShell: multiple CI revisions after 54025517; cited example cbdeb4d
• Latest release checked: yes, v0.0.88
• Possible duplicates checked: yes; e2fsprogs (GPL-2.0) required as macOS host dependency — request for configurable path and permissive alternatives #2308, bug: VM driver fails to create rootfs for images with files >2 GiB due to mke2fs -d bug #1762, and test(e2e): separate VM gateway provisioning from test execution #2314 are related but do not track this failure

Logs

Pulling image ghcr.io/nvidia/openshell-community/sandboxes/base:latest
Preparing rootfs
Formatting root disk
Error: sandbox entered error phase while provisioning: ProvisioningFailed:
failed to create ext4 rootfs image ...:
/usr/local/opt/e2fsprogs/bin/mkfs.ext4 not found.
Install e2fsprogs (mke2fs/mkfs.ext4) and retry

thread 'gateway_smoke' panicked at tests/smoke.rs:63:10:
sandbox create should succeed
test gateway_smoke ... FAILED

Definition of Done

• The rust-vm E2E job explicitly provides or validates the host tools required for ext4 root-disk creation.
• Formatter failures preserve the actionable status/stdout/stderr from executable candidates.
• The VM E2E path succeeds repeatedly on the supported GitHub-hosted Ubuntu runner.

[pimlock]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — clear path; the masked formatter stderr prevents absolute confirmation of the disk-pressure trigger

Summary

Reduce the rust-vm job's peak temporary-disk usage by deleting downloaded and expanded registry-layer staging after the merged rootfs is complete and before the ext4 image is written. Validate the e2fsprogs tools already present on the GitHub-hosted runner without installing or changing packages, and preserve actionable errors from formatter candidates that actually executed instead of replacing them with later missing Homebrew probes.

Scope

• .github/workflows/e2e-test.yml: validate the preinstalled e2fsprogs tools required by the rust-vm E2E lane without modifying the runner's package set.
• crates/openshell-driver-vm/src/driver.rs: reclaim staging_dir/layers after successful layer application and before rootfs preparation/ext4 formatting; propagate cleanup failures.
• crates/openshell-driver-vm/src/rootfs.rs: prioritize and retain status/stdout/stderr from executable formatter candidates, with candidate-injected logic that is unit-testable without changing global PATH.

Implementation Steps

1. Validate that mke2fs, mkfs.ext4, and debugfs resolve on the vanilla ubuntu-24.04 host before running the E2E suite. Do not install e2fsprogs in this job.
2. After registry layers have been successfully applied into the merged rootfs, remove the compressed and expanded layers/ staging tree before rootfs preparation and ext4 creation. Abort with an actionable error if cleanup fails.
3. Refactor ext4 formatter candidate handling so fallback candidates are still attempted, but later NotFound results cannot overwrite failures from commands that ran. When no candidate executes, retain the existing installation guidance.
4. Add focused unit coverage for staging cleanup and formatter-error selection, then run the driver tests and repository pre-commit suite.
5. Run the VM E2E lane on an Ubuntu/KVM host when available and use repeated GitHub-hosted rust-vm runs to validate the intermittent reliability fix.

Test Plan

• Unit tests: In rootfs.rs, cover a failing executable formatter followed by missing candidates, all candidates missing, and a later successful fallback without relying on global PATH. In driver.rs, verify layer cleanup removes layers/ while preserving the merged rootfs/.
• Integration tests: Run cargo test -p openshell-driver-vm and mise run pre-commit.
• E2E tests: Run mise run --no-deps --skip-deps e2e:vm on Ubuntu/KVM if available. Confirm at least three consecutive GitHub-hosted E2E (rust-vm) executions after the change because the original failure is intermittent.

Risks & Open Questions

• Disk pressure is strongly indicated but the exact system-formatter stderr is irretrievable from past logs because it was overwritten. The failures take long enough to show that at least one formatter executed, all matching runs used the same image digest, and a separate rust-vm attempt explicitly exhausted disk.
• Cleanup after all layers are applied removes the formatter-stage peak. If pull/apply itself later exhausts disk, deleting each blob after extraction and each layer root after application should be handled as a narrower follow-up.
• Cleanup failure should abort rather than warn; continuing would preserve the known high-water mark and recreate the flaky behavior.
• Validation-only intentionally relies on the current GitHub runner image. If a future image removes e2fsprogs, this job should fail immediately and provisioning should be addressed at the runner-image or shared host-setup level.
• Keep this issue limited to ext4 formatter reporting. The similar debugfs last-error pattern is not implicated and can be addressed separately.

Documentation Impact

No gateway configuration, Helm rendering, published user workflow, or stable architecture boundary changes. The VM driver README already documents e2fsprogs as a prerequisite, so no documentation update is expected.

LSM Compatibility

No SELinux/AppArmor compatibility impact is expected. Cleanup is host-side inside the VM driver's own staging directory before guest launch, and tests will avoid fork/exec probes that depend on cross-label /proc/<pid>/exe access.

---

Revision 2 — validate preinstalled e2fsprogs tools without installing packages
Revision 1 — initial plan

[pimlock]

Implemented in #2425.

Summary:

• Reclaim temporary compressed and extracted registry layers before formatting the merged VM rootfs, reducing peak runner disk usage without changing the durable image cache.
• Preserve actual formatter execution failures instead of replacing them with a later missing fallback-path error.
• Validate the GitHub-hosted rust-vm runner's existing mke2fs, mkfs.ext4, and debugfs tools without installing e2fsprogs in the job.

Verification:

• mise exec -- cargo test -p openshell-driver-vm — 135 tests passed (123 library, 12 binary).
• mise run pre-commit — passed, including formatting, clippy, workspace tests, and server feature tests.
• E2E phase not applicable under the build workflow because no files under e2e/ changed; PR CI will exercise the updated rust-vm workflow.

Deviations from the approved plan: none.