Evaluate libkrun/krunkit Podman-machine provider to address AVF virtio-fs root cause (#376/#397 follow-up)

[aviadshiber]

Context

The macOS mount-failure mitigations (#248, #276, #303, #376, #397) all work around a single upstream root cause: Apple Virtualization.framework's virtio-fs server has a cache-coherency/concurrency bug (Apple Feedback FB16008360) where metadata ops (mkdirat, getxattr, overlay metacopy) intermittently fail under load and across sleep/wake. As of the research done for #397, this is unfixed upstream (containers/podman#23061 closed as "Apple's bug"; #24725 open).

Idea

The libkrun / krunkit Podman-machine provider ships its own virtio-fs implementation (not AVF's), with stricter semantics that reportedly avoid AVF's laxness. Switching the default macOS provider could address the root cause directly and let us retire the accumulated workarounds (vfkit watchdog, sleep prevention, overlay-volume placement, status named-volume mirroring) over time.

What to evaluate

• Reliability: does krunkit's virtio-fs actually eliminate the exit_code=4 / error_type=mount_failure class under sustained + concurrent agent I/O and across sleep/wake?
• Operational cost: provider install/bootstrap, podman machine init --provider, migration for existing users.
• Trade-offs: performance, GPU passthrough, maturity vs. AVF, CI implications.
• Which existing mitigations become removable vs. should stay as defense-in-depth.

Pointers

• scripts/lib/podman-health.sh, scripts/lib/vfkit-watchdog.sh, scripts/lib/status-sync.sh, overlay-volume work in scripts/lib/overlay-sandbox.sh.
• CLAUDE.md "Mount Failure Detection" section enumerates the current stack.
• Refs: https://github.com/containers/libkrun · macOS bind mount permission issues: libkrun (❌) vs. applehv (✅ ) podman-container-tools/podman#27679

Tracked from the #397 overlay research. https://claude.ai/code/session_01GyUwb9P9abPA2WnUAqnCUU

[aviadshiber]

Research: libkrun/krunkit provider vs. the AVF virtio-fs root cause

TL;DR

The premise of this issue is confirmed and stronger than expected: krunkit ships its own in-process Rust virtio-fs server (libkrun uses Hypervisor.framework directly, bypassing Apple Virtualization.framework entirely), podman maintainers confirmed libkrun does not exhibit the AVF bug, and — big news — Podman v6.0.0 (2026-06-24) made libkrun the default macOS machine provider. However, the switch is days old, there's already an open FS-performance regression report against it, krunkit's own virtio-fs permission fixes only landed in June 2026, and sleep/wake behavior is publicly unvalidated. Recommended posture: opt-in experiment now, default later — and one of our mitigations (the #303 vfkit watchdog) is currently a silent no-op under krunkit and needs fixing regardless.

1. Reliability: does krunkit avoid the AVF bug?

• The AVF failure class (spurious EACCES/ENOENT on metadata ops — mkdirat, getxattr, overlay metacopy — under load and across sleep/wake) was traced to the FUSE server backing AVF's virtiofs (VZVirtioFileSystemDeviceConfiguration), tracked as Apple Feedback FB16008360. See podman#24725 → converted to discussion #24751.
• In that discussion the reporter verified libkrun does not exhibit the problem, because libkrun "uses the Hypervisor Framework directly rather than Apple's Virtualization Framework" — the virtio-fs device is implemented inside libkrun itself (Rust), not by macOS.
• Status correction for this issue's description: #23061 closed 2024-09-06 and #24725 closed 2024-12-03 (moved to #24751) — neither is open; Apple has shipped no fix.
• krunkit's virtio-fs has had its own issue class — stricter (more correct) permission semantics broke bind-mount writes where host/guest UIDs mismatch (discussion #27679). Actively fixed through June 2026: libkrun#759 (simplified permission semantics), krunkit#108 (configurable, v1.3.0), libkrun#734 (host UID/GID → guest root mapping on macOS), libkrun#693 (DirStream cache-coherence fix), plus a pjdfstest-based virtio-fs test suite (libkrun#639). Good hardening signal — but also proof the implementation was still getting correctness fixes weeks ago. Version-pin: krunkit ≥ 1.3.1.
• Sleep/wake: no public evidence either way. The AVF-specific failure mode can't occur (no AVF in the data path), but krunkit's own resilience across sleep/wake is exactly what we must test ourselves.

2. Maturity & support

Aspect	Status
Podman default	libkrun is the default macOS provider since v6.0.0 (2026-06-24); was deferred to 6.0 explicitly as a breaking change in #24751
krunkit release	v1.3.1 (2026-07-01), active repo (libkrun/krunkit — note org move from containers/)
Requirements	macOS 14+ and Apple Silicon only — Intel Macs cannot use libkrun (lima docs, Podman Desktop)
Install	podman GitHub .pkg installer bundles krunkit (#27056, closed 2025-09); brew install podman does not — needs brew tap libkrun/krun && brew install krunkit
Ecosystem	Podman Desktop defaults to libkrun on ARM64 (PR #12786); minikube and Lima 2.0 ship krunkit drivers (Lima: "experimental")

3. Trade-offs

• Performance — open question. podman#29087 (filed 2026-07-01, macos/machine/6.0): FS-heavy workload went 16s → ~97s after upgrading to 6.0; maintainer points at the libkrun default switch. No published virtio-fs benchmarks krunkit vs applehv exist. Kapsis agent workloads are FS-heavy — we must benchmark before recommending.
• No Rosetta under libkrun (Rosetta requires AVF, per maintainers in #24751). Kapsis builds arch-native arm64 images so low impact for us; affects amd64-only images.
• GPU (Vulkan/Venus) is libkrun-exclusive among macOS providers (Red Hat article) — irrelevant to isolation, nice future bonus.
• CI: neutral. Kapsis CI is all ubuntu-latest; GitHub-hosted macOS runners can't run podman machine with any provider (no nested virt on Apple Silicon runners — runner-images#9304).

4. Migration mechanics

• No in-place conversion — a new machine is required: podman machine init --provider libkrun. Images and named volumes (our per-agent Maven/Gradle caches, kapsis-*-status volumes) live inside the VM disk and are lost → one-time rebuild cost.
• Since podman 6.0, machines from different providers coexist and all machine commands see all of them; the configured provider only affects new machines. Only one machine runs at a time (#26281 still open), so A/B testing means stop/start cycling.

5. Impact on kapsis mitigations

Mitigation	Under krunkit	Action
vfkit watchdog (#303, scripts/lib/vfkit-watchdog.sh)	Silently no-ops today — pgrep -f \"vfkit.*${machine}\" finds nothing on a libkrun machine	Fix now: generalize to `(vfkit
Sleep prevention / caffeinate (#276)	Root-cause trigger removed in theory, unvalidated	Keep; first retirement candidate after burn-in
Status named-volume mirroring (#276, status-sync.sh)	Exists solely because AVF bind mounts die mid-run	Keep; second retirement candidate after burn-in
Pre-launch probe (podman-health.sh), entrypoint mount probe, liveness monitor (#248/#257)	Provider-agnostic, work unchanged	Keep as defense-in-depth
Overlay-volume placement (#376/#397, overlay-sandbox.sh)	Still beneficial (perf + isolation)	Keep

Recommendation

Opt-in experiment now; revisit default after podman 6.0.x stabilizes. The root cause is genuinely addressed — upstream agrees strongly enough to have flipped the default — but 10 days of field exposure, an open perf regression, June-2026 correctness fixes, and zero sleep/wake data mean kapsis shouldn't bet its default on it yet. Nothing gets deleted in phase 1; the mitigation stack must also survive for Intel/macOS-13/Rosetta users who can never use libkrun.

Proposed next steps

• Bugfix (independent of adoption): generalize vfkit-watchdog.sh process match to (vfkit|krunkit) so feat(liveness): host-side vfkit watchdog for fast mid-run virtio-fs drop detection #303 protection isn't silently lost on libkrun machines
• Add provider detection (podman machine ls --format '{{.VMType}}') to preflight + launch logging; consider a machine_provider field in status.json (⚠️ triggers the Dashboard Sync Rule — dashboard PR or dashboard-sync follow-up issue)
• Stand up a libkrun machine (podman ≥ 6.0, krunkit ≥ 1.3.1, macOS 14+/Apple Silicon) and run the feat: detect /workspace virtio-fs mount drop mid-run (silent disconnect) #248/[overlay mode] /kapsis-status/ bind mount intermittently inaccessible at container start — virtio-fs degradation, exits 1 with no output #276 reproduction scenarios: sustained concurrent agent I/O + forced sleep/wake cycles with prevent_sleep: false
• Benchmark a representative kapsis agent workload (clone + build + test) krunkit vs applehv; watch podman#29087
• Document the opt-in path in docs/ (new machine init, cache/image rebuild cost, Intel + Rosetta exclusions, brew tap for krunkit)
• After a multi-week clean burn-in: open follow-ups to retire status volume mirroring and caffeinate behind provider detection, and to recommend libkrun as the kapsis default on eligible hosts

_{Research conducted 2026-07-03. Note: containers/podman now redirects to podman-container-tools/podman; krunkit/libkrun moved from the containers org to libkrun.}