feat(sandbox): add Apple Container backend for macOS Virtualization.framework

[explicitcontextualunderstanding]

feat(sandbox): add Apple Container backend for macOS Virtualization.framework

What this solves

BenchFlow users on Apple Silicon can run public-network, single-container arm64
tasks locally with Apple's Virtualization.framework backend instead of Docker
Desktop or a cloud sandbox:

bench eval run ... --sandbox apple-container

The backend requires Apple Container 1.1+ and intentionally fails closed for
unsupported hosts, no-network tasks, multi-service tasks, unsafe kernel-zone
headroom, and snapshot workflows.

Implementation

• Uses Apple Container's native 1.1+ primitives:
  • container run --detach
  • container exec --interactive --user/--workdir
  • container cp
  • build/run --platform linux/arm64
• Runs the mandatory LiteLLM and replay proxies inside the VM, so agents never
  receive an unreachable host-loopback endpoint.
• Adds a native bidirectional ACP transport over container exec --interactive;
  Apple Container no longer falls through to Daytona internals.
• Preserves image-selection parity with Docker and Daytona:
  environment.docker_image wins unless force_build=True.
• Keeps /logs as the only host bind mount. Task, verifier, oracle, and skills
  content is copied through the native CLI, so the VM cannot mutate source files.
• Keeps provider secrets out of host process argv by writing mode-600 env files
  through stdin and deleting them before agent execution.
• Parses the live macOS zprint -H -L data.kalloc.1024 schema correctly, fails
  closed when it cannot verify usage, and serializes Apple rollouts within each
  BenchFlow process.
• Leaves global Apple BuildKit ownership alone during sandbox cleanup.
• Removes the empty sandbox-apple-container package extra and the stale
  PR-only design document.
• Upgrades mcp from 1.27.2 to 1.28.1, fixing the current CI dependency advisory.

Supported scope

• macOS 26+ on Apple Silicon
• Apple Container 1.1+
• public-network, single-container arm64 tasks
• no snapshots
• one active Apple rollout per BenchFlow process

The data.kalloc.1024 leak is system-wide, so concurrent BenchFlow processes are
still discouraged. Docker, Daytona, or Modal remain the supported backends for
no-network, multi-service, snapshot, and high-concurrency runs.

Exact-head validation

Current head: d6d9d42897c75e9e7fe870dd0ecf236385e73096

Real Apple Container lifecycle

• Official signed/notarized Apple Container 1.1.0 package
• Real detached VM lifecycle, native exec, file upload/download, dotfiles, and
  nested-directory transfer: passed
• Gated integration test:
  test_real_apple_container_lifecycle_and_copy: 1 passed

Real BenchFlow evaluations

• Oracle, citation-check, Apple Container: reward 1.0, 1/1, zero errors
• pi-acp + Azure OpenAI gpt-5.5 + xhigh, no-skill, Apple Container:
  reward 1.0, 1/1, zero errors, 10 tool calls
• Docker parity with the same task/agent/model/effort:
  reward 1.0, 1/1, zero errors, 6 tool calls; validator healthy with
  7/7 successful exchanges and 65,144 provider-accounted tokens
• Raw artifact audit:
  • 13 ACP events
  • 11/11 successful LLM request/response exchanges
  • all requests and responses recorded reasoning_effort=xhigh
  • 101,442 total tokens with provider usage
  • complete timing and verifier artifacts
  • zero skill invocations or skill-path access
  • no oracle/verifier access or reward-hacking behavior

Repository gates

• uv sync --extra dev --extra sandbox-daytona --extra sandbox-modal --locked
• uv run ruff check .
• uv run ruff format --check src tests
• uv run ty check src/
• uv lock --check
• git diff --check
• focused changed-surface suite: 190 passed, 1 skipped
• full suite: 4,945 passed, 9 skipped, 7 deselected
• exact CI-shaped pip-audit: No known vulnerabilities found

Remaining merge gates

• Current-head GitHub Actions are green: test, pip-audit, manifest parity,
  integration-light, and integration-scope.
• A human non-author must still re-review and clear the existing
  CHANGES_REQUESTED state; this is the only remaining merge gate.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Daily scan thermo-nuclear review (2026-07-25): requesting changes on current head f8ac25cc3754bb823b284936d181d618b7fa04d9.

High-confidence blockers:

1. src/benchflow/sandbox/apple_container.py:223 mounts the task environment_dir directly over /app as a writable host bind mount. That changes the core Docker/Daytona semantics: /app is no longer the image filesystem produced by the Dockerfile, and agent writes can mutate the repo/task environment on the host across trials. This needs a per-rollout staged workspace or another design that does not overwrite image state and does not let the agent write back into source task files.

2. src/benchflow/sandbox/apple_container.py:196 builds container run without honoring task_env_config.allow_internet / environment.network_mode=no-network. Other backends enforce this contract (Docker adds the no-network compose overlay; Modal/Daytona block network). As written, no-network tasks silently run with public network in the new backend.

3. src/benchflow/sandbox/apple_container.py:509 maps /logs and /app paths back to host paths using string-prefix checks and host_base / rel without normalizing/rejecting ... A path like /logs/../../... can escape the rollout bind root in the host-copy fast path. The host-side mount optimization needs a strict POSIX-path boundary check before any shutil.copy* / rmtree.

4. src/benchflow/sandbox/apple_container.py:29 parses the sample zprint columns incorrectly: the test itself labels parts[4] as elems and parts[6] as maxelts, but the implementation returns parts[6] as current usage. That makes the kalloc preflight/headroom gate unreliable.

Validation gates from local checkout:

• uv sync --extra dev --locked on a pristine PR worktree fails because uv.lock needs updating after adding sandbox-apple-container.
• After uv updates the local lock metadata, uv run --extra dev ty check src fails at src/benchflow/sandbox/apple_container.py:491 because _log_file: object | None has no typed .close().
• Focused PR tests passed only after local lock metadata was updated: tests/test_apple_container_sandbox.py tests/test_sandbox_provider_registry_drift.py -> 33 passed, 1 skipped.
• Live GitHub currently reports no checks on this PR.

Label sync: I added enhancement, P1, status:blocked, review:changes-requested, and area:sandbox. This should not be merged until the sandbox contract and CI/type/lock gates are fixed.

AI-generated automation review posted on behalf of Bingran You.

[bingran-you]

Users Simulation automation review (2026-07-25): blocked on current head f8ac25cc3754bb823b284936d181d618b7fa04d9.

User-simulation gates:

• Fresh isolated checkout: uv sync --extra dev --locked fails immediately because pyproject.toml adds sandbox-apple-container but uv.lock is not updated.
• After updating lock metadata only in the temp worktree, focused Apple Container tests pass (33 passed, 1 skipped) and focused ruff passes, but ty check src/benchflow/sandbox/apple_container.py fails: _log_file: object | None has no typed .close() at apple_container.py:491.
• Live GitHub has no usable check/artifact evidence for this head; workflows are action_required / no check rollup.

Thermo/sandbox blockers from review still apply: writable host bind over /app, no no-network enforcement for the new backend, unsafe host path prefix mapping for /logs / /app, and unreliable zprint kalloc parsing. No BenchFlow rollout artifact was accepted for this head.

Labels are already correct: keep status:blocked + review:changes-requested.

AI-generated automation review posted on behalf of Bingran You.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Users Simulation automation follow-up (2026-07-25): pushed 9508af50df200f9a7ad695762b7d24acd7dc2037 to address the current repo-owned blockers from head f8ac25cc3754bb823b284936d181d618b7fa04d9.

Fixed in this commit:

• uv.lock now includes the sandbox-apple-container extra metadata.
• Removed the writable /app host bind; /logs remains the only host-mounted fast path, while /app uploads go through container exec -i.
• Added strict POSIX boundary checks for /logs host-copy paths so .. traversal cannot escape the rollout bind root.
• Fixed data.kalloc.1024 parsing to read the elems column instead of maxelts.
• Typed _log_file as TextIO, clearing the previous ty failure.
• Fail closed for network_mode="no-network" on apple-container at runtime capability validation and backend construction because the backend does not yet enforce VM network isolation.
• Fixed CLI help drift so eval/environment sandbox help advertises the registered provider list including apple-container.

Validation on the pushed code:

• uv sync --extra dev --extra sandbox-daytona --locked
• uv run --extra dev --extra sandbox-daytona --locked ruff check .
• uv run --extra dev --extra sandbox-daytona --locked ruff format --check src tests
• uv run --extra dev --extra sandbox-daytona --locked ty check src/
• uv lock --check
• git diff --check
• uv run --extra dev --extra sandbox-daytona --locked python -m pytest tests/ -q -> 4932 passed, 17 skipped, 7 deselected

Labels intentionally stay status:blocked + review:changes-requested: live GitHub still reports the old CHANGES_REQUESTED review gate, and this fork PR still has no check-run/status rollup on the new head. Next step is human re-review plus CI/workflow approval/evidence.

AI-generated automation follow-up posted on behalf of Bingran You.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Users Simulation automation final-head follow-up (2026-07-25): final pushed head is 4c2dd37128e40cf7e8e3946ae3a31a0fde53d9e7.

After the code-fix commit 9508af50d, I added 4c2dd3712 only to correct the Apple Container PR doc from the stale 28-test count to the current focused result (31 passed, 1 skipped). No behavior changed after the validated code-fix commit.

The validation evidence remains:

• uv sync --extra dev --extra sandbox-daytona --locked
• ruff check ., ruff format --check src tests, ty check src/
• uv lock --check, git diff --check
• Apple Container focused tests: 31 passed, 1 skipped
• Full unit suite: 4932 passed, 17 skipped, 7 deselected

Labels intentionally remain status:blocked + review:changes-requested because GitHub still reports CHANGES_REQUESTED, and the fork PR still has no check-run/status rollup on the final head. Next action is human re-review plus CI/workflow approval/evidence.

AI-generated automation follow-up posted on behalf of Bingran You.

[explicitcontextualunderstanding]

Re-review request: all 4 blockers resolved, independently validated

All issues from the thermo-nuclear review on f8ac25c are addressed in 9508af5 + 4c2dd37. Independently validated on Apple Silicon (M2, macOS 26) — not relying on self-reported automation output.

Blocker Resolution

#	Issue	Fix Summary	Location
1	Writable host bind over /app	Removed. Only /logs is bind-mounted; /app uploads route via container exec -i (base64 stdin).	apple_container.py start() — no /app in cmd_args; upload_file() falls through to _run_cli("exec", "-i", ...).
2	No no-network enforcement	Fail-closed: _validate_definition() raises ValueError if allow_internet=False; runtime_capabilities.py flags the gap at task-validation time.	apple_container.py:148-153 runtime_capabilities.py:264-270
3	Path traversal in /logs host-copy	Three-layer defense: reject .. parts → .resolve() both paths → candidate.relative_to(host_root) containment check.	apple_container.py _mounted_host_path()
4	zprint kalloc parsing (elems vs maxelts)	Reads parts[4] (elems column) for current usage.	apple_container.py:38

Auxiliary fixes:

• uv.lock provides-extras updated.
• _log_file typed as TextIO | None (clears ty type-check failure).

---

Independent Validation (2026-07-25, M2 Mac, head 4c2dd371)

uv sync --extra dev --extra sandbox-daytona --locked   ✓
uv lock --check                                        ✓ (221 packages)
ruff check (all changed files)                         ✓
ruff format --check src tests                          ✓ (540 files)
ty check src/benchflow/sandbox/apple_container.py      ✓

pytest tests/test_apple_container_sandbox.py -k "not Integration"
  → 31 passed

pytest tests/ -q -k "not IntegrationLifecycle"
  → 4932 passed, 16 skipped, 8 deselected (126s)

pytest tests/test_apple_container_sandbox.py::TestIntegrationLifecycle
  → 1 passed (18.7s, real ubuntu:24.04 micro-VM lifecycle)

---

Remaining Gate

This fork PR has no CI check-run rollup (workflows require maintainer approval). Requesting:

1. Workflow approval so CI evidence lands on the PR
2. Re-review to clear CHANGES_REQUESTED

Happy to address any further feedback.

[bingran-you]

Daily scan re-review (2026-07-26) on current head 4c2dd37128e40cf7e8e3946ae3a31a0fde53d9e7.

I rechecked the final head in a fresh detached worktree. The concrete blockers from the earlier f8ac25c review are no longer reproduced in this local slice: the branch no longer bind-mounts /app, no-network tasks fail closed, /logs host-copy paths now use containment checks, zprint parsing reads the elems column, _log_file is typed, and the lock metadata is present.

Fresh local validation from this scan:

• uv sync --extra dev --extra sandbox-daytona --locked -> passed
• uv run --extra dev python -m pytest tests/test_apple_container_sandbox.py tests/test_sandbox_provider_registry_drift.py tests/test_runtime_capabilities.py -q -> 66 passed, 1 skipped
• changed-file ruff check -> passed
• uv run --extra dev ty check src -> passed
• git diff --check origin/main...HEAD -> passed

I am not clearing the merge gate from automation: live GitHub still has no check-run/status rollup for this PR head, the old formal CHANGES_REQUESTED state is still active, and I did not run the real macOS Apple Container lifecycle/E2E test from this machine. Labels should stay status:blocked + review:changes-requested until workflow approval/CI evidence and human re-review land.

AI-generated automation re-review posted on behalf of Bingran You.

[bingran-you]

Users Simulation current-head follow-up for 4c2dd37128e40cf7e8e3946ae3a31a0fde53d9e7.

Verdict: still blocked.

What passed:

• uv sync --extra dev --extra sandbox-daytona --locked
• focused sandbox/apple-container tests: 141 passed, 2 skipped
• full suite: 4932 passed, 17 skipped, 7 deselected
• repo ruff, format, ty, lock, diff checks
• CLI help/task-check and SDK Scene / RolloutConfig smoke

Current blockers:

• No CI/check-rollup or healthy current-head integration artifact. The local artifact probe was 0/1 healthy: no llm_trajectory.jsonl, no agent events, no token/tool usage, and no reward.
• The Apple backend still prefers building when a Dockerfile exists and ignores the manifest/prebuilt docker_image path; it also does not honor the Docker/Daytona-style force_build distinction, so user manifests do not behave consistently across backends.
• Persistent environment values are placed directly into the container run argv. That can expose provider keys through host process inspection; use an env-file style boundary instead of argv for secrets.
• stop() can stop the global buildkit VM, which creates a concurrency race for any user running more than the documented single Apple Container session.
• Static docs still do not describe Apple Container support or the current operational limits.

Labels look correct as-is: keep status:blocked and review:changes-requested until the sandbox-contract gaps and exact-head artifact/CI evidence are fixed.

This comment was generated by an AI agent on behalf of Bingran You.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Users Simulation current-head follow-up for f2d3556f42a1ddbb782afc425a6a43cce7a71697.

Pushed f2d3556f4 to address the current repo-owned Apple Container blockers from 4c2dd3712.

Fixed in this commit:

• Prebuilt/manifest image selection now matches Docker and Daytona: environment.docker_image wins unless force_build=True.
• force_build=True still rebuilds from the local Dockerfile and keeps the arm64 Dockerfile guard on the build path.
• Persistent environment values are no longer placed on the host-visible container run argv; exec-time commands use the existing env-file wrapper instead.
• The backend no longer stops the global buildkit VM after build or during sandbox stop, avoiding a concurrency race with unrelated Apple Container builds.
• Static docs now describe --sandbox apple-container and the current limits: Apple Silicon/macOS only, public-network tasks only, single-container, no snapshot support, low concurrency, prebuilt-image behavior, and BuildKit ownership.

Validation on the pushed code:

• uv sync --extra dev --extra sandbox-daytona --locked
• uv run --extra dev --extra sandbox-daytona --locked ruff check .
• uv run --extra dev --extra sandbox-daytona --locked ruff format --check src tests
• uv run --extra dev --extra sandbox-daytona --locked ty check src/
• uv lock --check
• git diff --check
• Apple Container focused tests: 36 passed, 1 skipped
• Apple Container + provider/runtime focused tests: 71 passed, 1 skipped
• Full unit suite: 4937 passed, 17 skipped, 7 deselected

Still blocked:

• This machine does not have the Apple container CLI, so I did not run the real macOS Apple Container lifecycle/E2E test here.
• GitHub Actions for this fork head are still action_required, so there is no current-head CI/check-rollup yet.
• GitHub still reports the active CHANGES_REQUESTED review gate.

Labels intentionally remain status:blocked + review:changes-requested until workflow approval/CI evidence, human re-review, and real Apple Container E2E or healthy artifact evidence land.

AI-generated automation follow-up posted on behalf of Bingran You.

[greptile-apps]

Your trial has ended. Reactivate Greptile to resume code reviews.

[bingran-you]

Thermo/end-to-end follow-up for current head d6d9d42897c75e9e7fe870dd0ecf236385e73096.

All repo-owned blockers are fixed on this head:

• native Apple Container 1.1+ detached lifecycle, ACP stdio transport, native
  user/workdir, arm64 build/run, and container cp
• sandbox-local model/replay proxy routing
• no writable task/skills mounts and no provider secrets on host argv
• correct fail-closed data.kalloc.1024 parsing/headroom plus process-local
  lifecycle serialization
• Docker/Daytona image-selection parity and no global BuildKit shutdown
• exact directory-content copy semantics, including dotfiles and nested trees
• the CI mcp advisory is fixed by locking 1.28.1

Exact-head local acceptance:

• real Apple Container 1.1 lifecycle integration: passed
• real Apple oracle evaluation: reward 1.0, 1/1, zero errors
• real no-skill pi-acp + Azure gpt-5.5 at xhigh: reward 1.0, 1/1,
  zero errors, 10 tool calls
• live Docker parity with the same task/agent/model/effort: reward 1.0, 1/1,
  zero errors; validator healthy with 7/7 exchanges, 65,144 tokens, and no
  skill/verifier access
• raw artifact validator: healthy; 11/11 LLM exchanges with usage, complete
  ACP/timing/verifier artifacts, xhigh recorded on every exchange, no skill
  access, no verifier/oracle access, and no reward hacking
• full suite: 4,945 passed, 9 skipped, 7 deselected
• Ruff lint/format, ty, lock, diff, and exact CI-shaped pip-audit: passed

The PR description now reflects the actual implementation and supported scope.
Current-head test, pip-audit, manifest-parity, integration-light, and
integration-scope workflows are all green. The only remaining merge gate is the
required human non-author re-review; the blocked labels should remain until that
review clears the historical CHANGES_REQUESTED state.

This comment was generated by an AI agent on behalf of Bingran You.

[bingran-you]

Approved on current head d6d9d42897c75e9e7fe870dd0ecf236385e73096.

The requested sandbox, isolation, secret-handling, proxy-placement, lifecycle,
file-transfer, resource-safety, dependency-audit, and parity blockers are
resolved. Real Apple Container and Docker task runs passed with healthy
artifacts, and all current-head GitHub checks are green.

[bingran-you]

@explicitcontextualunderstanding thank you for contributing the Apple Container
backend and for iterating through the review feedback. The implementation is now
validated end to end on a real Apple Container VM, with Docker parity and green
current-head CI. This has been squash-merged—appreciate the contribution!