From 07ea597107201abd24c3bb1425fe724ff4bcd492 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Mon, 29 Jun 2026 15:14:14 +0200 Subject: [PATCH 1/7] chore(spike): scaffold smolvm isolation spike and preflight Add the spikes/smolvm harness directory: a README stating the three unknowns (network/filesystem/agents-in-guest) and the execution-model caveat, a preflight script that verifies a smolvm install and reports backend/arch, and a findings template. No src/ changes; this is groundwork for the isolation probes. --- Run-Id: smolvm-isolation-spike-1782738661332 Short-Name: smolvm-isolation-spike Phase-Id: phase-01 Phase-Title: Spike scaffold and smolvm preflight Model: claude-sonnet-4-6 Effort: low Worktree: /Users/remyloubradou/.phax/worktrees/phax.smolvm-isolation-spike/phase-01 Session-Id: 0c33a1a2-8b3f-405d-aabd-e526821889c0 Gate-Log: /Users/remyloubradou/.phax/runs/phax.smolvm-isolation-spike/phase-01/checks-attempt-01.log --- phax.json | 9 +--- spikes/smolvm/00-preflight.sh | 51 ++++++++++++++++++ spikes/smolvm/README.md | 84 ++++++++++++++++++++++++++++++ spikes/smolvm/findings/TEMPLATE.md | 30 +++++++++++ 4 files changed, 166 insertions(+), 8 deletions(-) create mode 100644 spikes/smolvm/00-preflight.sh create mode 100644 spikes/smolvm/README.md create mode 100644 spikes/smolvm/findings/TEMPLATE.md diff --git a/phax.json b/phax.json index dc2152b2..c057f7db 100644 --- a/phax.json +++ b/phax.json @@ -6,14 +6,7 @@ "filesystem": { "allowWrite": ["~/.phax"] }, - "agentCommands": [ - "deno", - "ctx7", - "usage", - "pnpm gen:usage-spec", - "pnpm docs:cli", - "smolvm" - ] + "agentCommands": ["deno", "ctx7", "usage", "pnpm gen:usage-spec", "pnpm docs:cli", "smolvm"] }, "review": { "compliance": { "enabled": true } }, "publish": { diff --git a/spikes/smolvm/00-preflight.sh b/spikes/smolvm/00-preflight.sh new file mode 100644 index 00000000..7684441a --- /dev/null +++ b/spikes/smolvm/00-preflight.sh @@ -0,0 +1,51 @@ +#!/bin/sh +# 00-preflight.sh — verify smolvm is installed and report version/backend/arch +# Does NOT boot a VM. Exit non-zero if smolvm is missing. +set -eu + +fail() { + printf 'ERROR: %s\n' "$1" >&2 + exit 1 +} + +# Check smolvm is on PATH +if ! command -v smolvm > /dev/null 2>&1; then + fail "smolvm not found on PATH. Install it before running this spike. + See: https://github.com/wasm-forge/smolvm or your platform's package manager." +fi + +printf '=== smolvm preflight ===\n\n' + +# Version +printf '-- smolvm version --\n' +smolvm --version + +# Host architecture +printf '\n-- host arch --\n' +uname -m + +# OS/kernel +printf '\n-- host OS --\n' +uname -s -r + +# Resolved binary location +printf '\n-- smolvm binary --\n' +command -v smolvm + +# Attempt to surface the VMM backend smolvm will use. +# smolvm may expose this via a subcommand or env var; probe common options. +printf '\n-- VMM backend (best effort) --\n' +if smolvm info > /dev/null 2>&1; then + smolvm info +elif smolvm config > /dev/null 2>&1; then + smolvm config +else + # Fall back to inspecting env hints + if [ -n "${SMOLVM_BACKEND:-}" ]; then + printf 'SMOLVM_BACKEND=%s\n' "$SMOLVM_BACKEND" + else + printf '(smolvm does not expose a backend info command; check docs for VMM selection)\n' + fi +fi + +printf '\n=== preflight OK ===\n' diff --git a/spikes/smolvm/README.md b/spikes/smolvm/README.md new file mode 100644 index 00000000..fb6be58b --- /dev/null +++ b/spikes/smolvm/README.md @@ -0,0 +1,84 @@ +# smolvm Isolation Spike + +This directory contains the harness scripts and findings documents for the smolvm +isolation feasibility spike. The goal is to answer three unknowns that decide whether +phax's `isolated` mode should be built on smolvm before writing any production code. + +## Context: `isolated` mode in phax + +phax ships two security modes: `default` (no constraints) and `secure` +(provider-native guardrails, where the agent knows its limits). A third mode, +`isolated`, is reserved in the CLI and rejected before a run starts. The spec notes +`isolated` is intended for "an external sandbox (smolvm or similar)" — a microVM with +a host-mounted worktree, no `$HOME` access, and a real network allowlist. Separately, +spec-14 established that network allowlisting cannot be enforced at the provider-native +layer. The network boundary the developer actually wants — deny-by-default egress, +allow by domain — is exactly the gap a microVM is meant to fill. + +## The three unknowns + +### 1. Network — per-domain or per-IP, and is it a real boundary? + +smolvm documents hostname allowlisting (`--allow-host` / `allow_hosts = [...]`). We +must confirm it: + +- (a) blocks non-allowed domains, +- (b) still blocks a non-allowed host reached by hard-coded **IP** (the decisive test), +- (c) explain the enforcement mechanism (DNS/SNI proxy vs true egress filter). + +If a non-allowed IP bypasses the allowlist, the boundary is DNS-name filtering only — +not a security boundary. See `02-network.sh` and `findings/02-network.md`. + +### 2. Filesystem — deny-by-default with explicit shares + +Confirm the host FS is invisible to the guest, only the mounted worktree is reachable, +host `$HOME` and credentials are not accessible, and whether mounts can be read-only. +See `01-filesystem.sh` and `findings/01-filesystem.md`. + +### 3. Agents-in-guest + +Confirm Claude Code, Codex, and Mistral Vibe actually run in the libkrun Linux guest +(cross-arch on Apple Silicon), with credentials injected via env, constrained by the +network allowlist — and observe how each agent surfaces a denied egress (this motivates +the capability-preamble prompt in the synthesis). See `03-agents.sh` and +`findings/03-agents.md`. + +## Execution-model caveat + +Booting a microVM requires host virtualization (Hypervisor.framework / KVM), a real +`smolvm` install, real provider credentials, and a human (or e2e harness) watching +whether egress is actually blocked. A phax phase agent running in a worktree cannot +reliably nest a microVM or self-verify a network escape. + +Therefore, for the probe phases, **the agent's mechanical deliverable is the harness +script plus a findings document with an empty `## Results` / `## Verdict` section**; +the real VM run is performed out-of-band and its output is pasted into the findings +doc. Treat the synthesis in `docs/spikes/smolvm-isolation-findings.md` as provisional +until a real run fills in the Results sections. + +## How to run + +Run the harnesses in order on a host with smolvm installed: + +```sh +# 0. Preflight — verify smolvm is installed and report version/backend/arch +sh spikes/smolvm/00-preflight.sh + +# 1. Filesystem isolation probe +sh spikes/smolvm/01-filesystem.sh + +# 2. Network per-domain allowlist probe +sh spikes/smolvm/02-network.sh + +# 3. Agents-in-guest execution probe (requires provider credentials in env) +ANTHROPIC_API_KEY=... OPENAI_API_KEY=... MISTRAL_API_KEY=... \ + sh spikes/smolvm/03-agents.sh +``` + +After each run, paste the raw output into the `## Results` section of the corresponding +findings doc and fill in the `## Verdict`. + +## Synthesis + +Once all three probes have been run, the synthesis and go/no-go recommendation is in +`docs/spikes/smolvm-isolation-findings.md`. diff --git a/spikes/smolvm/findings/TEMPLATE.md b/spikes/smolvm/findings/TEMPLATE.md new file mode 100644 index 00000000..43ece2bd --- /dev/null +++ b/spikes/smolvm/findings/TEMPLATE.md @@ -0,0 +1,30 @@ +# Findings: [Probe name] + +## Environment + +- smolvm version: +- Host OS / arch: +- Guest arch: +- Date of run: +- Operator: + +## Procedure + + + +## Results + + + +## Verdict + + + +**Status:** (not yet run) + +**Conclusion:** + +## Open questions + + From 6a6c86d9492facdcbf0b8ae13072d3af0045370c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Mon, 29 Jun 2026 15:20:08 +0200 Subject: [PATCH 2/7] test(spike): add smolvm filesystem isolation probe Add a harness that mounts one host dir as /workspace and asserts the rest of the host FS (HOME, repo root, /etc) is invisible to the guest, plus a read-only-mount check. Findings doc captures procedure; Results/Verdict are filled from a real run out-of-band per the spike's execution-model caveat. --- Run-Id: smolvm-isolation-spike-1782738661332 Short-Name: smolvm-isolation-spike Phase-Id: phase-02 Phase-Title: Filesystem isolation probe Model: claude-sonnet-4-6 Effort: medium Worktree: /Users/remyloubradou/.phax/worktrees/phax.smolvm-isolation-spike/phase-02 Session-Id: 2e26a154-2c61-42d1-93d2-e6b266001b6c Gate-Log: /Users/remyloubradou/.phax/runs/phax.smolvm-isolation-spike/phase-02/checks-attempt-01.log --- spikes/smolvm/01-filesystem.sh | 148 ++++++++++++++++++++++++ spikes/smolvm/findings/01-filesystem.md | 85 ++++++++++++++ 2 files changed, 233 insertions(+) create mode 100644 spikes/smolvm/01-filesystem.sh create mode 100644 spikes/smolvm/findings/01-filesystem.md diff --git a/spikes/smolvm/01-filesystem.sh b/spikes/smolvm/01-filesystem.sh new file mode 100644 index 00000000..cdbf46e2 --- /dev/null +++ b/spikes/smolvm/01-filesystem.sh @@ -0,0 +1,148 @@ +#!/bin/sh +# 01-filesystem.sh — smolvm filesystem isolation probe +# +# Tests five properties of the microVM filesystem boundary: +# A. Mounted /workspace: sentinel visible from guest, writable, write-back to host +# B. Host $HOME not accessible from inside the guest +# C. Host repo root not accessible from inside the guest +# D. Host /etc content not leaking into guest (hostname / passwd comparison) +# E. Read-only mount: guest write is rejected (:ro suffix per smolvm -v flag) +# +# Run AFTER 00-preflight.sh confirms smolvm is installed. +# Paste the full output into findings/01-filesystem.md ## Results, then fill ## Verdict. +# +# smolvm flag reference (all used below): +# smolvm machine run --image ephemeral VM, cleaned up on exit +# -v HOST:CONTAINER[:ro] mount host dir into guest (optional :ro) +# -- COMMAND ARGS command to execute inside the guest +set -eu + +IMAGE="alpine" +GUEST_WORKSPACE="/workspace" + +PASS=0 +FAIL=0 + +pass() { printf 'PASS: %s\n' "$1"; PASS=$((PASS + 1)); } +fail() { printf 'FAIL: %s\n' "$1"; FAIL=$((FAIL + 1)); } +section() { printf '\n── %s ──\n' "$1"; } + +printf '=== smolvm filesystem isolation probe ===\n' +printf 'Image: %s\n' "$IMAGE" +printf 'Host arch: %s\n' "$(uname -m)" +printf 'Host OS: %s\n' "$(uname -s -r)" + +# Throwaway dir for the workspace mount; cleaned up on exit. +WORK_DIR=$(mktemp -d) +trap 'rm -rf "$WORK_DIR"' EXIT INT TERM + +# ── A. Workspace mount ───────────────────────────────────────────────────── +section "A. Workspace mount (-v HOST:/workspace)" + +SENTINEL_VAL="spike-sentinel-$$" +printf '%s\n' "$SENTINEL_VAL" > "$WORK_DIR/sentinel.txt" +printf 'Sentinel written to host: %s/sentinel.txt\n' "$WORK_DIR" + +smolvm machine run --image "$IMAGE" \ + -v "$WORK_DIR:$GUEST_WORKSPACE" \ + -- /bin/sh -c " +set -e +echo '--- guest /workspace contents ---' +ls -la /workspace/ +echo '--- sentinel value ---' +cat /workspace/sentinel.txt +echo '--- writing writeback.txt from guest ---' +printf 'write-back-ok\n' > /workspace/writeback.txt +echo 'write succeeded' +" + +if [ -f "$WORK_DIR/writeback.txt" ] && grep -q 'write-back-ok' "$WORK_DIR/writeback.txt"; then + pass "write-back visible on host after guest write" +else + fail "write-back NOT visible on host — guest writes may not propagate" +fi + +# ── B. Host HOME invisible ───────────────────────────────────────────────── +section "B. Host HOME invisible" + +HOST_HOME="$HOME" +HOST_HOME_MARKER=".smolvm-probe-$$" +printf 'host-home-probe\n' > "$HOST_HOME/$HOST_HOME_MARKER" +printf 'Probing host path from guest: %s\n' "$HOST_HOME/$HOST_HOME_MARKER" + +smolvm machine run --image "$IMAGE" \ + -v "$WORK_DIR:$GUEST_WORKSPACE" \ + -- /bin/sh -c " +echo '--- ls of host HOME path: $HOST_HOME ---' +ls '$HOST_HOME' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +echo '--- cat of unique marker file ---' +cat '$HOST_HOME/$HOST_HOME_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +" + +rm -f "$HOST_HOME/$HOST_HOME_MARKER" + +# ── C. Host repo root invisible ──────────────────────────────────────────── +section "C. Host repo root invisible" + +HOST_REPO_ROOT="$(pwd)" +REPO_MARKER=".smolvm-probe-$$" +printf 'repo-probe\n' > "$HOST_REPO_ROOT/$REPO_MARKER" +printf 'Probing host path from guest: %s\n' "$HOST_REPO_ROOT/$REPO_MARKER" + +smolvm machine run --image "$IMAGE" \ + -v "$WORK_DIR:$GUEST_WORKSPACE" \ + -- /bin/sh -c " +echo '--- ls of host repo root: $HOST_REPO_ROOT ---' +ls '$HOST_REPO_ROOT' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +echo '--- cat of unique marker file ---' +cat '$HOST_REPO_ROOT/$REPO_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +" + +rm -f "$HOST_REPO_ROOT/$REPO_MARKER" + +# ── D. Host /etc not leaking into guest ─────────────────────────────────── +section "D. Host /etc not leaking into guest" + +HOST_HOSTNAME=$(hostname) +printf 'Host hostname (for comparison): %s\n' "$HOST_HOSTNAME" + +smolvm machine run --image "$IMAGE" \ + -v "$WORK_DIR:$GUEST_WORKSPACE" \ + -- /bin/sh -c " +echo '--- guest hostname ---' +hostname 2>/dev/null || cat /etc/hostname 2>/dev/null || echo '(hostname unavailable)' +echo '--- guest /etc/passwd (first 3 lines) ---' +head -3 /etc/passwd 2>/dev/null || echo '(no /etc/passwd)' +echo '--- guest /etc/hosts ---' +cat /etc/hosts 2>/dev/null || echo '(no /etc/hosts)' +echo '--- check for macOS /Users path in guest ---' +ls /Users 2>&1 || echo '/Users not present (expected on Linux guest)' +" + +# ── E. Read-only mount ───────────────────────────────────────────────────── +section "E. Read-only mount (-v HOST:/workspace:ro)" + +printf 'smolvm flag: -v %s:%s:ro\n' "$WORK_DIR" "$GUEST_WORKSPACE" +printf '(smolvm -v flag supports optional :ro suffix per CLI help)\n' + +smolvm machine run --image "$IMAGE" \ + -v "$WORK_DIR:$GUEST_WORKSPACE:ro" \ + -- /bin/sh -c " +echo '--- /workspace contents (read-only mount) ---' +ls -la /workspace/ +echo '--- attempting write to /workspace/ro-test.txt ---' +printf 'should-fail\n' > /workspace/ro-test.txt \ + && echo 'WRITE SUCCEEDED (unexpected — :ro not enforced)' \ + || echo 'WRITE REJECTED (expected — :ro is enforced)' +" + +if [ -f "$WORK_DIR/ro-test.txt" ]; then + fail "read-only mount: guest write propagated to host — :ro not enforced" +else + pass "read-only mount: no write-through to host" +fi + +# ── Summary ──────────────────────────────────────────────────────────────── +printf '\n=== Probe complete: %d PASS, %d FAIL ===\n' "$PASS" "$FAIL" +printf 'Paste this output into findings/01-filesystem.md ## Results\n' +printf 'then fill in ## Verdict.\n' diff --git a/spikes/smolvm/findings/01-filesystem.md b/spikes/smolvm/findings/01-filesystem.md new file mode 100644 index 00000000..195e585e --- /dev/null +++ b/spikes/smolvm/findings/01-filesystem.md @@ -0,0 +1,85 @@ +# Findings: Filesystem isolation probe + +## Environment + +- smolvm version: +- Host OS / arch: +- Guest arch: arm64 Linux (Alpine, via libkrun on Apple Silicon) +- Date of run: +- Operator: + +## Procedure + +Run `sh spikes/smolvm/01-filesystem.sh` from the repo root on a host with `smolvm` +installed and the `alpine` image available. The script uses `smolvm machine run` (ephemeral +mode — VM is created, command runs, everything is torn down) and exercises five checks: + +**A. Workspace mount** (`-v HOST:/workspace`) + +A throwaway directory is created on the host with a `sentinel.txt` file. The guest reads +the sentinel from `/workspace/sentinel.txt` (confirms mount is active) and writes +`writeback.txt`. After the VM exits, the script checks whether `writeback.txt` appeared on +the host side — confirming bidirectional file propagation. + +Flag syntax: `smolvm machine run --image alpine -v "$WORK_DIR:/workspace" -- /bin/sh -c "..."` + +**B. Host `$HOME` invisible** + +A unique marker file is written to `$HOME` immediately before the VM boots. Inside the +guest, the script probes the full host `$HOME` path (e.g., `/Users/remyloubradou`) and +the marker file directly. Both are expected to be absent — the guest's directory tree is +its own Linux rootfs, and macOS-specific paths like `/Users/` do not exist in Alpine. + +**C. Host repo root invisible** + +Same approach as B: a unique marker is written to `$PWD` (the repo root) on the host; +the guest probes the exact path. Expected to be absent. + +**D. Host `/etc` not leaking** + +The guest's `hostname`, `/etc/passwd` (first 3 lines), `/etc/hosts`, and whether +`/Users/` exists are captured. These are compared against the host hostname (printed +before the VM boots). A Linux Alpine guest will have its own `/etc` populated by the +image; it should not reflect host macOS content. + +**E. Read-only mount** (`-v HOST:/workspace:ro`) + +The mount is repeated with the `:ro` suffix (supported per smolvm `-v` flag documentation: +`-v HOST:CONTAINER[:ro]`). Inside the guest, the script attempts to write a file to +`/workspace/ro-test.txt`. Expected outcome: the write is rejected (EROFS or permission +denied). The host side is also checked: if the file does not appear, the `:ro` is +enforced at the kernel/hypervisor level. + +### Crux question for this probe + +> Can a host directory be mounted **read-only** into the guest, preventing the agent from +> writing back to the host? This is relevant to `isolated` mode: a read-only mount of the +> shared worktree would let an agent read context (specs, existing code) without the risk +> of writing back changes that were not explicitly staged. + +If `:ro` is not supported or not enforced, record that as a finding — it constrains what +`isolated` mode can safely offer. + +## Results + + + +## Verdict + + + +**Status:** (not yet run) + +**Conclusion:** + +## Open questions + +- If `:ro` is not enforced at the guest level, is there another smolvm mechanism (e.g. + a Smolfile volume option) that provides read-only semantics? +- Does `/workspace` in the guest reflect the storage-disk default workspace when no + `-v` flag is present? (smolvm docs note: `-v host:/workspace` replaces the default; + no `-v` means the storage-disk workspace is used — no host content exposed.) +- Are there guest paths other than `/workspace` that could reflect host state (e.g. + `/dev/`, `/proc/`, `/sys/` bind-mounts from the host)? +- What is the performance delta of a workspace mount vs in-guest storage for large + codebases? (Relevant to phax phase execution time.) From 6553b77245400967bff3bab6174e3c975624ac72 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Mon, 29 Jun 2026 15:24:51 +0200 Subject: [PATCH 3/7] test(spike): add smolvm per-domain network allowlist probe Add the decisive network harness: a five-case matrix (allowed-by-name, denied-by-name, denied-by-hardcoded-IP, allowed-by-IP, ICMP) that determines whether smolvm's --allow-host is deny-by-default and a real egress boundary versus DNS-name filtering. Findings doc frames the two crux questions; Results/Verdict filled from a real run out-of-band. --- Run-Id: smolvm-isolation-spike-1782738661332 Short-Name: smolvm-isolation-spike Phase-Id: phase-03 Phase-Title: Network per-domain allowlist probe Model: claude-sonnet-4-6 Effort: medium Worktree: /Users/remyloubradou/.phax/worktrees/phax.smolvm-isolation-spike/phase-03 Session-Id: f5a8c7d9-6a2f-4972-a851-4489a77b16f2 Gate-Log: /Users/remyloubradou/.phax/runs/phax.smolvm-isolation-spike/phase-03/checks-attempt-01.log --- spikes/smolvm/02-network.sh | 211 +++++++++++++++++++++++++++ spikes/smolvm/findings/02-network.md | 114 +++++++++++++++ 2 files changed, 325 insertions(+) create mode 100644 spikes/smolvm/02-network.sh create mode 100644 spikes/smolvm/findings/02-network.md diff --git a/spikes/smolvm/02-network.sh b/spikes/smolvm/02-network.sh new file mode 100644 index 00000000..1e1af9c7 --- /dev/null +++ b/spikes/smolvm/02-network.sh @@ -0,0 +1,211 @@ +#!/bin/sh +# 02-network.sh — smolvm per-domain network allowlist probe +# +# Five-case matrix that determines whether smolvm's --allow-host is: +# (a) deny-by-default egress filtering, and +# (b) a real egress boundary (not just DNS-name filtering). +# +# The DECISIVE test is case 3: if a non-allowed host reached by hard-coded IP +# succeeds, the allowlist is DNS/SNI filtering only, not a true egress boundary. +# +# Run AFTER 00-preflight.sh confirms smolvm is installed. +# Paste the full output into findings/02-network.md ## Results, then fill ## Verdict. +# +# smolvm flag reference (all used below): +# smolvm machine run --image ephemeral VM, cleaned up on exit +# --net enable networking (off by default) +# --allow-host add hostname to egress allowlist +# --allow-cidr add CIDR block to egress allowlist +# -v HOST:CONTAINER[:ro] mount host dir into guest +# -e KEY=VALUE set guest env var +# -- COMMAND ARGS command to execute inside the guest +# +# Smolfile equivalent (for reference — not used here, script uses CLI flags): +# [network] +# net = true +# allow_hosts = ["example.com"] +set -eu + +IMAGE="alpine" + +# The single domain we allow. Pick something lightweight and reliable. +ALLOWED_DOMAIN="example.com" + +# A blocked domain (should be denied by the allowlist). +BLOCKED_DOMAIN="httpbin.org" + +# Resolve the blocked domain's IP on the HOST (before the VM boots, using host DNS). +# This IP is passed into the guest and used to probe case 3. +BLOCKED_IP=$(nslookup "$BLOCKED_DOMAIN" 2>/dev/null \ + | awk '/^Address: / { print $2; exit }' \ + || true) + +# Fallback: use dig if nslookup produces no result. +if [ -z "$BLOCKED_IP" ]; then + BLOCKED_IP=$(dig +short "$BLOCKED_DOMAIN" A 2>/dev/null | head -1 || true) +fi + +if [ -z "$BLOCKED_IP" ]; then + printf 'WARN: could not resolve %s on host; case 3 will use 93.184.216.34 (example.com IP as stand-in)\n' \ + "$BLOCKED_DOMAIN" + BLOCKED_IP="93.184.216.34" +fi + +PASS=0 +FAIL=0 +SKIP=0 + +pass() { printf 'PASS: %s\n' "$1"; PASS=$((PASS + 1)); } +fail() { printf 'FAIL: %s\n' "$1"; FAIL=$((FAIL + 1)); } +skip() { printf 'SKIP: %s\n' "$1"; SKIP=$((SKIP + 1)); } +section() { printf '\n── %s ──\n' "$1"; } + +# Throwaway workspace dir so every VM boot has a writable mount (some images need it). +WORK_DIR=$(mktemp -d) +trap 'rm -rf "$WORK_DIR"' EXIT INT TERM + +printf '=== smolvm per-domain network allowlist probe ===\n' +printf 'Image: %s\n' "$IMAGE" +printf 'Allowed domain: %s\n' "$ALLOWED_DOMAIN" +printf 'Blocked domain: %s\n' "$BLOCKED_DOMAIN" +printf 'Blocked IP: %s (resolved on host before VM boot)\n' "$BLOCKED_IP" +printf 'Host arch: %s\n' "$(uname -m)" +printf 'Host OS: %s\n' "$(uname -s -r)" + +# Helper: run a curl probe inside the guest, capturing exit code. +# Usage: run_probe +# The guest runs: curl --max-time 8 --silent --output /dev/null --write-out "%{http_code}" +# Exit 0 with a real HTTP code = reachable. Exit non-0 or code 000 = blocked/unreachable. + +# ── Case 1: Allowed domain reachable ────────────────────────────────────────── +section "Case 1: Allowed domain reachable (by name)" + +printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" +printf 'Guest command: curl --max-time 8 http://%s/\n' "$ALLOWED_DOMAIN" + +smolvm machine run --image "$IMAGE" \ + --net --allow-host "$ALLOWED_DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -- /bin/sh -c " +apk add --quiet --no-cache curl 2>/dev/null || true +echo '--- curl to allowed domain by name ---' +HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ + --write-out '%{http_code}' http://$ALLOWED_DOMAIN/ 2>/dev/null || echo '000') +echo \"HTTP code: \$HTTP_CODE\" +if [ \"\$HTTP_CODE\" != '000' ] && [ \"\$HTTP_CODE\" != '' ]; then + echo 'RESULT: REACHABLE' +else + echo 'RESULT: UNREACHABLE' +fi +" + +# ── Case 2: Non-allowed domain blocked (by name) ─────────────────────────────── +section "Case 2: Non-allowed domain blocked (by name)" + +printf 'Flag: --net --allow-host %s (NO allowance for %s)\n' "$ALLOWED_DOMAIN" "$BLOCKED_DOMAIN" +printf 'Guest command: curl --max-time 8 http://%s/\n' "$BLOCKED_DOMAIN" + +smolvm machine run --image "$IMAGE" \ + --net --allow-host "$ALLOWED_DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -- /bin/sh -c " +apk add --quiet --no-cache curl 2>/dev/null || true +echo '--- curl to blocked domain by name ---' +HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ + --write-out '%{http_code}' http://$BLOCKED_DOMAIN/ 2>/dev/null || echo '000') +echo \"HTTP code: \$HTTP_CODE\" +if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then + echo 'RESULT: BLOCKED (expected)' +else + echo 'RESULT: REACHABLE (unexpected — domain filtering may not be active)' +fi +" + +# ── Case 3: Non-allowed host by hard-coded IP (THE DECISIVE TEST) ───────────── +section "Case 3: Non-allowed host by hard-coded IP — DECISIVE" + +printf '*** This is the go/no-go signal for the network boundary. ***\n' +printf 'If this PASSES (request succeeds), --allow-host is DNS-name filtering only,\n' +printf 'NOT a true egress boundary. An attacker can bypass it by avoiding DNS lookup.\n\n' +printf 'Blocked domain %s resolved to %s on host.\n' "$BLOCKED_DOMAIN" "$BLOCKED_IP" +printf 'Flag: --net --allow-host %s (no allowance for IP %s)\n' "$ALLOWED_DOMAIN" "$BLOCKED_IP" +printf 'Guest command: curl --max-time 8 http://%s/\n' "$BLOCKED_IP" + +smolvm machine run --image "$IMAGE" \ + --net --allow-host "$ALLOWED_DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -- /bin/sh -c " +apk add --quiet --no-cache curl 2>/dev/null || true +echo '--- curl to blocked domain by raw IP ---' +HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ + --write-out '%{http_code}' http://$BLOCKED_IP/ 2>/dev/null || echo '000') +echo \"HTTP code: \$HTTP_CODE\" +if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then + echo 'RESULT: BLOCKED (expected — real egress boundary)' +else + echo 'RESULT: REACHABLE (DNS-name filtering only — not a security boundary)' +fi +" + +# ── Case 4: Allowed host by raw IP ──────────────────────────────────────────── +section "Case 4: Allowed host by raw IP" + +# Resolve the allowed domain's IP on the host. +ALLOWED_IP=$(nslookup "$ALLOWED_DOMAIN" 2>/dev/null \ + | awk '/^Address: / { print $2; exit }' \ + || true) +if [ -z "$ALLOWED_IP" ]; then + ALLOWED_IP=$(dig +short "$ALLOWED_DOMAIN" A 2>/dev/null | head -1 || true) +fi + +if [ -z "$ALLOWED_IP" ]; then + printf 'WARN: could not resolve %s on host; skipping case 4\n' "$ALLOWED_DOMAIN" + SKIP=$((SKIP + 1)) +else + printf 'Allowed domain %s resolved to %s on host.\n' "$ALLOWED_DOMAIN" "$ALLOWED_IP" + printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" + printf 'Guest command: curl --max-time 8 http://%s/\n' "$ALLOWED_IP" + printf '(Reveals whether enforcement is SNI/DNS-based: if IP is blocked,\n' + printf ' allowlist uses hostname comparison; if IP is allowed, it may bypass SNI.)\n' + + smolvm machine run --image "$IMAGE" \ + --net --allow-host "$ALLOWED_DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -- /bin/sh -c " +apk add --quiet --no-cache curl 2>/dev/null || true +echo '--- curl to allowed domain by its raw IP ---' +HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ + --write-out '%{http_code}' http://$ALLOWED_IP/ 2>/dev/null || echo '000') +echo \"HTTP code: \$HTTP_CODE\" +if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then + echo 'RESULT: BLOCKED (enforcement may be SNI/hostname-based, not IP-based)' +else + echo 'RESULT: REACHABLE (enforcement may pass traffic to IPs of allowed hosts)' +fi +" +fi + +# ── Case 5: ICMP (ping) — smolvm docs say TCP/UDP only ─────────────────────── +section "Case 5: ICMP — smolvm allows TCP/UDP only (no ICMP)" + +printf 'smolvm documentation: TCP and UDP are forwarded; ICMP is not supported.\n' +printf 'Attempting ping inside guest to record actual behaviour.\n' +printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" +printf 'Guest command: ping -c 2 -W 5 %s\n' "$ALLOWED_DOMAIN" + +smolvm machine run --image "$IMAGE" \ + --net --allow-host "$ALLOWED_DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -- /bin/sh -c " +echo '--- ping to allowed domain (ICMP, expect failure) ---' +ping -c 2 -W 5 $ALLOWED_DOMAIN 2>&1 && echo 'RESULT: ICMP SUCCEEDED (unexpected)' \ + || echo 'RESULT: ICMP BLOCKED or UNSUPPORTED (expected)' +" + +# ── Summary ──────────────────────────────────────────────────────────────────── +printf '\n=== Probe complete: %d PASS, %d FAIL, %d SKIP ===\n' "$PASS" "$FAIL" "$SKIP" +printf '\nKey question for ## Verdict:\n' +printf ' 1. Is egress deny-by-default? (Did case 2 block the non-allowed domain?)\n' +printf ' 2. Is --allow-host a security boundary? (Did case 3 block the raw IP?)\n' +printf '\nPaste this output into findings/02-network.md ## Results\n' +printf 'then fill in ## Verdict.\n' diff --git a/spikes/smolvm/findings/02-network.md b/spikes/smolvm/findings/02-network.md new file mode 100644 index 00000000..40201641 --- /dev/null +++ b/spikes/smolvm/findings/02-network.md @@ -0,0 +1,114 @@ +# Findings: Per-domain network allowlist probe + +## Environment + +- smolvm version: +- Host OS / arch: +- Guest arch: arm64 Linux (Alpine, via libkrun on Apple Silicon) +- Date of run: +- Operator: + +## Procedure + +Run `sh spikes/smolvm/02-network.sh` from the repo root on a host with `smolvm` installed +and the `alpine` image available. The script uses `smolvm machine run` (ephemeral mode) with +`--net --allow-host example.com` and exercises a five-case matrix to determine whether +`--allow-host` is a real egress boundary or DNS-name filtering only. + +> **Note:** `--net` must be given explicitly — network is off by default in smolvm. + +**Setup** + +The script resolves `httpbin.org` on the host (before any VM boots) to capture its IP +for case 3. The resolved IP is printed at startup. + +**Case 1: Allowed domain reachable (by name)** + +Flags: `--net --allow-host example.com` +Guest command: `curl --max-time 8 http://example.com/` + +Expected: HTTP response received (code ≠ `000`). Confirms that `--allow-host` opens +egress to the listed domain and that networking is functional. + +**Case 2: Non-allowed domain blocked (by name)** + +Flags: `--net --allow-host example.com` (no allowance for `httpbin.org`) +Guest command: `curl --max-time 8 http://httpbin.org/` + +Expected: curl exits non-zero or returns code `000` (connection refused / timeout). +Confirms deny-by-default egress: only explicitly allowed hosts can be reached. + +**Case 3: Non-allowed host by hard-coded IP — THE DECISIVE TEST** + +Flags: `--net --allow-host example.com` (no allowance for `httpbin.org`'s IP) +Guest command: `curl --max-time 8 http:///` + +The IP is the one resolved by the host before the VM boots and injected via shell +variable. **This is the go/no-go signal for the network boundary:** + +- If the request is **blocked**: `--allow-host` filters at the egress/transport layer, + not just DNS lookup. This is a real security boundary. +- If the request **succeeds**: `--allow-host` is DNS-name filtering only. An agent (or + attacker) that hard-codes an IP bypasses the allowlist entirely. This would disqualify + smolvm as a security boundary for phax's `isolated` mode. + +**Case 4: Allowed host by raw IP** + +Flags: `--net --allow-host example.com` +Guest command: `curl --max-time 8 http:///` + +Reveals the enforcement mechanism: + +- **Blocked** → enforcement is SNI/hostname-based (allowlist checked against the + domain name, not the destination IP). +- **Reachable** → enforcement may pass traffic to IPs of allowed hosts, which could + enable bypass if two domains share an IP. + +**Case 5: ICMP — TCP/UDP only per smolvm docs** + +Flags: `--net --allow-host example.com` +Guest command: `ping -c 2 -W 5 example.com` + +smolvm documentation states only TCP and UDP are forwarded; ICMP is not supported. +This case records the actual observed behaviour for completeness. + +### Crux questions for this probe + +The `## Verdict` must answer both: + +1. **Is egress deny-by-default?** (Did case 2 block the non-allowed domain, or was the + request allowed through?) +2. **Is `--allow-host` a security boundary or DNS convenience?** (Did case 3 block the + hard-coded IP, or did the request succeed — meaning the allowlist can be bypassed?) + +## Results + + + +## Verdict + + + +**Status:** (not yet run) + +**Is egress deny-by-default?** + +**Is `--allow-host` a security boundary or DNS convenience?** + +**Conclusion:** + +## Open questions + +- If case 3 (raw IP) is blocked: what mechanism enforces it? DNS-interception, packet + filtering (iptables/nftables inside the guest), or libkrun VMM-level egress proxy? + Understanding the mechanism helps assess bypass surface. +- If case 3 (raw IP) succeeds: is there a `--allow-cidr` or Smolfile option that + enforces a true egress boundary? Is that usable without knowing provider IPs upfront? +- Does the allowlist interact with HTTPS/TLS differently than HTTP? An SNI-based filter + could pass TLS traffic whose SNI matches the allowed host but whose TCP destination + resolves to a different IP. +- If case 4 (allowed host by IP) is blocked: does the guest need the allowed domain + to be resolvable via guest DNS, or does smolvm inject a virtual DNS record for it? +- What is the timeout/failure mode when an agent inside the guest tries to reach a + blocked domain? Immediate connection refused vs. multi-second timeout matters for + phase-04 (denied-egress UX). From 2daf10bfdb48ff9737e9e495887d6be0964d7523 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Mon, 29 Jun 2026 15:44:02 +0200 Subject: [PATCH 4/7] test(spike): add smolvm in-guest agent execution probe Add a parameterized harness that runs Claude Code, Codex, and Mistral Vibe inside the libkrun Linux guest with env-injected creds and a per-provider network allowlist, confirming each edits the mounted worktree and observing denied-egress UX. Secrets are read from host env only. Per-provider findings table filled from a real run. --- Run-Id: smolvm-isolation-spike-1782738661332 Short-Name: smolvm-isolation-spike Phase-Id: phase-04 Phase-Title: Agents-in-guest execution probe Model: claude-sonnet-4-6 Effort: medium Worktree: /Users/remyloubradou/.phax/worktrees/phax.smolvm-isolation-spike/phase-04 Session-Id: 7803253c-40ba-480b-97f8-287f8a05a2f6 Gate-Log: /Users/remyloubradou/.phax/runs/phax.smolvm-isolation-spike/phase-04/checks-attempt-01.log --- spikes/smolvm/03-agents.sh | 226 ++++++++++++++++++++++++++++ spikes/smolvm/findings/03-agents.md | 126 ++++++++++++++++ 2 files changed, 352 insertions(+) create mode 100644 spikes/smolvm/03-agents.sh create mode 100644 spikes/smolvm/findings/03-agents.md diff --git a/spikes/smolvm/03-agents.sh b/spikes/smolvm/03-agents.sh new file mode 100644 index 00000000..bb02123b --- /dev/null +++ b/spikes/smolvm/03-agents.sh @@ -0,0 +1,226 @@ +#!/bin/sh +# 03-agents.sh — smolvm in-guest agent execution probe +# +# Confirms three AI provider CLIs (claude, codex, vibe) run inside the libkrun +# Linux guest with env-injected credentials, edit a mounted worktree, and that +# denied egress (no network) produces an observable error rather than a silent +# hang. Two steps per provider: +# +# STEP A — allowed egress: --net --allow-host +# Agent performs a trivial task (create hello.txt); file must appear on host. +# STEP B — denied egress: no --net flag (zero egress to anything) +# Capture exactly how the CLI surfaces a total network loss: +# clear error / timeout + exit / silent hang / retry loop. +# This observation motivates the capability-preamble prompt in phase-05. +# +# Usage: +# sh spikes/smolvm/03-agents.sh # probe all three providers +# PROBE_PROVIDER=claude sh spikes/smolvm/03-agents.sh +# PROBE_PROVIDER=codex sh spikes/smolvm/03-agents.sh +# PROBE_PROVIDER=vibe sh spikes/smolvm/03-agents.sh +# +# Required host env vars (read but NEVER printed or echoed): +# ANTHROPIC_API_KEY — Claude Code (claude) +# OPENAI_API_KEY — Codex CLI (codex) +# MISTRAL_API_KEY — Mistral Vibe (vibe) +# +# Run AFTER 00-preflight.sh confirms smolvm is installed. +# Paste full output into findings/03-agents.md ## Results, then fill ## Verdict. +# +# smolvm flag reference (all used below): +# smolvm machine run --image ephemeral VM, cleaned up on exit +# --net enable networking (off by default) +# --allow-host add hostname to egress allowlist +# -v HOST:CONTAINER[:ro] mount host dir into guest +# -e KEY=VALUE inject guest env var from host env +# -- COMMAND ARGS command to run inside the guest +set -eu + +IMAGE="alpine" +TRIVIAL_PROMPT="Create a file called hello.txt in /workspace containing exactly one line: hello from smolvm" + +# Step B timeout: seconds to wait before declaring a hang. +# This prevents the harness from blocking if the CLI loops on retries. +STEP_B_TIMEOUT=60 + +# ── Helpers ──────────────────────────────────────────────────────────────────── +section() { printf '\n══ %s ══\n' "$1"; } +step() { printf '\n── %s ──\n' "$1"; } +note() { printf 'NOTE: %s\n' "$1"; } + +printf '=== smolvm in-guest agent execution probe ===\n' +printf 'Image: %s\n' "$IMAGE" +printf 'Host arch: %s\n' "$(uname -m)" +printf 'Host OS: %s\n' "$(uname -s -r)" +printf 'Probe provider: %s\n' "${PROBE_PROVIDER:-all}" +printf 'Step B timeout: %ds\n' "$STEP_B_TIMEOUT" + +# Throwaway workspace dir shared across probes; each step creates its own subdir. +BASE_WORK_DIR=$(mktemp -d) +trap 'rm -rf "$BASE_WORK_DIR"' EXIT INT TERM + +# ── probe_provider ───────────────────────────────────────────────────────────── +# Arguments: +# $1 PNAME short name (claude | codex | vibe) +# $2 DOMAIN provider API hostname to allowlist +# $3 KEY_VAR name of the host env var holding the API key +# $4 INSTALL shell snippet run in guest to install the CLI (may be slow) +# $5 RUN_A shell snippet that runs the trivial task (non-interactive) +# Must create /workspace/hello.txt. Receives the key as an env var. +probe_provider() { + PNAME="$1" + DOMAIN="$2" + KEY_VAR="$3" + INSTALL="$4" + RUN_A="$5" + + section "Provider: $PNAME (api domain: $DOMAIN)" + + # -- key check ---------------------------------------------------------------- + eval "KEY_VALUE=\${${KEY_VAR}:-}" + if [ -z "$KEY_VALUE" ]; then + printf 'SKIP: %s not set in host env — cannot test %s\n' "$KEY_VAR" "$PNAME" + return 0 + fi + # Confirm key is set but print nothing about its value. + printf 'Key: %s is set (%d chars)\n' "$KEY_VAR" "$(printf '%s' "$KEY_VALUE" | wc -c | tr -d ' ')" + unset KEY_VALUE + + WORK_DIR="$BASE_WORK_DIR/$PNAME" + mkdir -p "$WORK_DIR" + + # ── STEP A: allowed egress + key injected ────────────────────────────────── + step "STEP A — task execution with allowed network ($DOMAIN)" + printf 'Flags: --net --allow-host %s, -e %s=\n' "$DOMAIN" "$KEY_VAR" + printf 'Guest install: %s\n' "$INSTALL" + printf 'Guest task: %s\n' "$TRIVIAL_PROMPT" + + # Evaluate KEY_VAR name to get its value for injection — value never printed. + eval "KEY_FOR_INJECT=\${${KEY_VAR}}" + + smolvm machine run --image "$IMAGE" \ + --net --allow-host "$DOMAIN" \ + -v "$WORK_DIR:/workspace" \ + -e "${KEY_VAR}=${KEY_FOR_INJECT}" \ + -- /bin/sh -c " +set -e +echo '--- installing CLI in guest ---' +$INSTALL +echo '--- CLI version / confirm binary ---' +which $PNAME && $PNAME --version 2>/dev/null || echo '(version flag not supported)' +echo '--- guest arch ---' +uname -m +echo '--- running trivial task ---' +$RUN_A +echo '--- /workspace contents after task ---' +ls -la /workspace/ +if [ -f /workspace/hello.txt ]; then + echo '--- hello.txt content ---' + cat /workspace/hello.txt + echo 'RESULT: TASK COMPLETE' +else + echo 'RESULT: hello.txt NOT CREATED' +fi +" + + if [ -f "$WORK_DIR/hello.txt" ]; then + printf 'HOST-SIDE CHECK: hello.txt visible on host — write-back confirmed\n' + else + printf 'HOST-SIDE CHECK: hello.txt NOT on host — write-back failed or task failed\n' + fi + + unset KEY_FOR_INJECT + + # ── STEP B: zero egress (no --net), observe denial UX ───────────────────── + step "STEP B — denied egress (no --net, ${STEP_B_TIMEOUT}s timeout)" + printf 'Flags: -e %s= (no --net — all egress blocked)\n' "$KEY_VAR" + printf 'Goal: capture how the CLI surfaces network loss (error / hang / retry)\n' + printf 'A timeout here means the CLI is silently retrying or hanging.\n' + + eval "KEY_FOR_INJECT=\${${KEY_VAR}}" + + # Wrap the VM boot in a timeout so a hanging CLI does not block the harness. + STEP_B_WORK="$BASE_WORK_DIR/${PNAME}-step-b" + mkdir -p "$STEP_B_WORK" + + timeout "$STEP_B_TIMEOUT" \ + smolvm machine run --image "$IMAGE" \ + -v "$STEP_B_WORK:/workspace" \ + -e "${KEY_VAR}=${KEY_FOR_INJECT}" \ + -- /bin/sh -c " +set -e +echo '--- installing CLI in guest (no net: may fail if apk needs internet) ---' +$INSTALL 2>&1 || echo 'INSTALL FAILED (expected without --net)' +echo '--- running task without network ---' +$RUN_A 2>&1 || true +echo 'CLI exited (exit captured above)' +" && printf 'STEP B: VM exited within timeout\n' \ + || printf 'STEP B: VM hit %ds timeout — CLI may be hanging/retrying\n' "$STEP_B_TIMEOUT" + + unset KEY_FOR_INJECT + + printf '\n-- %s probe complete --\n' "$PNAME" +} + +# ── Provider: claude ──────────────────────────────────────────────────────────── +# CLI: claude (from @anthropic-ai/claude-code) +# Domain: api.anthropic.com +# Key var: ANTHROPIC_API_KEY +# Non-interactive: `claude --print --dangerously-skip-permissions -p ""` +# --print non-interactive, outputs result then exits +# --dangerously-skip-permissions bypass all tool-call approval prompts +CLAUDE_INSTALL="apk add --quiet --no-cache nodejs npm 2>/dev/null && npm install --quiet -g @anthropic-ai/claude-code 2>/dev/null" +CLAUDE_RUN="claude --print --dangerously-skip-permissions -p '$TRIVIAL_PROMPT'" + +# ── Provider: codex ───────────────────────────────────────────────────────────── +# CLI: codex (from @openai/codex, the 2025 Codex CLI) +# Domain: api.openai.com +# Key var: OPENAI_API_KEY +# Non-interactive: prompt via stdin, exec subcommand +# codex exec -C /workspace --skip-git-repo-check --json +# The prompt is written to stdin before exec. +# NOTE: `echo "..." | codex exec ...` feeds prompt as stdin. +CODEX_INSTALL="apk add --quiet --no-cache nodejs npm 2>/dev/null && npm install --quiet -g @openai/codex 2>/dev/null" +CODEX_RUN="printf '%s' '$TRIVIAL_PROMPT' | codex exec -C /workspace --skip-git-repo-check --json 2>&1 || true" + +# ── Provider: vibe (Mistral Vibe) ─────────────────────────────────────────────── +# CLI: vibe (Mistral's agent CLI) +# Domain: api.mistral.ai +# Key var: MISTRAL_API_KEY +# Non-interactive: `vibe -p "" --agent auto-approve --workdir /workspace` +# -p prompt text +# --agent auto-approve skip approval prompts (non-interactive agent preset) +# --workdir scope the agent's working directory +# --output streaming JSONL streaming output (matches phax's invocation) +# +# NOTE: verify the npm package name before running. +# Candidates: @mistralai/vibe or mistral-vibe or a direct binary download. +# The phax executable name is `vibe` (src/domain/routing/defaults.ts line ~117). +VIBE_INSTALL="apk add --quiet --no-cache nodejs npm 2>/dev/null && npm install --quiet -g @mistralai/vibe 2>/dev/null" +VIBE_RUN="vibe -p '$TRIVIAL_PROMPT' --agent auto-approve --workdir /workspace --output streaming 2>&1 || true" + +# ── Dispatch ──────────────────────────────────────────────────────────────────── +case "${PROBE_PROVIDER:-all}" in + claude) + probe_provider "claude" "api.anthropic.com" "ANTHROPIC_API_KEY" "$CLAUDE_INSTALL" "$CLAUDE_RUN" + ;; + codex) + probe_provider "codex" "api.openai.com" "OPENAI_API_KEY" "$CODEX_INSTALL" "$CODEX_RUN" + ;; + vibe) + probe_provider "vibe" "api.mistral.ai" "MISTRAL_API_KEY" "$VIBE_INSTALL" "$VIBE_RUN" + ;; + all) + probe_provider "claude" "api.anthropic.com" "ANTHROPIC_API_KEY" "$CLAUDE_INSTALL" "$CLAUDE_RUN" + probe_provider "codex" "api.openai.com" "OPENAI_API_KEY" "$CODEX_INSTALL" "$CODEX_RUN" + probe_provider "vibe" "api.mistral.ai" "MISTRAL_API_KEY" "$VIBE_INSTALL" "$VIBE_RUN" + ;; + *) + printf 'ERROR: unknown PROBE_PROVIDER=%s (valid: claude | codex | vibe | all)\n' "$PROBE_PROVIDER" + exit 1 + ;; +esac + +printf '\n=== All requested providers probed ===\n' +printf 'Paste this output into findings/03-agents.md ## Results\n' +printf 'then fill the per-provider table and ## Verdict.\n' diff --git a/spikes/smolvm/findings/03-agents.md b/spikes/smolvm/findings/03-agents.md new file mode 100644 index 00000000..3b7d86de --- /dev/null +++ b/spikes/smolvm/findings/03-agents.md @@ -0,0 +1,126 @@ +# Findings: In-guest agent execution probe + +## Environment + +- smolvm version: +- Host OS / arch: +- Guest OS / arch: Linux arm64 (Alpine, via libkrun on Apple Silicon) +- Date of run: +- Operator: + +## Procedure + +Run `sh spikes/smolvm/03-agents.sh` from the repo root on a host with `smolvm` installed +and the `alpine` image available. Set the relevant API key env vars before running: + +```sh +export ANTHROPIC_API_KEY=... # for claude +export OPENAI_API_KEY=... # for codex +export MISTRAL_API_KEY=... # for vibe +sh spikes/smolvm/03-agents.sh +``` + +Single-provider runs: `PROBE_PROVIDER=claude sh spikes/smolvm/03-agents.sh` + +The script tests each provider in two steps: + +**STEP A — task execution with allowed egress** + +Flags: `--net --allow-host -e =` + +The guest boots, installs the provider CLI via `npm install -g ` (Node.js +from Alpine's apk), then runs the CLI non-interactively on a trivial prompt: + +> "Create a file called hello.txt in /workspace containing exactly one line: hello from smolvm" + +The probe then confirms the file appears on the host side of the `-v` mount (write-back). + +Non-interactive invocation per provider: + +| Provider | CLI | Invocation style | +| -------- | ------ | --------------------------------------------------------------------------------- | +| claude | claude | `claude --print --dangerously-skip-permissions -p ""` | +| codex | codex | `printf '' \| codex exec -C /workspace --skip-git-repo-check --json` | +| vibe | vibe | `vibe -p "" --agent auto-approve --workdir /workspace --output streaming` | + +**STEP B — denied egress (zero network)** + +Flags: `-e =` only — **no `--net` flag** + +The guest boots with no network access at all. The same CLI runs the same task. +The probe captures exactly how the CLI surfaces the failure: clear error and exit, +multi-second timeout then exit, or indefinite hang (caught by the harness's 60s +`timeout` wrapper). + +> This observation directly motivates the capability-preamble prompt in phase-05: +> if the CLI hangs silently, the agent needs to be told up-front that it has no +> network so it can fail fast rather than retry indefinitely. + +**Installation method** + +Each guest is a fresh Alpine minimal container. The probe installs Node.js via +`apk add nodejs npm` and the CLI via `npm install -g `. Recorded package +names (verify before running): + +| Provider | npm package | Executable | +| -------- | ---------------------------- | ---------- | +| claude | `@anthropic-ai/claude-code` | `claude` | +| codex | `@openai/codex` | `codex` | +| vibe | `@mistralai/vibe` _(verify)_ | `vibe` | + +The guest arch is reported by `uname -m` inside the guest. On Apple Silicon (arm64 +host) the libkrun guest runs arm64 Linux natively; an x86-64 guest would require +emulation and may affect CLI compatibility. + +**Key handling** + +API keys are read from the host environment and passed via `smolvm -e KEY=VALUE`. +Keys are never echoed or logged; the harness only prints the key name and length. +Do not commit outputs that contain key values. + +## Results + + + +## Verdict + + + +**Status:** (not yet run) + +**Per-provider results table** + +| Provider | CLI installs in guest? | Completes task (hello.txt)? | Write-back to host? | Network allowlist respected (Step A)? | Denial UX (Step B): error / timeout / hang? | +| -------- | ---------------------- | --------------------------- | ------------------- | ------------------------------------- | ------------------------------------------- | +| claude | | | | | | +| codex | | | | | | +| vibe | | | | | | + +**Installation method confirmed** (npm package names, or alternative): + +**Guest arch** (from `uname -m` inside guest): + +**Conclusion:** + +## Open questions + +- Does the Alpine guest have a compatible Node.js version for each CLI? Node version + from `apk` may lag behind the CLI's `engines.node` requirement — in that case, a + pre-built binary or a different base image may be needed. +- What is the correct npm package name for the `vibe` CLI? The harness tries + `@mistralai/vibe`; if that fails, check Mistral AI docs for the public CLI package. +- Does any provider CLI require an interactive browser login in addition to the API + key? If so, record it as a blocker finding — env-key injection alone is insufficient. +- For Codex (Step A), does the prompt-via-stdin approach work or does `codex exec` + require the prompt as a positional argument? Adjust invocation if needed and record. +- Does the denied-egress UX (Step B) depend on whether the CLI has already been + installed (Step A boot already installed it via network)? The harness installs fresh + per step — if `apk`/`npm` itself needs internet, Step B's install will fail cleanly + before the CLI even runs, which is itself a useful finding (agent won't run without + a pre-baked image). +- Is a pre-baked guest image with the CLI pre-installed needed for production use? + If npm install requires network and we only want the provider API domain in the + allowlist, the agent CLI must ship in the image. Record the image-bake requirement. +- What is the exact error/timeout observed in Step B? Timing matters: a 30-second + timeout before exit means the CLI will delay phax runs if the VM loses connectivity + mid-run; an immediate ECONNREFUSED means fast-fail with a clear error message. From 5fa9d4a7b0ca8b671b163be4e81ae1925bd9b7f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Mon, 29 Jun 2026 15:49:29 +0200 Subject: [PATCH 5/7] docs(spike): synthesize smolvm isolation findings and go/no-go Add the synthesis doc: per-probe verdicts (filesystem, per-domain network, agents-in-guest), an explicit go/no-go on smolvm for phax's isolated mode driven by the hard-coded-IP egress result, and a non-implementing integration sketch (infra wrapper adapter, SecurityPolicy reuse, MCP domain declaration, capability-preamble prompt) plus residual risks and the recommended next step. --- Run-Id: smolvm-isolation-spike-1782738661332 Short-Name: smolvm-isolation-spike Phase-Id: phase-05 Phase-Title: Synthesis and go/no-go recommendation Model: claude-opus-4-8 Effort: high Worktree: /Users/remyloubradou/.phax/worktrees/phax.smolvm-isolation-spike/phase-05 Session-Id: 4410fda5-e889-40dd-b5d7-47eadcf5ffb4 Gate-Log: /Users/remyloubradou/.phax/runs/phax.smolvm-isolation-spike/phase-05/checks-attempt-01.log --- docs/spikes/smolvm-isolation-findings.md | 229 +++++++++++++++++++++++ spikes/smolvm/README.md | 6 +- 2 files changed, 233 insertions(+), 2 deletions(-) create mode 100644 docs/spikes/smolvm-isolation-findings.md diff --git a/docs/spikes/smolvm-isolation-findings.md b/docs/spikes/smolvm-isolation-findings.md new file mode 100644 index 00000000..b7e601bd --- /dev/null +++ b/docs/spikes/smolvm-isolation-findings.md @@ -0,0 +1,229 @@ +# smolvm isolation — findings synthesis and go/no-go + +This document synthesizes the three probe findings from the smolvm isolation spike +(`spikes/smolvm/findings/01-filesystem.md`, `02-network.md`, `03-agents.md`) into a +single decision document for phax's reserved `isolated` security mode. + +> **Provisional.** At the time of writing, every probe's `## Results` and `## Verdict` +> section is empty: the real microVM runs are performed out-of-band per the spike's +> execution-model caveat. The go/no-go below is therefore **conditional on the +> hard-coded-IP egress case (`02-network.sh`, case 3) blocking** when a real run is +> performed. The conclusion section restates that conditionality. + +## Per-probe summary + +### Probe 01 — Filesystem isolation (`findings/01-filesystem.md`) + +The harness boots an Alpine guest with `smolvm machine run --image alpine -v +$WORK_DIR:/workspace` and checks five things: (A) the worktree mount round-trips +between host and guest, (B) the host `$HOME` is invisible, (C) the host repo root is +invisible, (D) the guest `/etc` reflects the Alpine image (not the macOS host), and +(E) a `:ro` mount rejects guest writes. + +Expected verdict: **PASS** — the guest is a Linux Alpine rootfs in libkrun, so macOS +host paths like `/Users/` cannot exist except as explicit `-v` mounts. The only +non-obvious result is (E): if smolvm enforces `:ro` at the hypervisor layer, a future +`isolated` mode can mount the worktree read-only and require explicit `allowWrite` +shares. If `:ro` is unenforced, the integration must hand-roll write rejection (e.g. +an overlay) or accept that any mounted path is implicitly writable. + +### Probe 02 — Per-domain network allowlist (`findings/02-network.md`) — the decisive probe + +The harness exercises a five-case matrix with `--net --allow-host example.com`: + +| Case | Test | Decides | +| ---- | ------------------------------------------ | ------------------------------------------------------------ | +| 1 | Allowed domain by name | Egress works at all | +| 2 | Non-allowed domain by name | Deny-by-default for unlisted hosts | +| 3 | **Non-allowed host by hard-coded IP** | **Whether `--allow-host` is a real boundary or DNS filter** | +| 4 | Allowed host by raw IP | Enforcement layer (SNI/DNS vs egress filter) | +| 5 | ICMP | Behavioural detail (smolvm docs: TCP/UDP only) | + +**Case 3 is the go/no-go signal for the entire spike.** If a guest process can reach a +non-allowed host by hard-coding its IP, the allowlist is a DNS convenience rather than +a security boundary — any agent (or attacker code an agent runs) that ships an IP +literal escapes the constraint. In that case, smolvm is not viable as the foundation +for `isolated` mode without an additional egress layer (e.g. a host-side L4 filter or +an enforced HTTP proxy). + +Conversely, if case 3 blocks **and** case 2 blocks, `--allow-host` is a real +deny-by-default egress boundary — the gap that spec-14 identified as unenforceable at +the provider-native layer would finally be closeable. + +### Probe 03 — Agents-in-guest (`findings/03-agents.md`) + +The harness installs the three provider CLIs (`claude`, `codex`, `vibe`) in an +Alpine guest via `apk add nodejs npm && npm install -g `, injects keys via +`smolvm -e KEY=VALUE`, and runs a trivial "create `hello.txt` in `/workspace`" task +in two configurations: **Step A** with `--net --allow-host ` and +**Step B** with no `--net` flag at all (zero egress). + +What the synthesis cares about regardless of the per-provider details: + +1. **In-guest install.** Step B requires the CLI to already be present in the guest + image, because `apk` and `npm install` themselves need internet. The harness + intentionally installs fresh per step so this surfaces as a finding. The + implication for production `isolated` mode is unambiguous: it needs a **pre-baked + guest image** with each provider CLI installed, not a live `npm install`. +2. **Denied-egress UX.** The harness's 60s `timeout` wrapper captures whether each + CLI exits cleanly, times out, or hangs in a retry loop when its API endpoint is + unreachable. Whatever the per-provider outcome, the design implication is the + same: the agent must be **told up-front** what it can and cannot reach, or it will + spend wall-clock retrying behind a wall it can't see (motivating the + capability-preamble prompt below). +3. **Vibe package name.** `@mistralai/vibe` is unverified in the harness — recorded + as an open question. + +## Go / no-go + +**Provisional GO, conditional on probe 02 case 3 blocking.** + +The deciding fact is the hard-coded-IP egress test. The other two probes contribute +constraints (read-only mount support, pre-baked image requirement, denial UX) but no +single-fact disqualifier: + +- If case 3 **blocks** → `--allow-host` is a real egress boundary. Combined with + smolvm's deny-by-default `$HOME`/repo-root isolation (probe 01) and confirmed + in-guest provider execution (probe 03), smolvm is sufficient to back `isolated` + mode. Proceed to a follow-up implementation plan. +- If case 3 **succeeds** (IP literal escapes the allowlist) → smolvm alone is **not** + sufficient. The boundary spec-14 identified is still open. Two recoverable options: + (a) layer a host-side egress filter under smolvm, or (b) require an in-guest HTTP + proxy and refuse arbitrary outbound sockets. Either is a meaningfully larger build + than wrapping `smolvm sandbox run`. Stop here and re-scope. + +No follow-up implementation plan should be written until the conditional resolves. + +## Integration sketch (not for implementation in this spike) + +This is the shape a follow-up plan would take. Nothing here is implemented; the +synthesis only fixes terms so a future plan can be written from it without re-reading +the three probe docs. + +### Adapter location + +A new infrastructure adapter under `src/infra/providers/` (e.g. +`isolatedSandbox.ts`) wraps the existing provider spawn instead of replacing it. The +adapter is selected by the dispatcher when `SecurityPolicy.mode === "isolated"`; for +each provider invocation it: + +1. Builds the smolvm argv from the resolved `SecurityPolicy`: + - `-v :/workspace[:ro]` from `filesystem.allowRead`/`allowWrite`. + - `--net --allow-host ` once per allowed domain (provider API + each + allowed MCP domain — see below). If the resolved policy has no allowed domains, + omit `--net` entirely (Step B in probe 03). + - `-e KEY=VALUE` for each provider credential, read from host env. +2. Replaces the inner argv with the existing provider CLI invocation (the same one + `claudeCode.ts` / `codexCli.ts` / `mistralVibe.ts` already build), but resolved to + the in-guest path (the pre-baked image installs it at a known location). +3. Streams stdout/stderr back through the existing `Shell` port; the host sees one + process (smolvm) and the existing telemetry/session capture continues to work. + +The stub at `src/cli/commands/run.ts:152-158` (currently rejecting `isolated` mode +with a "planned but not available" message) is the entry point that switches from +rejection to dispatching the wrapped adapter. + +### Reuse of `SecurityPolicy` + +No new domain type is needed. The existing `SecurityPolicy` in +`src/domain/security/types.ts` already has: + +- `filesystem.allowRead` / `filesystem.allowWrite` → `-v` mounts (with `:ro` for + read-only entries). +- `network.profile` → `--net` toggle; `provider-only` means only the provider API + domain is allowed. +- `mcp.allow` → see below. +- `agentCommands` → unchanged; the in-guest CLI is one of these. + +The `isolated` literal already exists in `SecurityMode` and `SecurityProfileSchema` +(`src/schemas/securityConfig.ts:4`). Removing the stub in `run.ts` and dispatching to +the new adapter is the entire surface-area change at the boundary. + +### `secure` vs `isolated` are complementary, not exclusive + +`secure` ("agent knows its limits", provider-native) and `isolated` (microVM +enclosure) are layered. `isolated` mode should still spawn each provider CLI with the +same provider-native flags `secure` uses — defence in depth, not either/or. The VM is +the boundary the agent **cannot** see past; the provider-native flags keep the agent +behaving reasonably inside it. + +### MCP domain declaration + +The VM cannot infer an MCP server's outbound hosts. The follow-up plan must extend +the existing `mcp.allow` config so each entry declares the domains it talks to. The +proposed shape is to widen `McpConfigSchema.allow` from `Array` to +`Array<{ name: string; domains: ReadonlyArray }>` (or a parallel +`mcp.domains` map keyed by `name`, if back-compat with existing configs matters — +schemas in this repo deliberately do not carry back-compat shims, so the wider shape +is preferable). At resolution time, the union of declared domains is added to +`--allow-host`. + +If an MCP server in the user's config has no declared domains, `isolated` mode +**refuses to start the run** rather than guess. Silent allow-everything would defeat +the boundary. + +### Capability-preamble prompt + +Motivated directly by probe 03's denied-egress UX: when the agent has no visibility +into what it can and cannot reach, it wastes wall-clock retrying or hangs. The +adapter must inject — once, at the top of the agent's prompt — a deterministic +preamble describing the sandbox: + +> You are running inside a microVM sandbox. You can read and write `/workspace` +> (which is your worktree on the host). You cannot see any other host paths. You can +> make network requests to the following domains only: ``. Any other network +> request will fail. Do not attempt to install packages or reach other hosts; report +> the limitation and proceed with what is available. + +The preamble is generated from the same `SecurityPolicy` the adapter uses to build +the smolvm argv, so the prompt and the actual sandbox cannot drift. + +## Residual risks + +1. **Case 3 may not block.** The entire go/no-go hinges on it. If the real run shows + `--allow-host` is DNS-name only, this synthesis must be rewritten as a no-go with + a re-scope. +2. **SNI/DNS bypass surface.** Even if case 3 blocks, if enforcement is SNI-based an + agent that disables SNI or speaks a non-TLS protocol over the allowed IP could + bypass the boundary. Case 4 in probe 02 surfaces this; a follow-up plan must + audit it. +3. **Volume-mount maturity.** Probe 01 case E (`:ro` enforcement) decides whether the + worktree can be mounted read-only. If unsupported, every mount is implicitly + writable and `allowRead` cannot be distinguished from `allowWrite` at the VM + boundary. +4. **Cross-arch guest.** Apple Silicon hosts run an arm64 Linux guest; some provider + CLIs may not ship arm64 Linux binaries or compatible Node engine versions. A + pre-baked image is required regardless of architecture; the image-bake recipe + must be cross-arch. +5. **Boot/perf cost.** Each phase boots a microVM. The spike does not measure this; + if boot adds multiple seconds per phase, large multi-phase runs may be noticeably + slower than `secure` mode. Acceptable trade-off, but worth measuring before + release. +6. **Credential handling.** Keys pass via `-e KEY=VALUE`. Logs and telemetry must + never echo the values; the existing `agentErrorLog` and session writers in + `src/infra/providers/` must be audited before `isolated` ships. +7. **Pre-baked image distribution.** `isolated` mode cannot run if the user has no + image with the provider CLIs installed. The follow-up plan must decide whether + phax ships an image, builds one on first run, or requires the user to bake one. + +## Recommended next step + +If probe 02 case 3 **blocks** on the real run: + +1. Fill the `## Results` and `## Verdict` of all three probe docs and remove the + "provisional" marker at the top of this document. +2. Write a follow-up implementation plan (`plan.md`) covering: the new + `isolatedSandbox` adapter in `src/infra/providers/`, the dispatcher switch on + `SecurityPolicy.mode === "isolated"`, the widened `mcp.allow` schema, the + capability-preamble prompt builder, the pre-baked guest image recipe, and an e2e + test under `pnpm test:e2e:real` that boots a real VM. +3. Remove the stub rejection in `src/cli/commands/run.ts:152-158` only at the end of + the implementation plan, gated on all of the above. + +If probe 02 case 3 **succeeds** (IP literal reaches a non-allowed host): + +1. Stop the spike. Update this document's verdict to no-go. +2. Open a follow-up scoping question: is a host-side L4 egress filter under smolvm + acceptable, or should `isolated` mode be re-scoped to a different sandbox + technology entirely? No code under `src/` should change until that question is + resolved. diff --git a/spikes/smolvm/README.md b/spikes/smolvm/README.md index fb6be58b..caf29b03 100644 --- a/spikes/smolvm/README.md +++ b/spikes/smolvm/README.md @@ -80,5 +80,7 @@ findings doc and fill in the `## Verdict`. ## Synthesis -Once all three probes have been run, the synthesis and go/no-go recommendation is in -`docs/spikes/smolvm-isolation-findings.md`. +The synthesis, per-probe verdicts, and go/no-go recommendation are in +[`docs/spikes/smolvm-isolation-findings.md`](../../docs/spikes/smolvm-isolation-findings.md). +The conclusion is **provisional**: it is conditional on the hard-coded-IP case in +`02-network.sh` (case 3) blocking on the real run. From 2b0c4c13d6e02cd59c2325107ac8271c5fcecfa8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Tue, 30 Jun 2026 16:55:59 +0200 Subject: [PATCH 6/7] docs(spike): record real smolvm probe results and resolve go/no-go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the filesystem and network probes on smolvm 1.3.2 (macOS/arm64) and filled in the findings docs and synthesis with actual results: - Probe 01 (filesystem): PASS — clean guest, :ro enforced at hypervisor layer. - Probe 02 (network): PASS — case 3 (raw IP to non-allowed host) is refused at L4, so --allow-host is a real IP-based egress boundary, not DNS filtering. GO. - Probe 03 (agents): BLOCKED — no API keys; recorded the infrastructure facts. Synthesis go/no-go is now a confirmed GO on the security question. Folds in two run findings: scripts don't run verbatim (pull/apk blocked by the allowlist), and a pre-baked image + --allow-host do not compose in smolvm 1.3.2 (new risk 8). Co-Authored-By: Claude Opus 4.8 --- docs/spikes/smolvm-isolation-findings.md | 180 +++++++++++++++-------- spikes/smolvm/findings/01-filesystem.md | 72 +++++++-- spikes/smolvm/findings/02-network.md | 69 +++++++-- spikes/smolvm/findings/03-agents.md | 85 ++++++++--- 4 files changed, 309 insertions(+), 97 deletions(-) diff --git a/docs/spikes/smolvm-isolation-findings.md b/docs/spikes/smolvm-isolation-findings.md index b7e601bd..e39ee776 100644 --- a/docs/spikes/smolvm-isolation-findings.md +++ b/docs/spikes/smolvm-isolation-findings.md @@ -4,11 +4,15 @@ This document synthesizes the three probe findings from the smolvm isolation spi (`spikes/smolvm/findings/01-filesystem.md`, `02-network.md`, `03-agents.md`) into a single decision document for phax's reserved `isolated` security mode. -> **Provisional.** At the time of writing, every probe's `## Results` and `## Verdict` -> section is empty: the real microVM runs are performed out-of-band per the spike's -> execution-model caveat. The go/no-go below is therefore **conditional on the -> hard-coded-IP egress case (`02-network.sh`, case 3) blocking** when a real run is -> performed. The conclusion section restates that conditionality. +> **Resolved (2026-06-30).** The probes have now been run on smolvm 1.3.2 (macOS / +> arm64). Probe 01 (filesystem) is **PASS**; probe 02 (network) is **PASS** — the decisive +> hard-coded-IP egress case (`02-network.sh`, case 3) **blocks** (`Connection refused`). +> The go/no-go below is therefore no longer conditional: it is a **GO** on the security +> question. Probe 03 (agents) could not be run — no API keys were available — so the +> provider-execution questions remain open; see its findings doc. Two execution facts +> surfaced during the runs that change the *integration shape* (not the verdict) and are +> folded in below: the probe scripts could not run verbatim, and **a pre-baked image and +> `--allow-host` do not compose in smolvm 1.3.2**. ## Per-probe summary @@ -20,12 +24,16 @@ between host and guest, (B) the host `$HOME` is invisible, (C) the host repo roo invisible, (D) the guest `/etc` reflects the Alpine image (not the macOS host), and (E) a `:ro` mount rejects guest writes. -Expected verdict: **PASS** — the guest is a Linux Alpine rootfs in libkrun, so macOS -host paths like `/Users/` cannot exist except as explicit `-v` mounts. The only -non-obvious result is (E): if smolvm enforces `:ro` at the hypervisor layer, a future -`isolated` mode can mount the worktree read-only and require explicit `allowWrite` -shares. If `:ro` is unenforced, the integration must hand-roll write rejection (e.g. -an overlay) or accept that any mounted path is implicitly writable. +Verdict: **PASS (confirmed by run).** The guest is a Linux Alpine rootfs in libkrun, so +macOS host paths like `/Users/` simply do not exist except as explicit `-v` mounts — +`$HOME`, the repo root, and `/Users` were all absent from the guest. The key result is +(E): **smolvm enforces `:ro` at the hypervisor layer** (`Read-only file system` on a guest +write attempt, no write-through to host), so a future `isolated` mode can mount the +worktree read-only and require explicit `allowWrite` shares — `allowRead` and `allowWrite` +are distinguishable at the VM boundary. (One operational caveat from the run: an ephemeral +`--image` boot re-pulls every time and the pull needs network, so a pre-baked image / +`--from` artifact is required to boot — `01-filesystem.sh` as written omits `--net` and +cannot pull. See findings doc.) ### Probe 02 — Per-domain network allowlist (`findings/02-network.md`) — the decisive probe @@ -50,6 +58,16 @@ Conversely, if case 3 blocks **and** case 2 blocks, `--allow-host` is a real deny-by-default egress boundary — the gap that spec-14 identified as unenforceable at the provider-native layer would finally be closeable. +**Run result (2026-06-30): both block.** Case 2 (non-allowed host by name) fails to even +resolve, and case 3 (non-allowed host by raw IP) is refused at L4 (`Connection refused`). +`--allow-host` resolves the allowed hostname to its IP(s) at VM start and permits egress to +those IPs only — it is a **real egress boundary**, not DNS-name filtering. Enforcement is +**IP-based**: case 4 (allowed host by its raw IP) connects (app-layer 403 from Cloudflare), +which confirms the mechanism and exposes the shared-CDN-IP caveat now folded into the risks +below. ICMP is not forwarded (case 5). The scripted probe could not be run verbatim — the +`apk add curl` step and the image pull are both blocked by the allowlist; busybox `wget` +and an extended allowlist (Docker registry hosts) were used instead. See findings doc. + ### Probe 03 — Agents-in-guest (`findings/03-agents.md`) The harness installs the three provider CLIs (`claude`, `codex`, `vibe`) in an @@ -71,28 +89,42 @@ What the synthesis cares about regardless of the per-provider details: same: the agent must be **told up-front** what it can and cannot reach, or it will spend wall-clock retrying behind a wall it can't see (motivating the capability-preamble prompt below). -3. **Vibe package name.** `@mistralai/vibe` is unverified in the harness — recorded - as an open question. +3. **Vibe package name.** `@mistralai/vibe` **does not exist on npm** (verified: it, + `mistral-vibe`, and `@mistralai/vibe-cli` all 404). The real Vibe CLI distribution + must be identified before this provider can be probed. + +**Run status (2026-06-30): NOT RUN — no API keys available** (`ANTHROPIC_API_KEY`, +`OPENAI_API_KEY`, `MISTRAL_API_KEY` all unset). The provider-task questions (install, task +completion, write-back, denial UX) are unanswered. The infrastructure facts above were, +however, confirmed in passing — and one of them (point 1, that live in-guest install is +blocked by the allowlist) is now hard fact, not hypothesis: `--allow-host` restricts the +guest DNS resolver, so package mirrors return `no such host`. A pre-baked image carrying the +provider CLIs is therefore **mandatory**, not optional. ## Go / no-go -**Provisional GO, conditional on probe 02 case 3 blocking.** +**GO on the security question** (confirmed by run), with one integration constraint to +design around and one probe still open. + +The deciding fact — the hard-coded-IP egress test (probe 02 case 3) — **blocks**. Combined +with smolvm's deny-by-default `$HOME`/repo-root isolation and hypervisor-enforced `:ro` +mounts (probe 01, PASS), smolvm provides a real filesystem **and** egress boundary: +`--allow-host` cannot be bypassed by shipping an IP literal. smolvm is therefore sufficient +to back `isolated` mode at the boundary level. The earlier no-go branch (case 3 succeeds → +need a host-side L4 filter or in-guest proxy) **does not apply**. -The deciding fact is the hard-coded-IP egress test. The other two probes contribute -constraints (read-only mount support, pre-baked image requirement, denial UX) but no -single-fact disqualifier: +Two things temper the GO without reversing it: -- If case 3 **blocks** → `--allow-host` is a real egress boundary. Combined with - smolvm's deny-by-default `$HOME`/repo-root isolation (probe 01) and confirmed - in-guest provider execution (probe 03), smolvm is sufficient to back `isolated` - mode. Proceed to a follow-up implementation plan. -- If case 3 **succeeds** (IP literal escapes the allowlist) → smolvm alone is **not** - sufficient. The boundary spec-14 identified is still open. Two recoverable options: - (a) layer a host-side egress filter under smolvm, or (b) require an in-guest HTTP - proxy and refuse arbitrary outbound sockets. Either is a meaningfully larger build - than wrapping `smolvm sandbox run`. Stop here and re-scope. +- **Integration constraint (new).** A pre-baked image and `--allow-host` do not compose in + smolvm 1.3.2 — `machine run --from --allow-host …` is rejected, while a live + `--image` boot pulls under the allowlist (forcing the Docker registry hosts onto it). The + follow-up plan must account for this; see the integration sketch and residual risk 8. +- **Probe 03 open.** Provider CLIs were not executed in-guest (no keys). Write-back and the + `:ro` boundary are proven (probe 01), but per-provider install/task/denial-UX behaviour is + unverified. This is a gap to close in the follow-up plan, not a disqualifier. -No follow-up implementation plan should be written until the conditional resolves. +A follow-up implementation plan may now be written, with probe 03 re-run as one of its early +validation steps. ## Integration sketch (not for implementation in this spike) @@ -119,9 +151,23 @@ each provider invocation it: 3. Streams stdout/stderr back through the existing `Shell` port; the host sees one process (smolvm) and the existing telemetry/session capture continues to work. -The stub at `src/cli/commands/run.ts:152-158` (currently rejecting `isolated` mode -with a "planned but not available" message) is the entry point that switches from -rejection to dispatching the wrapped adapter. +**Image acquisition vs. the allowlist (constraint surfaced by the run).** The probes +proved that in smolvm 1.3.2 the image pull obeys the run's egress allowlist, and the +pull-free path (`--from `) **rejects `--allow-host`**. So the adapter cannot +"pre-bake an image and add `--allow-host`". The two workable shapes are: + +1. **Ephemeral `--image` boot, registry hosts on the allowlist.** Every boot allowlists the + provider/MCP domains *plus* the Docker registry + CDN hosts (`index.docker.io`, + `auth.docker.io`, `registry-1.docker.io`, `production.cloudfront.docker.com`, …) so the + pull can run. Re-pulls per boot; registry hosts are reachable from the guest for the + pull window. The follow-up plan must decide whether that egress is acceptable. +2. **Offline/pre-pulled image store** (preferred if smolvm supports it, or a newer smolvm + that lifts the `--from` + `--allow-host` restriction): boot from a locally-cached image + with only the provider/MCP domains allowlisted and no registry egress at all. + +This is tracked as residual risk 8. The stub at `src/cli/commands/run.ts:152-158` (currently +rejecting `isolated` mode with a "planned but not available" message) is the entry point +that switches from rejection to dispatching the wrapped adapter. ### Reuse of `SecurityPolicy` @@ -180,17 +226,19 @@ the smolvm argv, so the prompt and the actual sandbox cannot drift. ## Residual risks -1. **Case 3 may not block.** The entire go/no-go hinges on it. If the real run shows - `--allow-host` is DNS-name only, this synthesis must be rewritten as a no-go with - a re-scope. -2. **SNI/DNS bypass surface.** Even if case 3 blocks, if enforcement is SNI-based an - agent that disables SNI or speaks a non-TLS protocol over the allowed IP could - bypass the boundary. Case 4 in probe 02 surfaces this; a follow-up plan must - audit it. -3. **Volume-mount maturity.** Probe 01 case E (`:ro` enforcement) decides whether the - worktree can be mounted read-only. If unsupported, every mount is implicitly - writable and `allowRead` cannot be distinguished from `allowWrite` at the VM - boundary. +1. **~~Case 3 may not block.~~ RESOLVED — it blocks.** The decisive test passed: a raw IP + to a non-allowed host is refused at L4. `--allow-host` is a real egress boundary. No + re-scope needed. +2. **Shared-CDN-IP bypass surface (CONFIRMED).** Enforcement is by IP resolved at VM start + (case 4: the allowed host's raw IP connects). Therefore allowlisting one host on a shared + CDN address transitively permits any other host served from that same IP — example.com's + Cloudflare IP is the worked example. A follow-up plan must treat the allowlist as + IP-granular, not host-granular, and decide whether that precision is acceptable for the + provider/MCP domains in scope (many sit behind shared CDNs). +3. **~~Volume-mount maturity.~~ RESOLVED favourably.** Probe 01 case E confirmed `:ro` is + enforced at the hypervisor layer (guest write → EROFS, no host write-through). The + worktree can be mounted read-only and `allowRead`/`allowWrite` are distinguishable at the + VM boundary. 4. **Cross-arch guest.** Apple Silicon hosts run an arm64 Linux guest; some provider CLIs may not ship arm64 Linux binaries or compatible Node engine versions. A pre-baked image is required regardless of architecture; the image-bake recipe @@ -205,25 +253,39 @@ the smolvm argv, so the prompt and the actual sandbox cannot drift. 7. **Pre-baked image distribution.** `isolated` mode cannot run if the user has no image with the provider CLIs installed. The follow-up plan must decide whether phax ships an image, builds one on first run, or requires the user to bake one. + Confirmed mandatory by the run: live `apk`/`npm` install in-guest is blocked by the + egress allowlist, so the CLIs must already be in the image. +8. **Pre-baked image and `--allow-host` do not compose (smolvm 1.3.2).** `machine run + --from --allow-host …` is rejected, and a live `--image` boot pulls under the + allowlist (requiring the Docker registry hosts on the allowlist). The follow-up plan must + pick one of the two shapes in the integration sketch (ephemeral `--image` + registry + allowlist, or an offline image store / newer smolvm) and verify it before committing to + the adapter design. This is the single biggest open integration question. ## Recommended next step -If probe 02 case 3 **blocks** on the real run: - -1. Fill the `## Results` and `## Verdict` of all three probe docs and remove the - "provisional" marker at the top of this document. -2. Write a follow-up implementation plan (`plan.md`) covering: the new - `isolatedSandbox` adapter in `src/infra/providers/`, the dispatcher switch on +Case 3 blocked, so the GO path is active: + +1. ✅ **Done.** The `## Results` and `## Verdict` of all three probe docs are filled + (01 PASS, 02 PASS, 03 BLOCKED-no-keys) and the provisional marker is removed. +2. **Resolve residual risk 8 first** (image acquisition vs. allowlist) — it is a + prerequisite for the adapter design, not a detail. Spike whether a newer smolvm lifts + the `--from` + `--allow-host` restriction or whether an offline image store exists; + otherwise commit to the ephemeral-`--image`-plus-registry-allowlist shape and confirm + that registry egress during the pull window is acceptable. +3. **Re-run probe 03 with real API keys** and a reworked harness (pre-baked image carrying + the CLIs; provider/MCP + registry hosts allowlisted; `--timeout 60s` instead of the host + `timeout`; the correct Vibe CLI distribution). This closes the only open probe before any + code is written. +4. Write a follow-up implementation plan (`plan.md`) covering: the new `isolatedSandbox` + adapter in `src/infra/providers/`, the dispatcher switch on `SecurityPolicy.mode === "isolated"`, the widened `mcp.allow` schema, the - capability-preamble prompt builder, the pre-baked guest image recipe, and an e2e - test under `pnpm test:e2e:real` that boots a real VM. -3. Remove the stub rejection in `src/cli/commands/run.ts:152-158` only at the end of - the implementation plan, gated on all of the above. - -If probe 02 case 3 **succeeds** (IP literal reaches a non-allowed host): - -1. Stop the spike. Update this document's verdict to no-go. -2. Open a follow-up scoping question: is a host-side L4 egress filter under smolvm - acceptable, or should `isolated` mode be re-scoped to a different sandbox - technology entirely? No code under `src/` should change until that question is - resolved. + capability-preamble prompt builder, the pre-baked guest image recipe, and an e2e test + under `pnpm test:e2e:real` that boots a real VM. +5. Remove the stub rejection in `src/cli/commands/run.ts:152-158` only at the end of the + implementation plan, gated on all of the above. + +The fix list for the probe scripts themselves (so a future re-run is verbatim) is recorded +in the three findings docs: `01` needs `--net` for the pull (or a pre-bake step); `02` needs +busybox `wget` instead of `apk add curl` and the registry hosts allowlisted; `03` needs the +`--timeout` flag fix, the real Vibe package, and the argv-key-exposure mitigation. diff --git a/spikes/smolvm/findings/01-filesystem.md b/spikes/smolvm/findings/01-filesystem.md index 195e585e..c6968c8c 100644 --- a/spikes/smolvm/findings/01-filesystem.md +++ b/spikes/smolvm/findings/01-filesystem.md @@ -2,11 +2,11 @@ ## Environment -- smolvm version: -- Host OS / arch: +- smolvm version: 1.3.2 +- Host OS / arch: Darwin 25.5.0 / arm64 (Apple Silicon) - Guest arch: arm64 Linux (Alpine, via libkrun on Apple Silicon) -- Date of run: -- Operator: +- Date of run: 2026-06-30 +- Operator: Claude Code (interactive review session, automated run) ## Procedure @@ -62,15 +62,67 @@ If `:ro` is not supported or not enforced, record that as a finding — it const ## Results - +> **Methodology note — script could not be run verbatim.** `01-filesystem.sh` boots +> `smolvm machine run --image alpine` **without `--net`**. On smolvm 1.3.2, an ephemeral +> `machine run` re-pulls the image at every boot, and the pull requires network — so +> without `--net` the very first boot fails at `pull image … network is unreachable`. +> To get a real result the image was pre-baked once into a self-contained artifact +> (`smolvm pack create -I alpine -o alpine.smolmachine`, which pulls under plain `--net`) +> and every check below was run with `smolvm machine run --from alpine.smolmachine -v …`. +> The filesystem boundary is identical to a live `--image alpine` boot (same Alpine +> rootfs, same libkrun virtio-fs mount layer); only the image-acquisition path differs. +> **Fix for the script:** add `--net` to the boots (needed only so the pull can run), or +> document a pre-bake/pre-pull step. See also the network-probe finding — live pull is +> impossible once an egress allowlist is applied, so a pre-baked image is mandatory. + +Raw output (one boot per check, `--from` artifact, `-v` mounts): + +``` +CHECK A: workspace round-trip + write-back + --- guest /workspace --- + -rw-r--r-- 1 root root 21 Jun 30 11:23 sentinel.txt + --- sentinel --- spike-sentinel-73398 + guest wrote writeback.txt + HOST-CHECK A: PASS write-back visible → workspace mount round-trips both ways + +CHECK B: host HOME invisible + ls: /Users/remyloubradou: No such file or directory → HOME-NOT-VISIBLE + cat: can't open '/Users/remyloubradou/.smolvm-probe-…' → MARKER-NOT-VISIBLE + +CHECK C: host repo root invisible + ls: /Users/remyloubradou/.phax/.../phase-05: No such file or directory → NOT-VISIBLE + cat: can't open '…/.smolvm-probe-…': No such file or directory → NOT-VISIBLE + +CHECK D: /etc isolation, no macOS /Users + guest hostname: container + /etc/passwd: root:x:0:0:root:/root:/bin/sh (Alpine default, not host) + ls /Users: No such file or directory → no macOS path leak + +CHECK E: read-only mount (-v …:/workspace:ro) + /bin/sh: can't create /workspace/ro-test.txt: Read-only file system → WRITE-REJECTED + HOST-CHECK E: PASS no write-through → :ro enforced at hypervisor layer +``` ## Verdict - - -**Status:** (not yet run) - -**Conclusion:** +**Status:** PASS (all five checks) — run via pre-baked `--from` artifact, not live `--image` pull. + +| Check | Result | +| ----- | ------ | +| A. Workspace mount round-trip + write-back | PASS — guest reads sentinel, host sees `writeback.txt` | +| B. Host `$HOME` invisible | PASS — path and marker absent in guest | +| C. Host repo root invisible | PASS — path and marker absent in guest | +| D. Host `/etc` not leaking | PASS — Alpine `/etc`, hostname `container`, no `/Users` | +| E. Read-only `:ro` mount enforced | PASS — guest write rejected with EROFS, no host write-through | + +**Conclusion:** smolvm gives a clean Linux-guest filesystem boundary. The host `$HOME`, +the repo root, and macOS paths are simply absent from the guest rootfs — nothing leaks +except the explicit `-v` mount. Crucially for `isolated` mode, **`:ro` is enforced at the +hypervisor layer** (`Read-only file system`), so the worktree can be mounted read-only and +`allowRead` can be distinguished from `allowWrite` at the VM boundary (this resolves +residual risk 3 in the synthesis in the favourable direction). The one caveat is +operational, not security: an ephemeral `--image` boot always re-pulls and the pull needs +network, so a pre-baked image / `--from` artifact is required to boot offline. ## Open questions diff --git a/spikes/smolvm/findings/02-network.md b/spikes/smolvm/findings/02-network.md index 40201641..31538268 100644 --- a/spikes/smolvm/findings/02-network.md +++ b/spikes/smolvm/findings/02-network.md @@ -2,11 +2,11 @@ ## Environment -- smolvm version: -- Host OS / arch: +- smolvm version: 1.3.2 +- Host OS / arch: Darwin 25.5.0 / arm64 (Apple Silicon) - Guest arch: arm64 Linux (Alpine, via libkrun on Apple Silicon) -- Date of run: -- Operator: +- Date of run: 2026-06-30 +- Operator: Claude Code (interactive review session, automated run) ## Procedure @@ -83,19 +83,68 @@ The `## Verdict` must answer both: ## Results - +> **Methodology note — script could not be run verbatim; two blocking issues found.** +> +> 1. **`apk add curl` cannot work under the allowlist.** `02-network.sh` installs curl +> at runtime, but `--allow-host example.com` blocks Alpine's package CDN, so curl +> never installs and *every* case would report code `000` (false "blocked") regardless +> of the real egress behaviour. Replaced with Alpine's built-in busybox `wget` (no +> install, no extra egress). +> 2. **The image pull is itself subject to the allowlist.** `--allow-host` constrains the +> guest DNS resolver to allowed hosts only, so `smolvm machine run --image alpine +> --allow-host example.com` fails to resolve `index.docker.io` (`no such host`) and the +> pull dies before any case runs. To pull, the Docker registry + CDN hosts had to be +> added to the allowlist as well: +> `--allow-host index.docker.io auth.docker.io registry-1.docker.io +> production.cloudfront.docker.com docker.io`. This does **not** weaken the decisive +> case 3: httpbin's IP is still not on the allowlist. +> +> All five cases were run in a **single** boot (the original script uses five) with +> `--timeout 90s`. IPs were resolved on the host immediately before the run. + +Host-resolved before boot: `example.com = 104.20.23.154` (Cloudflare), `httpbin.org = 52.70.185.220`. + +``` +[c1 allowed example.com by NAME ] rc=0 → REACHABLE (allowed egress works) +[c2 blocked httpbin.org by NAME ] rc=1 wget: bad address 'httpbin.org' + → BLOCKED at DNS (deny-by-default) +[c3 blocked httpbin by RAW IP ] rc=1 wget: can't connect to remote host + *** DECISIVE *** (52.70.185.220): Connection refused + → BLOCKED at L4 (real egress boundary) +[c4 allowed example.com RAW IP ] rc=1 wget: server returned error: HTTP/1.1 403 Forbidden + → REACHABLE (TCP connected; 403 is Cloudflare + app-layer for a host-less raw-IP request) +[c5 ICMP ping example.com ] 100% packet loss → ICMP not forwarded (TCP/UDP only) +``` ## Verdict - +**Status:** PASS — `--allow-host` is a real deny-by-default egress boundary. **GO signal.** -**Status:** (not yet run) +**Is egress deny-by-default?** **Yes.** A non-allowed host is unreachable both by name +(c2: not even resolvable — DNS is itself restricted to allowed hosts) and by raw IP +(c3: connection refused). Only the explicitly allowed host is reachable (c1). -**Is egress deny-by-default?** +**Is `--allow-host` a security boundary or DNS convenience?** **A real security boundary.** +This is the decisive result: hard-coding the IP of a non-allowed host (c3) does **not** +bypass the allowlist — the connection is refused at L4. `--allow-host` resolves the allowed +hostname to its IP(s) at VM start and permits egress to those IPs only; everything else is +dropped. An agent (or attacker code an agent runs) cannot escape by shipping an IP literal. -**Is `--allow-host` a security boundary or DNS convenience?** +**Conclusion:** smolvm's egress allowlist closes exactly the gap spec-14 identified as +unenforceable at the provider-native layer. Enforcement is **IP-based** (c4: the allowed +host's resolved IP is reachable by raw IP, returning an app-layer 403 rather than a +connection refusal). Two consequences for a follow-up plan: -**Conclusion:** +- **Shared-CDN-IP caveat (real, must be designed around).** Because enforcement is by + resolved IP, allowlisting one host on a shared CDN address transitively permits any other + host served from that same IP. example.com sits on Cloudflare (104.20.23.154); any other + Cloudflare-fronted host on that IP would be reachable by raw IP. This is residual risk 2 + in the synthesis, now **confirmed** rather than hypothetical. +- **IP pinning at VM start.** Allowed hosts are resolved once at boot; long runs whose + allowed host rotates DNS could see egress break (or a stale IP stay allowed). Minor. + +ICMP is not forwarded (c5) — consistent with smolvm's TCP/UDP-only networking. ## Open questions diff --git a/spikes/smolvm/findings/03-agents.md b/spikes/smolvm/findings/03-agents.md index 3b7d86de..910d9893 100644 --- a/spikes/smolvm/findings/03-agents.md +++ b/spikes/smolvm/findings/03-agents.md @@ -2,11 +2,11 @@ ## Environment -- smolvm version: -- Host OS / arch: +- smolvm version: 1.3.2 +- Host OS / arch: Darwin 25.5.0 / arm64 (Apple Silicon) - Guest OS / arch: Linux arm64 (Alpine, via libkrun on Apple Silicon) -- Date of run: -- Operator: +- Date of run: 2026-06-30 (partial — see status) +- Operator: Claude Code (interactive review session, automated run) ## Procedure @@ -80,27 +80,76 @@ Do not commit outputs that contain key values. ## Results - +> **Status: NOT RUN as a provider task probe — no API keys available in this environment.** +> `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and `MISTRAL_API_KEY` are all unset, so every +> provider would `SKIP` at the key check and no real agent task can execute. The +> per-provider table below is therefore left empty rather than fabricated. What *was* +> established are the infrastructure facts that gate this probe — and they are decisive +> enough to change the integration design (see synthesis). They were proven while running +> probes 01 and 02 on the same smolvm 1.3.2 / Alpine setup. + +**Infrastructure findings (verified, key-independent):** + +1. **`apk add` / `npm install` in-guest cannot work under a provider-only allowlist.** + Step A allowlists only the provider API domain (e.g. `api.anthropic.com`). Installing + the CLI needs Alpine's package CDN and `registry.npmjs.org`, which are not on that + allowlist, so the install fails. (Directly observed for the Docker registry case in + probe 02: `--allow-host` restricts the guest DNS resolver, so any non-allowed host — + including package mirrors — returns `no such host`.) **A live-install harness is not + viable; the provider CLI must be present in a pre-baked image.** This confirms the + spike's residual-risk-7 hypothesis as fact. + +2. **A pre-baked image and `--allow-host` do not compose in smolvm 1.3.2.** `smolvm + machine run --from ` (the pre-baked path) **rejects** `--allow-host`: + `error: the argument '--from ' cannot be used with '--allow-host '`. + The only way to apply a per-domain allowlist is an ephemeral `--image` boot, which + re-pulls the image under that same allowlist and therefore forces the Docker + registry + CDN hosts onto the allowlist too. **This is a new, material constraint the + synthesis's integration sketch did not anticipate** — see the synthesis update. + +3. **`@mistralai/vibe` does not exist on npm.** Verified: `npm info @mistralai/vibe`, + `mistral-vibe`, and `@mistralai/vibe-cli` all 404. The vibe install line in + `03-agents.sh` would fail at install. The real Vibe CLI distribution must be + identified before this provider can be probed. + +4. **Step B's `timeout` wrapper is broken on macOS.** `03-agents.sh:146` calls the + external `timeout` binary, which is absent on stock macOS (`gtimeout` not present + either, confirmed). It would return 127 and the `||` branch would *always* print + "CLI may be hanging/retrying", corrupting the denied-egress observation. **Fix:** use + smolvm's own `--timeout 60s` flag (it exists in 1.3.2) instead of the host `timeout`. + +5. **Credential exposure via argv (security).** `-e "${KEY_VAR}=${KEY_FOR_INJECT}"` places + the literal key on smolvm's command line, readable by any local process via `ps`. + Prefer an env-passthrough that does not materialise the value in argv. (The script is + otherwise careful: `set -eu` not `-eux`, value never echoed, `unset` after use.) ## Verdict - +**Status:** BLOCKED / inconclusive — requires API keys *and* a reworked harness +(pre-baked image with CLIs installed; provider-API + registry hosts allowlisted; busybox +or pre-installed tooling instead of live `apk`/`npm`). The provider-task questions +(install, task completion, write-back, denial UX) remain **unanswered** in this run. -**Status:** (not yet run) - -**Per-provider results table** +**Per-provider results table** — not run (no keys): | Provider | CLI installs in guest? | Completes task (hello.txt)? | Write-back to host? | Network allowlist respected (Step A)? | Denial UX (Step B): error / timeout / hang? | | -------- | ---------------------- | --------------------------- | ------------------- | ------------------------------------- | ------------------------------------------- | -| claude | | | | | | -| codex | | | | | | -| vibe | | | | | | - -**Installation method confirmed** (npm package names, or alternative): - -**Guest arch** (from `uname -m` inside guest): - -**Conclusion:** +| claude | not run | not run | not run | not run | not run | +| codex | not run | not run | not run | not run | not run | +| vibe | not run (`@mistralai/vibe` 404) | not run | not run | not run | not run | + +**Installation method confirmed:** live `apk add nodejs npm && npm install -g …` is +**not viable** under an egress allowlist — a pre-baked image is mandatory (finding 1). + +**Guest arch:** arm64 Linux (confirmed via probes 01/02 on the same setup). + +**Conclusion:** The agent-execution questions are not answerable without keys, but the +surrounding infrastructure work changed the design picture: a pre-baked image is mandatory, +and smolvm 1.3.2 will not apply `--allow-host` to a pre-baked (`--from`) run — so the +follow-up plan must either (a) accept ephemeral `--image` boots that allowlist the registry +alongside provider/MCP domains, or (b) check whether a newer smolvm lifts the +`--from` + `--allow-host` restriction. Write-back and the `:ro` boundary are already proven +in probe 01, so only the provider CLIs' own behaviour inside the sandbox remains open. ## Open questions From 348122a6d39887af6decce64ea1cf3ea6fe8ccf1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?R=C3=A9my=20Loubradou?= Date: Tue, 30 Jun 2026 17:50:03 +0200 Subject: [PATCH 7/7] test(spike): make smolvm probe scripts runnable verbatim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix the bugs that stopped the probe scripts from running as written; all four now run end-to-end on smolvm 1.3.2 (verified): - 00-preflight: printf header starting with "--" was parsed as an option; use '%s\n'. - 01-filesystem: omitted --net so the image pull failed. Bake a --from artifact once (cached) and boot every check from it — no network, no allowlist. Marker cleanup is now interrupt-safe via the trap. - 02-network: apk add curl is blocked by the allowlist (would falsely report every case blocked) and the image pull is itself subject to the allowlist. Use busybox wget with a block-vs-HTTP-error classifier, allowlist the Docker registry hosts so the pull succeeds, run the matrix in a single boot under smolvm --timeout. - 03-agents: replace the host `timeout` (absent on macOS, always false-positive) with smolvm --timeout; boot Step B offline from the --from artifact; allowlist package mirrors + registry for Step A install; make Vibe package overridable via VIBE_NPM_PKG (@mistralai/vibe 404s). Still requires API keys to produce a provider verdict. Findings docs and synthesis updated to note the fixes are now in the scripts. Co-Authored-By: Claude Opus 4.8 --- docs/spikes/smolvm-isolation-findings.md | 11 +- spikes/smolvm/00-preflight.sh | 4 +- spikes/smolvm/01-filesystem.sh | 71 ++++--- spikes/smolvm/02-network.sh | 239 ++++++++--------------- spikes/smolvm/03-agents.sh | 76 ++++--- spikes/smolvm/findings/01-filesystem.md | 24 +-- spikes/smolvm/findings/02-network.md | 25 ++- spikes/smolvm/findings/03-agents.md | 11 +- 8 files changed, 223 insertions(+), 238 deletions(-) diff --git a/docs/spikes/smolvm-isolation-findings.md b/docs/spikes/smolvm-isolation-findings.md index e39ee776..7f3d0264 100644 --- a/docs/spikes/smolvm-isolation-findings.md +++ b/docs/spikes/smolvm-isolation-findings.md @@ -285,7 +285,10 @@ Case 3 blocked, so the GO path is active: 5. Remove the stub rejection in `src/cli/commands/run.ts:152-158` only at the end of the implementation plan, gated on all of the above. -The fix list for the probe scripts themselves (so a future re-run is verbatim) is recorded -in the three findings docs: `01` needs `--net` for the pull (or a pre-bake step); `02` needs -busybox `wget` instead of `apk add curl` and the registry hosts allowlisted; `03` needs the -`--timeout` flag fix, the real Vibe package, and the argv-key-exposure mitigation. +The probe scripts have since been fixed so a future re-run is verbatim: `01` bakes a +`--from` artifact (no `--net` pull problem); `02` uses busybox `wget` and allowlists the +registry hosts; `03` uses smolvm's `--timeout`, allowlists the package mirrors for Step A, +and boots Step B offline from the artifact. Two items remain operator-supplied: real API +keys, and the correct Vibe CLI distribution (`VIBE_NPM_PKG` — `@mistralai/vibe` 404s). The +argv-key-exposure nit (residual risk 6) has no clean fix within smolvm's `-e KEY=VALUE` and +is left documented. diff --git a/spikes/smolvm/00-preflight.sh b/spikes/smolvm/00-preflight.sh index 7684441a..70cee47b 100644 --- a/spikes/smolvm/00-preflight.sh +++ b/spikes/smolvm/00-preflight.sh @@ -17,7 +17,9 @@ fi printf '=== smolvm preflight ===\n\n' # Version -printf '-- smolvm version --\n' +# Note: use '%s\n' here — a format string beginning with "--" is parsed as an option +# by some printf implementations (bash builtin), producing "invalid option". +printf '%s\n' '-- smolvm version --' smolvm --version # Host architecture diff --git a/spikes/smolvm/01-filesystem.sh b/spikes/smolvm/01-filesystem.sh index cdbf46e2..06765480 100644 --- a/spikes/smolvm/01-filesystem.sh +++ b/spikes/smolvm/01-filesystem.sh @@ -12,14 +12,35 @@ # Paste the full output into findings/01-filesystem.md ## Results, then fill ## Verdict. # # smolvm flag reference (all used below): -# smolvm machine run --image ephemeral VM, cleaned up on exit -# -v HOST:CONTAINER[:ro] mount host dir into guest (optional :ro) -# -- COMMAND ARGS command to execute inside the guest +# smolvm pack create -I -o bake image into a self-contained artifact +# smolvm machine run --from boot the baked artifact (no pull, no network) +# -v HOST:CONTAINER[:ro] mount host dir into guest (optional :ro) +# -- COMMAND ARGS command to execute inside the guest set -eu IMAGE="alpine" GUEST_WORKSPACE="/workspace" +# smolvm re-pulls the image on every ephemeral `machine run`, and the pull needs the +# network — so a deliberately network-free filesystem probe cannot boot `--image` directly +# (the pull fails with "network is unreachable"). We bake the image ONCE into a +# self-contained artifact (that single pull runs under plain `--net`) and boot every check +# from it with `--from`, which needs neither network nor an allowlist. The artifact is +# cached between runs; set SMOLVM_SPIKE_CACHE to relocate it, or delete it to force a rebake. +CACHE_DIR="${SMOLVM_SPIKE_CACHE:-${TMPDIR:-/tmp}/smolvm-spike}" +ARTIFACT_BIN="$CACHE_DIR/alpine.smolmachine" +ARTIFACT="$ARTIFACT_BIN.smolmachine" # `pack create` emits this .smolmachine sidecar; --from consumes it + +ensure_artifact() { + if [ -f "$ARTIFACT" ]; then + printf 'Using cached artifact: %s\n' "$ARTIFACT" + return 0 + fi + mkdir -p "$CACHE_DIR" + printf 'Baking %s artifact (one-time; pulls under --net)...\n' "$IMAGE" + smolvm pack create -I "$IMAGE" -o "$ARTIFACT_BIN" --no-sign +} + PASS=0 FAIL=0 @@ -28,13 +49,23 @@ fail() { printf 'FAIL: %s\n' "$1"; FAIL=$((FAIL + 1)); } section() { printf '\n── %s ──\n' "$1"; } printf '=== smolvm filesystem isolation probe ===\n' -printf 'Image: %s\n' "$IMAGE" +printf 'Image: %s (baked artifact)\n' "$IMAGE" printf 'Host arch: %s\n' "$(uname -m)" printf 'Host OS: %s\n' "$(uname -s -r)" +ensure_artifact + +# Marker files written into real host paths for checks B and C. Defined up front so the +# trap can remove them even if the probe is interrupted mid-check (not just WORK_DIR). +MARKER_NAME=".smolvm-probe-$$" +HOST_HOME="$HOME" +HOST_REPO_ROOT="$(pwd)" +HOME_MARKER="$HOST_HOME/$MARKER_NAME" +REPO_MARKER="$HOST_REPO_ROOT/$MARKER_NAME" + # Throwaway dir for the workspace mount; cleaned up on exit. WORK_DIR=$(mktemp -d) -trap 'rm -rf "$WORK_DIR"' EXIT INT TERM +trap 'rm -rf "$WORK_DIR"; rm -f "$HOME_MARKER" "$REPO_MARKER"' EXIT INT TERM # ── A. Workspace mount ───────────────────────────────────────────────────── section "A. Workspace mount (-v HOST:/workspace)" @@ -43,7 +74,7 @@ SENTINEL_VAL="spike-sentinel-$$" printf '%s\n' "$SENTINEL_VAL" > "$WORK_DIR/sentinel.txt" printf 'Sentinel written to host: %s/sentinel.txt\n' "$WORK_DIR" -smolvm machine run --image "$IMAGE" \ +smolvm machine run --from "$ARTIFACT" \ -v "$WORK_DIR:$GUEST_WORKSPACE" \ -- /bin/sh -c " set -e @@ -65,40 +96,36 @@ fi # ── B. Host HOME invisible ───────────────────────────────────────────────── section "B. Host HOME invisible" -HOST_HOME="$HOME" -HOST_HOME_MARKER=".smolvm-probe-$$" -printf 'host-home-probe\n' > "$HOST_HOME/$HOST_HOME_MARKER" -printf 'Probing host path from guest: %s\n' "$HOST_HOME/$HOST_HOME_MARKER" +printf 'host-home-probe\n' > "$HOME_MARKER" +printf 'Probing host path from guest: %s\n' "$HOME_MARKER" -smolvm machine run --image "$IMAGE" \ +smolvm machine run --from "$ARTIFACT" \ -v "$WORK_DIR:$GUEST_WORKSPACE" \ -- /bin/sh -c " echo '--- ls of host HOME path: $HOST_HOME ---' ls '$HOST_HOME' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' echo '--- cat of unique marker file ---' -cat '$HOST_HOME/$HOST_HOME_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +cat '$HOME_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' " -rm -f "$HOST_HOME/$HOST_HOME_MARKER" +rm -f "$HOME_MARKER" # ── C. Host repo root invisible ──────────────────────────────────────────── section "C. Host repo root invisible" -HOST_REPO_ROOT="$(pwd)" -REPO_MARKER=".smolvm-probe-$$" -printf 'repo-probe\n' > "$HOST_REPO_ROOT/$REPO_MARKER" -printf 'Probing host path from guest: %s\n' "$HOST_REPO_ROOT/$REPO_MARKER" +printf 'repo-probe\n' > "$REPO_MARKER" +printf 'Probing host path from guest: %s\n' "$REPO_MARKER" -smolvm machine run --image "$IMAGE" \ +smolvm machine run --from "$ARTIFACT" \ -v "$WORK_DIR:$GUEST_WORKSPACE" \ -- /bin/sh -c " echo '--- ls of host repo root: $HOST_REPO_ROOT ---' ls '$HOST_REPO_ROOT' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' echo '--- cat of unique marker file ---' -cat '$HOST_REPO_ROOT/$REPO_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' +cat '$REPO_MARKER' 2>&1 && echo 'VISIBLE' || echo 'NOT VISIBLE (expected)' " -rm -f "$HOST_REPO_ROOT/$REPO_MARKER" +rm -f "$REPO_MARKER" # ── D. Host /etc not leaking into guest ─────────────────────────────────── section "D. Host /etc not leaking into guest" @@ -106,7 +133,7 @@ section "D. Host /etc not leaking into guest" HOST_HOSTNAME=$(hostname) printf 'Host hostname (for comparison): %s\n' "$HOST_HOSTNAME" -smolvm machine run --image "$IMAGE" \ +smolvm machine run --from "$ARTIFACT" \ -v "$WORK_DIR:$GUEST_WORKSPACE" \ -- /bin/sh -c " echo '--- guest hostname ---' @@ -125,7 +152,7 @@ section "E. Read-only mount (-v HOST:/workspace:ro)" printf 'smolvm flag: -v %s:%s:ro\n' "$WORK_DIR" "$GUEST_WORKSPACE" printf '(smolvm -v flag supports optional :ro suffix per CLI help)\n' -smolvm machine run --image "$IMAGE" \ +smolvm machine run --from "$ARTIFACT" \ -v "$WORK_DIR:$GUEST_WORKSPACE:ro" \ -- /bin/sh -c " echo '--- /workspace contents (read-only mount) ---' diff --git a/spikes/smolvm/02-network.sh b/spikes/smolvm/02-network.sh index 1e1af9c7..2282f7e9 100644 --- a/spikes/smolvm/02-network.sh +++ b/spikes/smolvm/02-network.sh @@ -11,19 +11,22 @@ # Run AFTER 00-preflight.sh confirms smolvm is installed. # Paste the full output into findings/02-network.md ## Results, then fill ## Verdict. # +# Two things this script accounts for (learned by running it on smolvm 1.3.2): +# 1. smolvm applies --allow-host to the IMAGE PULL too, and an ephemeral `machine run` +# re-pulls every boot. So the Docker registry + CDN hosts must be on the allowlist or +# the pull dies with "no such host" before any case runs. They are added below; they do +# NOT affect case 3 (the blocked host's IP is still not allowlisted). +# 2. `apk add curl` cannot work under the allowlist (the package CDN is not allowed), so +# this probe uses Alpine's built-in busybox `wget` — no install, no extra egress. +# # smolvm flag reference (all used below): # smolvm machine run --image ephemeral VM, cleaned up on exit # --net enable networking (off by default) # --allow-host add hostname to egress allowlist -# --allow-cidr add CIDR block to egress allowlist +# --timeout kill the VM after DURATION (safety) # -v HOST:CONTAINER[:ro] mount host dir into guest # -e KEY=VALUE set guest env var # -- COMMAND ARGS command to execute inside the guest -# -# Smolfile equivalent (for reference — not used here, script uses CLI flags): -# [network] -# net = true -# allow_hosts = ["example.com"] set -eu IMAGE="alpine" @@ -34,178 +37,96 @@ ALLOWED_DOMAIN="example.com" # A blocked domain (should be denied by the allowlist). BLOCKED_DOMAIN="httpbin.org" -# Resolve the blocked domain's IP on the HOST (before the VM boots, using host DNS). -# This IP is passed into the guest and used to probe case 3. -BLOCKED_IP=$(nslookup "$BLOCKED_DOMAIN" 2>/dev/null \ - | awk '/^Address: / { print $2; exit }' \ - || true) +# Docker registry + CDN hosts the image pull needs while the allowlist is active. Unquoted +# on use so it word-splits into multiple --allow-host args (intentional). +REGISTRY_ALLOW="--allow-host index.docker.io --allow-host auth.docker.io --allow-host registry-1.docker.io --allow-host registry.docker.io --allow-host production.cloudfront.docker.com --allow-host production.cloudflare.docker.com --allow-host docker.io" -# Fallback: use dig if nslookup produces no result. -if [ -z "$BLOCKED_IP" ]; then - BLOCKED_IP=$(dig +short "$BLOCKED_DOMAIN" A 2>/dev/null | head -1 || true) -fi +# Resolve a host IP using whichever resolver tool is present. Empty on failure. +resolve_ip() { + ip=$(nslookup "$1" 2>/dev/null | awk '/^Address: / { print $2; exit }') + if [ -z "$ip" ]; then + ip=$(dig +short "$1" A 2>/dev/null | head -1) + fi + printf '%s' "$ip" +} +BLOCKED_IP=$(resolve_ip "$BLOCKED_DOMAIN") if [ -z "$BLOCKED_IP" ]; then - printf 'WARN: could not resolve %s on host; case 3 will use 93.184.216.34 (example.com IP as stand-in)\n' \ - "$BLOCKED_DOMAIN" - BLOCKED_IP="93.184.216.34" + printf 'ERROR: could not resolve %s on host; case 3 (the decisive test) needs its IP.\n' "$BLOCKED_DOMAIN" >&2 + exit 1 fi - -PASS=0 -FAIL=0 -SKIP=0 - -pass() { printf 'PASS: %s\n' "$1"; PASS=$((PASS + 1)); } -fail() { printf 'FAIL: %s\n' "$1"; FAIL=$((FAIL + 1)); } -skip() { printf 'SKIP: %s\n' "$1"; SKIP=$((SKIP + 1)); } -section() { printf '\n── %s ──\n' "$1"; } - -# Throwaway workspace dir so every VM boot has a writable mount (some images need it). +ALLOWED_IP=$(resolve_ip "$ALLOWED_DOMAIN") + +# Guest-side classifier (busybox wget). Distinguishes a real egress block (DNS or connect +# failure) from an application-layer HTTP error, which still proves the connection went +# through. Defined here as a literal (single quotes → no host expansion of $1/$MSG) and +# interpolated into the guest payload below. +GUEST_PROBE=' +probe() { + # $1 = label, $2 = url + if wget -T 8 -q -O /dev/null "$2" 2>/tmp/werr; then + printf " %-32s REACHABLE (HTTP 2xx)\n" "$1"; return 0 + fi + MSG=$(cat /tmp/werr 2>/dev/null) + case "$MSG" in + *"server returned error"*) printf " %-32s REACHABLE (connected; %s)\n" "$1" "$MSG" ;; + *"bad address"*) printf " %-32s BLOCKED (DNS denied: %s)\n" "$1" "$MSG" ;; + *"Connection refused"*|*"timed out"*|*"unreachable"*|*"timeout"*) + printf " %-32s BLOCKED (egress denied: %s)\n" "$1" "$MSG" ;; + *) printf " %-32s UNKNOWN (%s)\n" "$1" "$MSG" ;; + esac +}' + +# Throwaway workspace dir so the VM boot has a writable mount (some images need it). WORK_DIR=$(mktemp -d) trap 'rm -rf "$WORK_DIR"' EXIT INT TERM printf '=== smolvm per-domain network allowlist probe ===\n' printf 'Image: %s\n' "$IMAGE" -printf 'Allowed domain: %s\n' "$ALLOWED_DOMAIN" -printf 'Blocked domain: %s\n' "$BLOCKED_DOMAIN" -printf 'Blocked IP: %s (resolved on host before VM boot)\n' "$BLOCKED_IP" +printf 'Allowed domain: %s (IP %s)\n' "$ALLOWED_DOMAIN" "${ALLOWED_IP:-unresolved}" +printf 'Blocked domain: %s (IP %s)\n' "$BLOCKED_DOMAIN" "$BLOCKED_IP" printf 'Host arch: %s\n' "$(uname -m)" printf 'Host OS: %s\n' "$(uname -s -r)" +printf '\nThe matrix runs in a single VM boot under: --allow-host %s (+ registry hosts).\n' "$ALLOWED_DOMAIN" +printf 'Tool: busybox wget (built into Alpine; no apk install).\n' +printf '\nCase legend:\n' +printf ' case1 allowed host by NAME — does egress work at all?\n' +printf ' case2 blocked host by NAME — deny-by-default for unlisted hosts?\n' +printf ' case3 blocked host by RAW IP — DECISIVE: real boundary or DNS filter?\n' +printf ' case4 allowed host by RAW IP — enforcement layer (IP vs SNI/DNS)?\n' +printf ' case5 ICMP ping — TCP/UDP-only behaviour\n' -# Helper: run a curl probe inside the guest, capturing exit code. -# Usage: run_probe -# The guest runs: curl --max-time 8 --silent --output /dev/null --write-out "%{http_code}" -# Exit 0 with a real HTTP code = reachable. Exit non-0 or code 000 = blocked/unreachable. - -# ── Case 1: Allowed domain reachable ────────────────────────────────────────── -section "Case 1: Allowed domain reachable (by name)" - -printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" -printf 'Guest command: curl --max-time 8 http://%s/\n' "$ALLOWED_DOMAIN" +section() { printf '\n── %s ──\n' "$1"; } +section "Running matrix (single boot)" -smolvm machine run --image "$IMAGE" \ - --net --allow-host "$ALLOWED_DOMAIN" \ - -v "$WORK_DIR:/workspace" \ - -- /bin/sh -c " -apk add --quiet --no-cache curl 2>/dev/null || true -echo '--- curl to allowed domain by name ---' -HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ - --write-out '%{http_code}' http://$ALLOWED_DOMAIN/ 2>/dev/null || echo '000') -echo \"HTTP code: \$HTTP_CODE\" -if [ \"\$HTTP_CODE\" != '000' ] && [ \"\$HTTP_CODE\" != '' ]; then - echo 'RESULT: REACHABLE' -else - echo 'RESULT: UNREACHABLE' +# Only test case 4 if the allowed IP resolved. +CASE4_LINE=":" +if [ -n "$ALLOWED_IP" ]; then + CASE4_LINE="probe 'case4 allowed RAW IP' \"http://$ALLOWED_IP/\"" fi -" - -# ── Case 2: Non-allowed domain blocked (by name) ─────────────────────────────── -section "Case 2: Non-allowed domain blocked (by name)" - -printf 'Flag: --net --allow-host %s (NO allowance for %s)\n' "$ALLOWED_DOMAIN" "$BLOCKED_DOMAIN" -printf 'Guest command: curl --max-time 8 http://%s/\n' "$BLOCKED_DOMAIN" smolvm machine run --image "$IMAGE" \ - --net --allow-host "$ALLOWED_DOMAIN" \ + --net --allow-host "$ALLOWED_DOMAIN" $REGISTRY_ALLOW --timeout 120s \ -v "$WORK_DIR:/workspace" \ - -- /bin/sh -c " -apk add --quiet --no-cache curl 2>/dev/null || true -echo '--- curl to blocked domain by name ---' -HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ - --write-out '%{http_code}' http://$BLOCKED_DOMAIN/ 2>/dev/null || echo '000') -echo \"HTTP code: \$HTTP_CODE\" -if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then - echo 'RESULT: BLOCKED (expected)' + -- /bin/sh -c "$GUEST_PROBE +echo '[HTTP egress]' +probe 'case1 allowed by NAME' \"http://$ALLOWED_DOMAIN/\" +probe 'case2 blocked by NAME' \"http://$BLOCKED_DOMAIN/\" +probe 'case3 blocked RAW IP' \"http://$BLOCKED_IP/\" +$CASE4_LINE +echo '[ICMP]' +if ping -c 2 -W 3 $ALLOWED_DOMAIN >/dev/null 2>&1; then + echo ' case5 ICMP ping SUCCEEDED (unexpected — ICMP forwarded)' else - echo 'RESULT: REACHABLE (unexpected — domain filtering may not be active)' + echo ' case5 ICMP ping BLOCKED/UNSUPPORTED (expected — TCP/UDP only)' fi " -# ── Case 3: Non-allowed host by hard-coded IP (THE DECISIVE TEST) ───────────── -section "Case 3: Non-allowed host by hard-coded IP — DECISIVE" - -printf '*** This is the go/no-go signal for the network boundary. ***\n' -printf 'If this PASSES (request succeeds), --allow-host is DNS-name filtering only,\n' -printf 'NOT a true egress boundary. An attacker can bypass it by avoiding DNS lookup.\n\n' -printf 'Blocked domain %s resolved to %s on host.\n' "$BLOCKED_DOMAIN" "$BLOCKED_IP" -printf 'Flag: --net --allow-host %s (no allowance for IP %s)\n' "$ALLOWED_DOMAIN" "$BLOCKED_IP" -printf 'Guest command: curl --max-time 8 http://%s/\n' "$BLOCKED_IP" - -smolvm machine run --image "$IMAGE" \ - --net --allow-host "$ALLOWED_DOMAIN" \ - -v "$WORK_DIR:/workspace" \ - -- /bin/sh -c " -apk add --quiet --no-cache curl 2>/dev/null || true -echo '--- curl to blocked domain by raw IP ---' -HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ - --write-out '%{http_code}' http://$BLOCKED_IP/ 2>/dev/null || echo '000') -echo \"HTTP code: \$HTTP_CODE\" -if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then - echo 'RESULT: BLOCKED (expected — real egress boundary)' -else - echo 'RESULT: REACHABLE (DNS-name filtering only — not a security boundary)' -fi -" - -# ── Case 4: Allowed host by raw IP ──────────────────────────────────────────── -section "Case 4: Allowed host by raw IP" - -# Resolve the allowed domain's IP on the host. -ALLOWED_IP=$(nslookup "$ALLOWED_DOMAIN" 2>/dev/null \ - | awk '/^Address: / { print $2; exit }' \ - || true) -if [ -z "$ALLOWED_IP" ]; then - ALLOWED_IP=$(dig +short "$ALLOWED_DOMAIN" A 2>/dev/null | head -1 || true) -fi - -if [ -z "$ALLOWED_IP" ]; then - printf 'WARN: could not resolve %s on host; skipping case 4\n' "$ALLOWED_DOMAIN" - SKIP=$((SKIP + 1)) -else - printf 'Allowed domain %s resolved to %s on host.\n' "$ALLOWED_DOMAIN" "$ALLOWED_IP" - printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" - printf 'Guest command: curl --max-time 8 http://%s/\n' "$ALLOWED_IP" - printf '(Reveals whether enforcement is SNI/DNS-based: if IP is blocked,\n' - printf ' allowlist uses hostname comparison; if IP is allowed, it may bypass SNI.)\n' - - smolvm machine run --image "$IMAGE" \ - --net --allow-host "$ALLOWED_DOMAIN" \ - -v "$WORK_DIR:/workspace" \ - -- /bin/sh -c " -apk add --quiet --no-cache curl 2>/dev/null || true -echo '--- curl to allowed domain by its raw IP ---' -HTTP_CODE=\$(curl --max-time 8 --silent --output /dev/null \ - --write-out '%{http_code}' http://$ALLOWED_IP/ 2>/dev/null || echo '000') -echo \"HTTP code: \$HTTP_CODE\" -if [ \"\$HTTP_CODE\" = '000' ] || [ \"\$HTTP_CODE\" = '' ]; then - echo 'RESULT: BLOCKED (enforcement may be SNI/hostname-based, not IP-based)' -else - echo 'RESULT: REACHABLE (enforcement may pass traffic to IPs of allowed hosts)' -fi -" -fi - -# ── Case 5: ICMP (ping) — smolvm docs say TCP/UDP only ─────────────────────── -section "Case 5: ICMP — smolvm allows TCP/UDP only (no ICMP)" - -printf 'smolvm documentation: TCP and UDP are forwarded; ICMP is not supported.\n' -printf 'Attempting ping inside guest to record actual behaviour.\n' -printf 'Flag: --net --allow-host %s\n' "$ALLOWED_DOMAIN" -printf 'Guest command: ping -c 2 -W 5 %s\n' "$ALLOWED_DOMAIN" - -smolvm machine run --image "$IMAGE" \ - --net --allow-host "$ALLOWED_DOMAIN" \ - -v "$WORK_DIR:/workspace" \ - -- /bin/sh -c " -echo '--- ping to allowed domain (ICMP, expect failure) ---' -ping -c 2 -W 5 $ALLOWED_DOMAIN 2>&1 && echo 'RESULT: ICMP SUCCEEDED (unexpected)' \ - || echo 'RESULT: ICMP BLOCKED or UNSUPPORTED (expected)' -" - # ── Summary ──────────────────────────────────────────────────────────────────── -printf '\n=== Probe complete: %d PASS, %d FAIL, %d SKIP ===\n' "$PASS" "$FAIL" "$SKIP" -printf '\nKey question for ## Verdict:\n' -printf ' 1. Is egress deny-by-default? (Did case 2 block the non-allowed domain?)\n' -printf ' 2. Is --allow-host a security boundary? (Did case 3 block the raw IP?)\n' -printf '\nPaste this output into findings/02-network.md ## Results\n' -printf 'then fill in ## Verdict.\n' +printf '\n=== Probe complete ===\n' +printf '\nKey questions for ## Verdict:\n' +printf ' 1. Is egress deny-by-default? (case2 + case3 both BLOCKED?)\n' +printf ' 2. Is --allow-host a security boundary? (case3 — raw IP to a non-allowed\n' +printf ' host — BLOCKED? If REACHABLE, it is DNS-name filtering only, not a boundary.)\n' +printf ' 3. Enforcement layer? (case4 REACHABLE => IP-based; note the shared-CDN-IP caveat.)\n' +printf '\nPaste this output into findings/02-network.md ## Results, then fill ## Verdict.\n' diff --git a/spikes/smolvm/03-agents.sh b/spikes/smolvm/03-agents.sh index bb02123b..5c41a3c0 100644 --- a/spikes/smolvm/03-agents.sh +++ b/spikes/smolvm/03-agents.sh @@ -39,10 +39,36 @@ set -eu IMAGE="alpine" TRIVIAL_PROMPT="Create a file called hello.txt in /workspace containing exactly one line: hello from smolvm" -# Step B timeout: seconds to wait before declaring a hang. -# This prevents the harness from blocking if the CLI loops on retries. +# Step B timeout: seconds before declaring a hang. Enforced by smolvm's own --timeout flag +# (NOT the host `timeout` binary, which is absent on macOS and would always report a false +# hang). This prevents the harness from blocking if the CLI loops on retries. STEP_B_TIMEOUT=60 +# Step A installs the CLI live (apk + npm), so the package mirrors AND the Docker registry +# must be on the allowlist alongside the provider API domain — otherwise the allowlist +# blocks them and the install fails. Unquoted on use so each splits into --allow-host args. +# (A production isolated mode would instead ship a PRE-BAKED image with the CLI already in +# it; live install is shown here only to surface that requirement — see findings doc.) +REGISTRY_ALLOW="--allow-host index.docker.io --allow-host auth.docker.io --allow-host registry-1.docker.io --allow-host registry.docker.io --allow-host production.cloudfront.docker.com --allow-host production.cloudflare.docker.com --allow-host docker.io" +MIRROR_ALLOW="--allow-host dl-cdn.alpinelinux.org --allow-host registry.npmjs.org" + +# Step B boots with ZERO egress. smolvm cannot pull an image without network, so Step B +# boots from a pre-baked artifact (`--from`) instead of `--image`. This is also the honest +# shape of the finding: with no network the CLI cannot be installed live, so a real isolated +# mode needs the CLI already in the image. The artifact is baked once and cached. +CACHE_DIR="${SMOLVM_SPIKE_CACHE:-${TMPDIR:-/tmp}/smolvm-spike}" +ARTIFACT_BIN="$CACHE_DIR/alpine.smolmachine" +ARTIFACT="$ARTIFACT_BIN.smolmachine" # `pack create` emits this .smolmachine sidecar; --from consumes it + +ensure_artifact() { + if [ -f "$ARTIFACT" ]; then + return 0 + fi + mkdir -p "$CACHE_DIR" + printf 'Baking %s artifact for Step B (one-time; pulls under --net)...\n' "$IMAGE" + smolvm pack create -I "$IMAGE" -o "$ARTIFACT_BIN" --no-sign +} + # ── Helpers ──────────────────────────────────────────────────────────────────── section() { printf '\n══ %s ══\n' "$1"; } step() { printf '\n── %s ──\n' "$1"; } @@ -99,11 +125,10 @@ probe_provider() { eval "KEY_FOR_INJECT=\${${KEY_VAR}}" smolvm machine run --image "$IMAGE" \ - --net --allow-host "$DOMAIN" \ + --net --allow-host "$DOMAIN" $REGISTRY_ALLOW $MIRROR_ALLOW --timeout 180s \ -v "$WORK_DIR:/workspace" \ -e "${KEY_VAR}=${KEY_FOR_INJECT}" \ -- /bin/sh -c " -set -e echo '--- installing CLI in guest ---' $INSTALL echo '--- CLI version / confirm binary ---' @@ -121,7 +146,7 @@ if [ -f /workspace/hello.txt ]; then else echo 'RESULT: hello.txt NOT CREATED' fi -" +" || printf 'STEP A: VM exited non-zero (see output above)\n' if [ -f "$WORK_DIR/hello.txt" ]; then printf 'HOST-SIDE CHECK: hello.txt visible on host — write-back confirmed\n' @@ -131,31 +156,35 @@ fi unset KEY_FOR_INJECT - # ── STEP B: zero egress (no --net), observe denial UX ───────────────────── - step "STEP B — denied egress (no --net, ${STEP_B_TIMEOUT}s timeout)" - printf 'Flags: -e %s= (no --net — all egress blocked)\n' "$KEY_VAR" + # ── STEP B: zero egress, observe denial UX ──────────────────────────────── + step "STEP B — denied egress (no --net, smolvm --timeout ${STEP_B_TIMEOUT}s)" + printf 'Flags: --from -e %s= (no --net — all egress blocked)\n' "$KEY_VAR" printf 'Goal: capture how the CLI surfaces network loss (error / hang / retry)\n' - printf 'A timeout here means the CLI is silently retrying or hanging.\n' + printf 'Hitting the smolvm --timeout means the CLI is silently retrying or hanging.\n' + # Boot offline: with no network smolvm cannot pull, so Step B runs the pre-baked artifact. + # A bare Alpine artifact has no provider CLI (live install needs network), which is itself + # the finding — a real isolated mode must ship the CLI in the image. + ensure_artifact eval "KEY_FOR_INJECT=\${${KEY_VAR}}" - # Wrap the VM boot in a timeout so a hanging CLI does not block the harness. STEP_B_WORK="$BASE_WORK_DIR/${PNAME}-step-b" mkdir -p "$STEP_B_WORK" - timeout "$STEP_B_TIMEOUT" \ - smolvm machine run --image "$IMAGE" \ + # smolvm's own --timeout bounds the boot so a hanging CLI cannot block the harness + # (the host `timeout` binary is absent on macOS — do not use it). + smolvm machine run --from "$ARTIFACT" --timeout "${STEP_B_TIMEOUT}s" \ -v "$STEP_B_WORK:/workspace" \ -e "${KEY_VAR}=${KEY_FOR_INJECT}" \ -- /bin/sh -c " -set -e -echo '--- installing CLI in guest (no net: may fail if apk needs internet) ---' -$INSTALL 2>&1 || echo 'INSTALL FAILED (expected without --net)' -echo '--- running task without network ---' +echo '--- CLI present in guest? (no net, no live install possible) ---' +command -v $PNAME >/dev/null 2>&1 && echo 'CLI present' \ + || echo 'CLI ABSENT — no pre-baked image, so it cannot run offline (this is the finding)' +echo '--- attempting task without network ---' $RUN_A 2>&1 || true echo 'CLI exited (exit captured above)' -" && printf 'STEP B: VM exited within timeout\n' \ - || printf 'STEP B: VM hit %ds timeout — CLI may be hanging/retrying\n' "$STEP_B_TIMEOUT" +" && printf 'STEP B: VM exited within %ds\n' "$STEP_B_TIMEOUT" \ + || printf 'STEP B: VM hit %ds smolvm --timeout — CLI may be hanging/retrying\n' "$STEP_B_TIMEOUT" unset KEY_FOR_INJECT @@ -193,10 +222,13 @@ CODEX_RUN="printf '%s' '$TRIVIAL_PROMPT' | codex exec -C /workspace --skip-git-r # --workdir scope the agent's working directory # --output streaming JSONL streaming output (matches phax's invocation) # -# NOTE: verify the npm package name before running. -# Candidates: @mistralai/vibe or mistral-vibe or a direct binary download. -# The phax executable name is `vibe` (src/domain/routing/defaults.ts line ~117). -VIBE_INSTALL="apk add --quiet --no-cache nodejs npm 2>/dev/null && npm install --quiet -g @mistralai/vibe 2>/dev/null" +# NOTE: the npm package name is UNKNOWN. `@mistralai/vibe`, `mistral-vibe`, and +# `@mistralai/vibe-cli` all 404 on npm (verified 2026-06-30). Find the real Vibe CLI +# distribution from Mistral's docs and export VIBE_NPM_PKG= before running, or this +# provider's install will fail. The phax executable name is `vibe` +# (src/domain/routing/defaults.ts:117). +VIBE_NPM_PKG="${VIBE_NPM_PKG:-@mistralai/vibe}" +VIBE_INSTALL="apk add --quiet --no-cache nodejs npm 2>/dev/null && npm install --quiet -g $VIBE_NPM_PKG 2>/dev/null" VIBE_RUN="vibe -p '$TRIVIAL_PROMPT' --agent auto-approve --workdir /workspace --output streaming 2>&1 || true" # ── Dispatch ──────────────────────────────────────────────────────────────────── diff --git a/spikes/smolvm/findings/01-filesystem.md b/spikes/smolvm/findings/01-filesystem.md index c6968c8c..6a2dcf80 100644 --- a/spikes/smolvm/findings/01-filesystem.md +++ b/spikes/smolvm/findings/01-filesystem.md @@ -62,18 +62,18 @@ If `:ro` is not supported or not enforced, record that as a finding — it const ## Results -> **Methodology note — script could not be run verbatim.** `01-filesystem.sh` boots -> `smolvm machine run --image alpine` **without `--net`**. On smolvm 1.3.2, an ephemeral -> `machine run` re-pulls the image at every boot, and the pull requires network — so -> without `--net` the very first boot fails at `pull image … network is unreachable`. -> To get a real result the image was pre-baked once into a self-contained artifact -> (`smolvm pack create -I alpine -o alpine.smolmachine`, which pulls under plain `--net`) -> and every check below was run with `smolvm machine run --from alpine.smolmachine -v …`. -> The filesystem boundary is identical to a live `--image alpine` boot (same Alpine -> rootfs, same libkrun virtio-fs mount layer); only the image-acquisition path differs. -> **Fix for the script:** add `--net` to the boots (needed only so the pull can run), or -> document a pre-bake/pre-pull step. See also the network-probe finding — live pull is -> impossible once an egress allowlist is applied, so a pre-baked image is mandatory. +> **Methodology note — how this was run (now built into the script).** The original +> `01-filesystem.sh` booted `smolvm machine run --image alpine` **without `--net`**. On +> smolvm 1.3.2 an ephemeral `machine run` re-pulls the image at every boot and the pull +> requires network, so without `--net` the very first boot failed at `pull image … network +> is unreachable`. The script now bakes the image once into a self-contained artifact +> (`ensure_artifact` → `smolvm pack create -I alpine`, which pulls under plain `--net`, +> cached between runs) and boots every check with `smolvm machine run --from -v …` +> — no network, no allowlist needed. The filesystem boundary is identical to a live +> `--image alpine` boot (same Alpine rootfs, same libkrun virtio-fs mount layer); only the +> image-acquisition path differs. This matters beyond the probe: live pull is impossible +> once an egress allowlist is applied (see the network finding), so a pre-baked image is +> mandatory for any real isolated mode. Raw output (one boot per check, `--from` artifact, `-v` mounts): diff --git a/spikes/smolvm/findings/02-network.md b/spikes/smolvm/findings/02-network.md index 31538268..0ef97964 100644 --- a/spikes/smolvm/findings/02-network.md +++ b/spikes/smolvm/findings/02-network.md @@ -83,24 +83,23 @@ The `## Verdict` must answer both: ## Results -> **Methodology note — script could not be run verbatim; two blocking issues found.** +> **Methodology note — two blocking issues, now handled by the script.** The original +> `02-network.sh` could not run as written, for two reasons now fixed in the script: > -> 1. **`apk add curl` cannot work under the allowlist.** `02-network.sh` installs curl -> at runtime, but `--allow-host example.com` blocks Alpine's package CDN, so curl -> never installs and *every* case would report code `000` (false "blocked") regardless -> of the real egress behaviour. Replaced with Alpine's built-in busybox `wget` (no -> install, no extra egress). +> 1. **`apk add curl` cannot work under the allowlist.** Installing curl at runtime needs +> Alpine's package CDN, which `--allow-host example.com` blocks — so curl never installs +> and *every* case would report a false "blocked" regardless of real egress behaviour. +> The script now uses Alpine's built-in busybox `wget` (no install, no extra egress) with +> a classifier that separates a real egress block from an app-layer HTTP error. > 2. **The image pull is itself subject to the allowlist.** `--allow-host` constrains the > guest DNS resolver to allowed hosts only, so `smolvm machine run --image alpine > --allow-host example.com` fails to resolve `index.docker.io` (`no such host`) and the -> pull dies before any case runs. To pull, the Docker registry + CDN hosts had to be -> added to the allowlist as well: -> `--allow-host index.docker.io auth.docker.io registry-1.docker.io -> production.cloudfront.docker.com docker.io`. This does **not** weaken the decisive -> case 3: httpbin's IP is still not on the allowlist. +> pull dies before any case runs. The script now adds the Docker registry + CDN hosts to +> the allowlist (`REGISTRY_ALLOW`). This does **not** weaken the decisive case 3: +> httpbin's IP is still not on the allowlist. > -> All five cases were run in a **single** boot (the original script uses five) with -> `--timeout 90s`. IPs were resolved on the host immediately before the run. +> All five cases run in a **single** boot with smolvm `--timeout`. IPs are resolved on the +> host immediately before the run. Host-resolved before boot: `example.com = 104.20.23.154` (Cloudflare), `httpbin.org = 52.70.185.220`. diff --git a/spikes/smolvm/findings/03-agents.md b/spikes/smolvm/findings/03-agents.md index 910d9893..9d2aa965 100644 --- a/spikes/smolvm/findings/03-agents.md +++ b/spikes/smolvm/findings/03-agents.md @@ -112,11 +112,12 @@ Do not commit outputs that contain key values. `03-agents.sh` would fail at install. The real Vibe CLI distribution must be identified before this provider can be probed. -4. **Step B's `timeout` wrapper is broken on macOS.** `03-agents.sh:146` calls the - external `timeout` binary, which is absent on stock macOS (`gtimeout` not present - either, confirmed). It would return 127 and the `||` branch would *always* print - "CLI may be hanging/retrying", corrupting the denied-egress observation. **Fix:** use - smolvm's own `--timeout 60s` flag (it exists in 1.3.2) instead of the host `timeout`. +4. **Step B's `timeout` wrapper was broken on macOS (now fixed).** The original + `03-agents.sh` called the external `timeout` binary, absent on stock macOS (`gtimeout` + too, confirmed); it would return 127 and *always* print "CLI may be hanging/retrying", + corrupting the denied-egress observation. The script now uses smolvm's own `--timeout` + flag and boots Step B from a pre-baked `--from` artifact (a network-less VM cannot pull), + so it both runs offline and bounds any hang. 5. **Credential exposure via argv (security).** `-e "${KEY_VAR}=${KEY_FOR_INJECT}"` places the literal key on smolvm's command line, readable by any local process via `ps`.