macanderson · macanderson · Aug 1, 2026 · Aug 1, 2026
@@ -23,6 +23,30 @@ bench/evidence/frontier/sentinel.sh          # two cheap trials, end to end
 bench/evidence/frontier/primary.sh small fb-small-01
 ```
 
+## Pick a backend
+
+`FB_ENV` selects where trials run — `docker` (this host, the default) or
+`modal` (a sandbox provisioned per task, which is what the benchmark's own
+authors used).
+
+```bash
+FB_ENV=modal FB_CONCURRENCY=32 bench/evidence/frontier/fetch_dataset.sh
+```
+
+This is not a preference. A submittable run must cover all 74 tasks — four of
+which want a GPU — without overriding any task's declared resources, so it needs
+a machine that can hand a container 32 GB and an H100 on demand. On `docker` the
+plan reports what this host can take and names what it cannot; on `modal` the
+host's shape stops being a term in the measurement, and the plan is 74/74.
+
+Modal needs `modal token new` (or `MODAL_TOKEN_ID`/`MODAL_TOKEN_SECRET`); the
+preflight checks for it and distinguishes "not authenticated" from "SDK not
+installed", because they have different fixes. `warm_images.sh` skips on Modal —
+the builds happen in the cloud, so a local pull warms a cache no trial reads.
+
+The glibc-2.17 assert runs on both. Modal is a different scheduler, not a
+different libc.
+
 ## What differs from Terminal-Bench, and why each difference exists
 
 **A second Harbor pin (0.20.0), not the audited 0.6.1.** Every one of the 74
@@ -72,7 +96,13 @@ scored 0.0 passes stage 2, correctly.
 `plan.py` reports what the host can take. On a 10 GB / 6-CPU Docker VM (a
 typical Mac) it admits **48 of 74** tasks — 4 GPU-excluded and 22 over the memory
 budget. That is a real constraint, not a bug, and it is why a submittable run
-needs a bigger machine: see [SUBMISSION.md](SUBMISSION.md).
+uses `FB_ENV=modal`, where the same plan is 74/74: see
+[SUBMISSION.md](SUBMISSION.md).
+
+Worth knowing before you conclude the laptop is hopeless: 18 of those 22 want
+exactly 8192 MB, which needs roughly a 10.2 GB VM once headroom is taken. Raising
+Docker Desktop's memory allocation moves local coverage from 48 to 66 without
+touching anything here.
 
 The memory exclusion is deliberately conservative. `memory_mb` is a cap Docker
 accepts even when the VM is smaller, so an 8 GB task on a 10 GB daemon usually

@@ -45,30 +45,53 @@ reward hacking, then merge.
 | Dataset pinned to the leaderboard's `DATASET_REF` | **Done.** `env.sh` pins `sha256:97fd2ba3…`, the leaderboard's own ref |
 | Harbor ≥ 0.20.0 | **Done.** This lane pins 0.20.0 |
 | Default execution settings; no timeout or resource overrides | **Done.** Nothing here overrides either; tiering only chooses what runs *concurrently* |
-| All 74 tasks, no subsetting | **Blocked by hardware** — see below |
+| All 74 tasks, no subsetting | **Done on Modal.** `FB_ENV=modal` plans 74/74 with zero exclusions; `FB_ENV=docker` on a laptop plans 48 |
 | ≥ 5 trials per task | **Supported, off by default.** `FB_ATTEMPTS=5`; default 1 is the dev baseline |
-| `--upload --public` | **Not wired.** Deliberate: uploading publishes trajectories, which is a decision to take explicitly, not a flag to inherit |
+| `--upload --public` | **Wired, opt-in.** `FB_UPLOAD=1 FB_UPLOAD_VISIBILITY=public`. Off by default because uploading publishes every trajectory — a decision to take explicitly, never inherited from a shell |
 | Agent + model disclosure metadata | Supplied at `lb submit` time, not by this lane |
 
 Note the metric: **accuracy = trials with reward > 0, over all trials**. Not
 reward == 1.0, and errored trials count as reward 0 rather than being dropped.
 Infrastructure flakiness therefore lands directly in the score, which is exactly
 why `warm_images.sh` and the preflight exist.
 
-## The two real blockers
+## The two hardware blockers, and how the lane clears them
 
 **1. GPUs.** Four tasks need one, and the benchmark authors ran them on a single
-H100 each. Subsetting is not allowed for a submission, so those four cannot be
-excluded the way the dev baseline excludes them — they must actually run.
+H100 each. Subsetting is not allowed, so they must actually run.
 
-**2. Host size.** On a 10 GB / 6-CPU Docker VM, `plan.py` admits 48 of 74 tasks;
-22 more are simply over the memory budget. The full set declares up to 32 GB
-memory, 16 CPUs, and ~1.9 TB of storage in aggregate.
+**2. Host size.** On a 10 GB / 6-CPU Docker VM `plan.py` admits 48 of 74; 22 more
+are over the memory budget. The full set declares up to 32 GB memory, 16 CPUs,
+and ~1.9 TB of storage in aggregate.
 
-Both point the same way: **a submission run does not happen on a Mac.** The
-benchmark's own guidance is Modal (`uv tool install 'harbor[modal]'`, `--env
-modal`), with Daytona as an alternative that recently added GPU support. Harbor's
-repo CI defaults to `env: modal`.
+Both point the same way: **a submission run does not happen on a Mac.** So the
+lane takes a backend:
+
+```bash
+FB_ENV=modal bench/evidence/frontier/fetch_dataset.sh      # plans 74/74
+FB_ENV=modal FB_CONCURRENCY=32 bench/evidence/frontier/sentinel.sh
+```
+
+`FB_ENV=modal` changes three things, each because a docker assumption is simply
+false there rather than merely cautious:
+
+- **Nothing is excluded for capacity.** Modal builds a sandbox per trial from
+  that trial's own declared resources, so "does this task fit" is the wrong
+  question. Carrying the laptop's exclusions over would produce a 48-task
+  result that looks like a complete one.
+- **GPU tasks default to *on*.** Harbor passes each task's `gpus`/`gpu_types`
+  straight through to Modal. Defaulting them off there would silently drop the
+  four tasks a submission is required to cover. `--no-allow-gpu` still wins if
+  you say it.
+- **`warm_images.sh` skips.** The builds happen in the cloud; a local pull warms
+  a cache no trial reads.
+
+What does *not* change: the glibc-2.17 portability assert still runs. Modal is a
+different scheduler, not a different libc — the SUT still has to exec inside a
+Linux container.
+
+Daytona is the documented alternative that recently added GPU support; Harbor's
+own repo CI defaults to `env: modal`.
 
 ## Cost, honestly
 
@@ -91,6 +114,22 @@ was.
    run all 74 at five attempts uncapped with `--upload --public`.
 4. File the PR once #1405 merges.
 
-Steps 1 and 2 are what this directory is for. Step 3 needs a Modal environment
-this lane does not yet configure — `primary.sh` hardcodes `--env docker`, and
-adding a Modal path is the next piece of work, not something already done.
+Step 3 is now a matter of setting variables rather than writing code:
+
+```bash
+export FB_ENV=modal FB_ATTEMPTS=5 FB_CONCURRENCY=32
+export STELLA_BUDGET=            # explicitly empty = no per-trial cap
+export FB_UPLOAD=1 FB_UPLOAD_VISIBILITY=public
+bench/evidence/frontier/fetch_dataset.sh
+bench/evidence/frontier/primary.sh all fb-submission-01
+```
+
+`primary.sh` prints `posture=` on every run and names every reason a run would
+be refused — wrong tier, too few attempts, any excluded task, upload off,
+visibility private — so a baseline cannot be mistaken for a submission at the
+moment a number appears. It says `posture=submission-shaped` only when all five
+hold.
+
+Do not skip the preregistration. This is ~370 trials of multi-hour work with no
+spend cap; that is a four-figure decision and it deserves the same treatment the
+Terminal-Bench claim got.
@@ -50,6 +50,20 @@ export FB_TASK_COUNT="${FB_TASK_COUNT:-74}"
 export FB_ATTEMPTS="${FB_ATTEMPTS:-1}"
 export FB_SUBMISSION_MIN_ATTEMPTS=5
 
+# Upload the finished job to Harbor Hub. Off by default and opt-in per run,
+# because it is not a transport detail: the leaderboard requires the trials to
+# be *publicly readable* — CI re-derives every trial from the uploaded job and a
+# maintainer's judge reads every trajectory — so setting this publishes the
+# agent's full reasoning and every command it ran, permanently and to everyone.
+# That is a deliberate act. It should never be inherited from a shell that
+# happened to have it set for some earlier run.
+export FB_UPLOAD="${FB_UPLOAD:-0}"
+export FB_UPLOAD_VISIBILITY="${FB_UPLOAD_VISIBILITY:-private}"
+case "$FB_UPLOAD_VISIBILITY" in
+  public|private) ;;
+  *) echo "FATAL: FB_UPLOAD_VISIBILITY must be 'public' or 'private'"; return 1 2>/dev/null || exit 1 ;;
+esac
+
 # Frontier-Bench tasks run far longer than Terminal-Bench ones: declared agent
 # timeouts run from 30 minutes to 8 hours, with a 2-hour mode, against expert
 # time estimates measured in whole days. Terminal-Bench's $0.60 per-trial cap
@@ -84,6 +98,23 @@ source "$FB_HERE/../run/env.sh"
 # So this lane pins its own Harbor, in its own virtualenv, and asserts it. The
 # Terminal-Bench venv at bench/harbor_adapter/.venv is left untouched.
 export FB_HARBOR_VERSION="${FB_HARBOR_VERSION:-0.20.0}"
+
+# Where trials actually run. `docker` is this host; `modal` is the cloud
+# sandbox the benchmark's own authors used.
+#
+# The distinction is not a preference. A submittable run must cover all 74
+# tasks, four of which want a GPU, and must not override any task's declared
+# resources — so it needs a machine that can hand a container 32 GB and an
+# H100 on demand. This laptop cannot, and no amount of scheduling makes it.
+# On `docker` the plan honestly reports what the host can take and excludes
+# the rest by name; on `modal` each sandbox is provisioned per task, so the
+# host's shape stops being a term in the measurement at all.
+export FB_ENV="${FB_ENV:-docker}"
+case "$FB_ENV" in
+  docker|modal) ;;
+  *) echo "FATAL: FB_ENV must be 'docker' or 'modal' (got '$FB_ENV')"; return 1 2>/dev/null || exit 1 ;;
+esac
+
 export VENV="$FB_HERE/.venv"
 export PATH="$VENV/bin:$PATH"
 export PYTHONPATH="$TB_REPO/bench/harbor_adapter"
@@ -106,7 +137,16 @@ mkdir -p "$JOBS"
 # The default is therefore to exclude them and say so, loudly, in the score's
 # denominator — not to include them and quietly absorb four guaranteed zeros.
 # Set FB_ALLOW_GPU=1 only on a host that actually has one; `plan.py` checks.
-export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}"
+#
+# On Modal the hazard does not exist: sandboxes are provisioned per task and
+# Harbor passes each task's declared `gpus`/`gpu_types` straight through, so
+# the four run for real. Defaulting them off there would silently drop the
+# tasks a submission is required to cover.
+if [ "$FB_ENV" = modal ]; then
+  export FB_ALLOW_GPU="${FB_ALLOW_GPU:-1}"
+else
+  export FB_ALLOW_GPU="${FB_ALLOW_GPU:-0}"
+fi
 
 # Memory the plan leaves to the Docker daemon itself rather than handing to a
 # task. A task declaring more than what remains is excluded as unrunnable.
@@ -148,6 +188,34 @@ fb_assert_cli_flags() {
     return 1; }
 }
 
+# Whatever FB_ENV names must actually be reachable before a run starts.
+#
+# Note what is *not* branched here: the glibc portability assert runs for both.
+# Modal is a different scheduler, not a different libc — the SUT still has to
+# exec inside a Linux container there, and a binary that cannot would fail the
+# same way it did on 2026-07-31, as an ordinary reward-0 row.
+fb_assert_backend() {
+  case "$FB_ENV" in
+    docker)
+      docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; }
+      ;;
+    modal)
+      "$VENV/bin/python" -c 'import modal' 2>/dev/null || {
+        echo "FATAL: the modal SDK is not installed in $VENV"
+        echo "       Rebuild the venv: bench/evidence/frontier/setup_venv.sh"
+        return 1; }
+      # Harbor's own ModalEnvironment.preflight accepts either form; mirror it
+      # exactly rather than inventing a stricter rule it would then contradict.
+      if [ ! -f "$HOME/.modal.toml" ] &&
+         { [ -z "${MODAL_TOKEN_ID:-}" ] || [ -z "${MODAL_TOKEN_SECRET:-}" ]; }; then
+        echo "FATAL: Modal is not authenticated."
+        echo "       Run 'modal token new', or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET."
+        return 1
+      fi
+      ;;
+  esac
+}
+
 fb_preflight() {
   test -n "${OPENROUTER_API_KEY:-}" || { echo "FATAL: OPENROUTER_API_KEY unset"; return 1; }
   test -x "$STELLA_BINARY" || { echo "FATAL: no SUT binary at $STELLA_BINARY (run ../run/build_sut.sh)"; return 1; }
@@ -163,7 +231,7 @@ fb_preflight() {
     echo "       0.6.1 silently ignores this dataset's verifier environment_mode."
     return 1; }
   fb_assert_cli_flags || return 1
-  docker info >/dev/null 2>&1 || { echo "FATAL: docker unreachable"; return 1; }
+  fb_assert_backend || return 1
 }
 
 # The Terminal-Bench lane's `preflight` is still defined and still asserts

@@ -18,4 +18,5 @@ test "$got" = "$FB_TASK_COUNT" || {
   exit 1; }
 
 python3 "$(dirname "${BASH_SOURCE[0]}")/plan.py" "$DATASET_DIR" \
-  -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt"
+  -o "$FB_PLAN" --images-out "$TB_ROOT/frontier-images.txt" \
+  --backend "$FB_ENV" ${FB_CONCURRENCY:+--concurrency "$FB_CONCURRENCY"}
@@ -83,6 +83,18 @@
 _HOST_FORMAT = '{{.MemTotal}}|{{.NCPU}}|{{(index .Runtimes "nvidia").Path}}'
 
 
+def _elastic_capacity() -> dict:
+    """Capacity for a backend that provisions per task rather than sharing a host.
+
+    Modal builds a sandbox per trial from that trial's own declared resources,
+    so there is no fixed pool to divide and nothing to exclude — asking "does
+    this task fit" is the wrong question there. Returning unbounded capacity
+    makes every downstream check trivially true, which is the honest encoding:
+    the constraint genuinely does not exist rather than being ignored.
+    """
+    return {"memory_mb": None, "cpus": None, "gpu": True, "source": "modal (per-task sandbox)"}
+
+
 def _host_capacity() -> dict:
     """What the Docker daemon says it can actually give a container."""
     cap = {"memory_mb": None, "cpus": None, "gpu": False, "source": "unavailable"}
@@ -166,15 +178,22 @@ def _tier(mem_mb: int) -> str:
     return TIERS[-1][0]
 
 
-def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -> dict:
+def build_plan(
+    dataset_dir: Path,
+    *,
+    allow_gpu: bool,
+    memory_headroom_mb: int,
+    backend: str = "docker",
+    concurrency: int | None = None,
+) -> dict:
     roots = sorted(p.parent for p in dataset_dir.glob("*/*/task.toml"))
     if not roots:
         roots = sorted(p.parent for p in dataset_dir.glob("*/task.toml"))
     if not roots:
         sys.exit(f"FATAL: no task.toml under {dataset_dir}")
 
     tasks = [_read_task(p) for p in roots]
-    host = _host_capacity()
+    host = _elastic_capacity() if backend == "modal" else _host_capacity()
     host_mem = host.get("memory_mb")
     host_cpus = host.get("cpus")
 
@@ -208,9 +227,16 @@ def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -
         members = [t for t in runnable if t["slug"] in set(tier["tasks"])]
         peak_mem = max(t["memory_mb"] for t in members)
         peak_cpus = max(t["cpus"] for t in members)
-        by_mem = (usable_mem // peak_mem) if usable_mem else 1
-        by_cpu = (host_cpus // peak_cpus) if host_cpus else 1
-        tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu))
+        if concurrency is not None:
+            # An explicit ceiling replaces the arithmetic entirely. On a
+            # per-task backend there is no shared pool to divide, so deriving
+            # concurrency from capacity would compute 1 from two unknowns and
+            # serialise a run that has no reason to be serial.
+            tier["concurrency"] = max(1, concurrency)
+        else:
+            by_mem = (usable_mem // peak_mem) if usable_mem else 1
+            by_cpu = (host_cpus // peak_cpus) if host_cpus else 1
+            tier["concurrency"] = max(1, min(MAX_CONCURRENCY, by_mem, by_cpu))
         tier["peak_memory_mb"] = peak_mem
         tier["peak_cpus"] = peak_cpus
         tier["tasks"].sort()
@@ -224,6 +250,7 @@ def build_plan(dataset_dir: Path, *, allow_gpu: bool, memory_headroom_mb: int) -
 
     return {
         "dataset_dir": str(dataset_dir),
+        "backend": backend,
         "host": host,
         "memory_headroom_mb": memory_headroom_mb,
         "allow_gpu": allow_gpu,
@@ -245,26 +272,51 @@ def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("dataset_dir", type=Path)
     ap.add_argument("-o", "--output", type=Path, required=True)
-    ap.add_argument("--allow-gpu", action="store_true",
-                    default=os.environ.get("FB_ALLOW_GPU") == "1")
+    # Tri-state on purpose. The default has to depend on the backend — a Modal
+    # run that quietly drops the four GPU tasks is unsubmittable and looks
+    # exactly like a complete one — but an operator saying "no GPUs" must still
+    # win over that. `None` distinguishes "unset" from "explicitly off", which
+    # `store_true` alone cannot.
+    ap.add_argument("--allow-gpu", action="store_true", default=None)
+    ap.add_argument("--no-allow-gpu", dest="allow_gpu", action="store_false")
     # Memory, not storage. The two were one knob in the first draft of this
     # file and a 20 GB *storage* headroom subtracted from a 10 GB Docker VM's
     # *memory* excluded all 74 tasks with a negative budget — the plan was
     # empty and said so politely.
     ap.add_argument("--memory-headroom-mb", type=int,
                     default=int(os.environ.get("FB_MEMORY_HEADROOM_MB", "2048")))
     ap.add_argument("--images-out", type=Path, default=None)
+    ap.add_argument("--backend", choices=("docker", "modal"),
+                    default=os.environ.get("FB_ENV", "docker"),
+                    help="docker: schedule against this host. modal: per-task "
+                         "sandboxes, so nothing is excluded for capacity.")
+    ap.add_argument("--concurrency", type=int, default=None,
+                    help="Override the derived per-tier concurrency. Required "
+                         "in spirit for modal, where there is no host pool to "
+                         "divide.")
     args = ap.parse_args()
 
-    plan = build_plan(args.dataset_dir, allow_gpu=args.allow_gpu,
-                      memory_headroom_mb=args.memory_headroom_mb)
+    concurrency = args.concurrency
+    if concurrency is None and args.backend == "modal":
+        concurrency = int(os.environ.get("FB_CONCURRENCY", "16"))
+
+    allow_gpu = args.allow_gpu
+    if allow_gpu is None:
+        allow_gpu = args.backend == "modal" or os.environ.get("FB_ALLOW_GPU") == "1"
+
+    plan = build_plan(args.dataset_dir, allow_gpu=allow_gpu,
+                      memory_headroom_mb=args.memory_headroom_mb,
+                      backend=args.backend, concurrency=concurrency)
     args.output.write_text(json.dumps(plan, indent=1) + "\n")
     if args.images_out:
         args.images_out.write_text("\n".join(plan["base_images"]) + "\n")
 
     host = plan["host"]
-    print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs "
-          f"/ gpu={host.get('gpu')} ({host.get('source')})")
+    if plan["backend"] == "modal":
+        print(f"backend: modal — {host.get('source')}; host capacity is not a constraint")
+    else:
+        print(f"host: {host.get('memory_mb')} MB / {host.get('cpus')} CPUs "
+              f"/ gpu={host.get('gpu')} ({host.get('source')})")
     for name, tier in plan["tiers"].items():
         print(f"  tier {name:<7} tasks={len(tier['tasks']):<3} "
               f"concurrency={tier['concurrency']} peak={tier['peak_memory_mb']}MB/"