Skip to content

bench: add a todo-enforcement A/B cell for Terminal-Bench - #3942

Closed
atishpatel wants to merge 1 commit into
mainfrom
buzz-agent/todo-enabled-by-default
Closed

bench: add a todo-enforcement A/B cell for Terminal-Bench#3942
atishpatel wants to merge 1 commit into
mainfrom
buzz-agent/todo-enabled-by-default

Conversation

@atishpatel

Copy link
Copy Markdown
Contributor

What this is

A runnable A/B cell, not a claimed improvement. It makes two things testable
on Terminal-Bench 2.1 that today are untestable:

  1. todo enforcement. buzz-dev-mcp already exposes the todo tool in every
    cell, and buzz-acp's base_prompt.md already tells the agent to use it. What's
    missing is enforcement: MCP_HOOK_SERVERS is unset, so hook_servers resolves to
    HookServers::None and the _Stop/_PostCompact hooks never fire. The list is
    advisory — an agent can open items and still end_turn.
  2. Acceptance-criteria prompting. Persona rules that ask the agent to write the
    task's stated conditions into todo first, then show the command and its literal
    output for each before reporting.

Changes

File Change
manifest.py todo_enforcement: bool = False on ExperimentManifest
container_runtime.py when on, _agent_env sets MCP_HOOK_SERVERS=* and BUZZ_AGENT_STOP_MAX_REJECTIONS=2; result metadata records agent_todo_enforcement
personas/worker-tb-criteria.md
personas/orchestrator-tb-criteria.md
new — byte-identical to the existing personas plus rules 6–7
manifests/tb-cobol-sonnet-haiku-criteria.yaml new — treatment arm
tests/test_manifest.py
tests/test_container_runtime.py
default-off, condition-identity, and env-gating coverage

Default is off, matching buzz-agent's own default. BUZZ_AGENT_STOP_MAX_REJECTIONS=2
is deliberately below buzz-agent's default of 3 — trials run against a hard wall clock.

Why a paired cell instead of editing the existing one

_verify_artifact hashes each persona file and raises on mismatch, so editing a
persona in place would break every manifest that records its sha256. The control
personas are untouched; tb-cobol-sonnet-haiku-criteria.yaml is byte-identical to
tb-cobol-sonnet-haiku.yaml except for the two intervention dimensions, so the
comparison isn't confounded. Both arms hash differently (b7ddb4ef4cc3 vs
2103bd40a514), so results stay attributable.

Evidence — read this before assuming it will help

Measured against 179 archived trials (laptop-jobs-20260728), 53.1% pass:

  • 84 failures: 52 partial (62%), 31 all-fail (37%), 1 no CTRF.
  • 11/84 (13%) were AgentTimeoutError. Enforcement spends wall clock, so it
    can plausibly make this slice worse — that's part of what the A/B measures.
  • 18 failures hand-analyzed for "would the acceptance-criteria prompt have saved
    this?": 5 yes / 10 no / 3 unclear. Honest ceiling in the threshold cohort is
    2–3 of 11.

The dominant failure mode is not failure to notice the shortfall. In 6 of 11
threshold cases the agent wrote the shortfall into its own DONE: message and
finished anyway. A prompt that says "enumerate the criteria" doesn't fix an agent
that already enumerated them. That's the argument for the _Stop hook — a
mechanical refusal to end_turn — over prose alone, and the reason both dimensions
are in the same cell.

Where the prose plausibly does help is narrower than "didn't enumerate": it's
criterion substitutionbash -i assumed to imply .bashrc sourcing and never
tested; "no invalid accesses" reported as satisfying "no leaks" while valgrind said
definitely lost: 10,485,784 bytes. Rules 6–7 target that specifically.

Two caveats I'd want a reviewer to hold me to:

  • Correlation trap. Passing runs have more quantified DONE: messages, but that's
    very likely reverse causation (runs that verified had output to quote). Don't read
    it as the intervention working.
  • The claim that "'faster than the reference' is not satisfied by matching it" is
    half wrong on this data — query-optimize beat the reference ~400× (6m29.9s → 0.9s)
    and still lost, to a hidden golden query at 1.05 × golden.

Follow-up needed (not in this PR)

solo-tb.md and the luna/terra manifests don't exist on main — they live on
benchmark/harness-accounting-and-solo. The solo cell and those manifests need the
same treatment arm there, or the flag only covers the orchestrated cells.

Verification

  • pytest -q37 passed
  • ruff check . → clean; ruff format --check . → 22 files formatted
  • Both manifests load; all 4 persona sha256 values verify
  • Control re-confirmed unchanged: condition=tb-cobol-sonnet46-2haiku45, todo_enforcement=False

Not run: an actual benchmark pass. This PR only makes the experiment expressible.

buzz-dev-mcp already exposes the `todo` tool in every benchmark cell, and
base_prompt.md already instructs the agent to use it. What is missing is
enforcement: MCP_HOOK_SERVERS is unset, so hook_servers resolves to
HookServers::None and the _Stop/_PostCompact hooks never fire. The list is
advisory — an agent can open items and still end its turn.

Add `todo_enforcement` to ExperimentManifest (default false, matching
buzz-agent's own default). When on, _agent_env sets MCP_HOOK_SERVERS=*
and BUZZ_AGENT_STOP_MAX_REJECTIONS=2 — fewer retries than buzz-agent's
default of 3, because trials run against a hard wall clock.

Ship it as a paired cell rather than a change to every run:
tb-cobol-sonnet-haiku-criteria.yaml is byte-identical to
tb-cobol-sonnet-haiku.yaml except for the two intervention dimensions
(todo_enforcement, and -criteria personas that add the acceptance-criteria
rules). Control personas are untouched so existing manifests keep
verifying against their recorded sha256.

Measured against 179 archived trials (53.1% pass, 62% of failures
partial): hand-analysis of 18 failures put the prose intervention at
5 yes / 10 no / 3 unclear. This lands it as a runnable comparison, not
as a claimed improvement.

Signed-off-by: Atish Patel <atish@squareup.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
@atishpatel
atishpatel requested a review from a team as a code owner July 31, 2026 15:47
@atishpatel atishpatel closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant