bench: add a todo-enforcement A/B cell for Terminal-Bench - #3942
Closed
atishpatel wants to merge 1 commit into
Closed
bench: add a todo-enforcement A/B cell for Terminal-Bench#3942atishpatel wants to merge 1 commit into
atishpatel wants to merge 1 commit into
Conversation
buzz-dev-mcp already exposes the `todo` tool in every benchmark cell, and base_prompt.md already instructs the agent to use it. What is missing is enforcement: MCP_HOOK_SERVERS is unset, so hook_servers resolves to HookServers::None and the _Stop/_PostCompact hooks never fire. The list is advisory — an agent can open items and still end its turn. Add `todo_enforcement` to ExperimentManifest (default false, matching buzz-agent's own default). When on, _agent_env sets MCP_HOOK_SERVERS=* and BUZZ_AGENT_STOP_MAX_REJECTIONS=2 — fewer retries than buzz-agent's default of 3, because trials run against a hard wall clock. Ship it as a paired cell rather than a change to every run: tb-cobol-sonnet-haiku-criteria.yaml is byte-identical to tb-cobol-sonnet-haiku.yaml except for the two intervention dimensions (todo_enforcement, and -criteria personas that add the acceptance-criteria rules). Control personas are untouched so existing manifests keep verifying against their recorded sha256. Measured against 179 archived trials (53.1% pass, 62% of failures partial): hand-analysis of 18 failures put the prose intervention at 5 yes / 10 no / 3 unclear. This lands it as a runnable comparison, not as a claimed improvement. Signed-off-by: Atish Patel <atish@squareup.com> Co-authored-by: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A runnable A/B cell, not a claimed improvement. It makes two things testable
on Terminal-Bench 2.1 that today are untestable:
todoenforcement.buzz-dev-mcpalready exposes thetodotool in everycell, and
buzz-acp'sbase_prompt.mdalready tells the agent to use it. What'smissing is enforcement:
MCP_HOOK_SERVERSis unset, sohook_serversresolves toHookServers::Noneand the_Stop/_PostCompacthooks never fire. The list isadvisory — an agent can open items and still
end_turn.task's stated conditions into
todofirst, then show the command and its literaloutput for each before reporting.
Changes
manifest.pytodo_enforcement: bool = FalseonExperimentManifestcontainer_runtime.py_agent_envsetsMCP_HOOK_SERVERS=*andBUZZ_AGENT_STOP_MAX_REJECTIONS=2; result metadata recordsagent_todo_enforcementpersonas/worker-tb-criteria.mdpersonas/orchestrator-tb-criteria.mdmanifests/tb-cobol-sonnet-haiku-criteria.yamltests/test_manifest.pytests/test_container_runtime.pyDefault is off, matching
buzz-agent's own default.BUZZ_AGENT_STOP_MAX_REJECTIONS=2is deliberately below
buzz-agent's default of 3 — trials run against a hard wall clock.Why a paired cell instead of editing the existing one
_verify_artifacthashes each persona file and raises on mismatch, so editing apersona in place would break every manifest that records its
sha256. The controlpersonas are untouched;
tb-cobol-sonnet-haiku-criteria.yamlis byte-identical totb-cobol-sonnet-haiku.yamlexcept for the two intervention dimensions, so thecomparison isn't confounded. Both arms hash differently (
b7ddb4ef4cc3vs2103bd40a514), so results stay attributable.Evidence — read this before assuming it will help
Measured against 179 archived trials (
laptop-jobs-20260728), 53.1% pass:AgentTimeoutError. Enforcement spends wall clock, so itcan plausibly make this slice worse — that's part of what the A/B measures.
this?": 5 yes / 10 no / 3 unclear. Honest ceiling in the threshold cohort is
2–3 of 11.
The dominant failure mode is not failure to notice the shortfall. In 6 of 11
threshold cases the agent wrote the shortfall into its own
DONE:message andfinished anyway. A prompt that says "enumerate the criteria" doesn't fix an agent
that already enumerated them. That's the argument for the
_Stophook — amechanical refusal to
end_turn— over prose alone, and the reason both dimensionsare in the same cell.
Where the prose plausibly does help is narrower than "didn't enumerate": it's
criterion substitution —
bash -iassumed to imply.bashrcsourcing and nevertested; "no invalid accesses" reported as satisfying "no leaks" while valgrind said
definitely lost: 10,485,784 bytes. Rules 6–7 target that specifically.Two caveats I'd want a reviewer to hold me to:
DONE:messages, but that'svery likely reverse causation (runs that verified had output to quote). Don't read
it as the intervention working.
half wrong on this data —
query-optimizebeat the reference ~400× (6m29.9s → 0.9s)and still lost, to a hidden golden query at
1.05 × golden.Follow-up needed (not in this PR)
solo-tb.mdand the luna/terra manifests don't exist onmain— they live onbenchmark/harness-accounting-and-solo. The solo cell and those manifests need thesame treatment arm there, or the flag only covers the orchestrated cells.
Verification
pytest -q→ 37 passedruff check .→ clean;ruff format --check .→ 22 files formattedsha256values verifycondition=tb-cobol-sonnet46-2haiku45,todo_enforcement=FalseNot run: an actual benchmark pass. This PR only makes the experiment expressible.