Per-task check_timeout_s: infrastructure-booting checks stop timing out#53
Open
davidfoy wants to merge 2 commits into
Open
Per-task check_timeout_s: infrastructure-booting checks stop timing out#53davidfoy wants to merge 2 commits into
davidfoy wants to merge 2 commits into
Conversation
Pins the contract before implementation: absent field = today's behavior (module CHECK_TIMEOUT_S, consulted at call time); present = positive int threaded into the check's wait_for; timeout message names the limit AND its source (task check_timeout_s / default); lint warns on generic infra-boot checks (docker/compose) lacking the field. Slow ticket-verbatim cases (sleep 90 vs the real 60s default) gated behind RINGER_SLOW_ORACLE=1. Fails on unmodified ringer.py; passes on the reference patch. Castle wall: lint patterns are generic only — no product boot-script names in ringer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The verifier hard-coded CHECK_TIMEOUT_S = 60 for every check, so any check that boots real infrastructure (a docker-backed test stack ~90s) always timed out and marked a good worker failed — the OA-138 failure mode. - TaskSpec gains optional check_timeout_s (int|None); absent = today's behavior (module CHECK_TIMEOUT_S), present = positive int, threaded through verify -> _run_check's asyncio.wait_for, resolved at call time. - Timeout message names the limit AND its source: '(task check_timeout_s)' vs '(default)', so retry prompts and post-mortems show whether the cap was chosen or inherited. - lint warns when a check matches generic infra-boot signals (docker / compose up) while check_timeout_s is unset — the OA-138 footgun. Patterns are generic only; product-specific boot-script names stay out of ringer (castle wall). - README documents the field. Invariants preserved: stdin closed, sandbox explicit, verification executes the artifact, logs carry raw worker output only. Zero behavior change without the field set. Implemented by a Ringer worker (gpt-5.5 high, first-try PASS) against a red-proven oracle; independently reviewed and re-verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
davidfoy
marked this pull request as draft
July 16, 2026 21:07
davidfoy
marked this pull request as ready for review
July 16, 2026 21:08
davidfoy
marked this pull request as draft
July 16, 2026 21:18
davidfoy
marked this pull request as ready for review
July 16, 2026 21:18
seanpmgallagher
added a commit
to seanpmgallagher/ringer
that referenced
this pull request
Jul 23, 2026
…r slow build-gate checks # Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi — courtesy PR from a fork. We hit this running Ringer daily and figured it might be useful upstream.
Problem
Verifier._run_checkhard-codesCHECK_TIMEOUT_S = 60for every check. Any check that boots real infrastructure before it can assert — a docker-backed test stack takes ~90s (Mongo + seed + uvicorn) — always trips the 60s cap and marks a correct worker failed, then burns the one retry on the same wall. In our case it cost a good first-try implementation ~104k tokens and polluted the model scoreboard until hand-annotated.Change
Optional per-task manifest field
check_timeout_s— opt-in, zero behavior change when unset.TaskSpec.check_timeout_s: int | None— absent = today's behavior (moduleCHECK_TIMEOUT_S), present = positive int (validated), threaded throughverify → _run_check'sasyncio.wait_for, resolved at call time.check timed out after {n}s (task check_timeout_s)vs(default), so retry prompts and post-mortems show whether the cap was chosen or inherited.lintwarns when a check matches generic infra-boot signals (docker/compose up) whilecheck_timeout_sis unset — the exact footgun. Patterns are intentionally generic; no product-specific script names.Invariants preserved
Stdin closed (DEVNULL), sandbox mode explicit, verification executes the artifact, logs carry raw worker output only. No change to default behavior for any existing manifest.
Tests
Adds
tests/test_check_timeout.py, red-proven before implementation (fails on unmodifiedringer.py). Fast tier shrinks the module constant to prove both the task-override and default branches in seconds; a ticket-verbatim slow tier (realsleep 90vs the real 60s default) is gated behindRINGER_SLOW_ORACLE=1. Full suite green locally (229 passed, 2 skipped) — including the## Contributorscredit rule.Happy to adjust naming, the lint patterns, or the docs to fit your conventions — your call on all of it.