sync: merge upstream main (12 commits) — keep TRIP kit + launcher, adopt upstream model config at GPT-5.6 - #6
Merged
Merged
Conversation
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ts#51) * demo manifest: punctuation-proof specs + task_type probe Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * registry: openrouter/x-ai/grok-4.5 -> Grok 4.5 / xAI (kills the xAI? UNREGISTERED row) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "registry: openrouter/x-ai/grok-4.5 -> Grok 4.5 / xAI (kills the xAI? UNREGISTERED row)" This reverts commit 09bcc7b0ba41f1b2317bb7897bf71c80dcce895c. * Self-update mechanism + scoreboard signal contract Self-update: throttled git fetch + ff-only pull + re-exec on every invocation (main, clean tracked tree only), './ringer.py self-update' subcommand, HUD staleness thread that restarts itself when disk code is newer than the running process, Ringside update banner when blocked. Escape hatches: --no-self-update, RINGER_NO_SELF_UPDATE, [update] auto. Signal contract (Jon 2026-07-15): every scoreboard surface renders Model·effort | Lab | Harness | API/Plan | Tier | Tasks | First try | Pass | Tokens | Speed | Last used | Notes — no slugs in chart rows, Notes pulled live from docs/MODEL-NOTES.md. Canonical-route enforcement: registry noncanonical_slugs, lint/run refusal with --allow-noncanonical-route, misrouted rows shown under true identity, never ranked. Baseline: 3 stale test expectations repaired. Built as a 3-task Ringer run (gpt-5.6-sol); suite 199 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ateBJones-Projects#30) * fix: commit design-reference fixture instead of hardcoded local path tests/test_design_reference.py read its CSS reference fixture from an absolute path that only existed on the original committer's machine (/private/tmp/claude-501/-Users-jonathanedwards-.../design-reference.html), so test_renderer_tokens_match_design_reference errored with FileNotFoundError on every other clone. The path also leaked the committer's OS username and a client/project directory name into a public repo. Commit the reference fixture at tests/fixtures/design-reference.html (derived from the current ARTIFACT_BASE_CSS, which it was already meant to lock in) and point REFERENCE at it via a path relative to the test file. tests/test_design_reference.py::test_live_page_uses_reference_structure fails independently of this change (pre-existing on main) and is out of scope here. * Fail loudly when the design-reference fixture is missing With the reference committed as a fixture, an absent file means the guard itself is broken — assert instead of skipping so it can never silently deactivate again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Jonathan Edwards <justfinethanku@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add local ringer regression tests
14 stdlib-unittest tests driven by /bin/sh engines: no provider CLIs, no
network, deterministic. Rebased 2026-07-10: two dashboard tests written
against the pre-Ringside per-run dashboard contract were adapted (final-state
coverage kept, port-collision test removed with the removed behavior). README
gains the args_template argv-semantics note.
* Handle graceful shutdown signals
run_manifest wraps the orchestrator in a cancellable task with SIGINT/SIGTERM
handlers returning exit 130, handlers removed in a finally, active-run
registration preserved around it. Composes with main's newer in-runner
CancelledError cleanup: the runner kills workers and finishes state; this
gives the process a deterministic interrupt exit code and prompt cancellation.
* One-shot signal handler + second-signal regression test
A repeat SIGINT/SIGTERM during cleanup cancelled the in-progress worker
kill and state flush, recreating the orphan problem the handler fixes.
The handler now cancels once and reports that shutdown is in progress on
repeats. Also: README placeholder list covers {model}/{model_args}/
{engine_args}, and the e2e tests pin RINGER_NO_SELF_UPDATE=1 so a
self-update can never re-exec mid-test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Jonathan Edwards <justfinethanku@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…Jones-Projects#37) * Name the failure when task setup dies before the worker spawns Failed tasks keep their worktrees (by design, for post-mortems), so a follow-up run with the same run_name collides with the stale taskdir. In the field that surfaced as verdict ERROR at 0.0s with an empty log, empty check output, and no message anywhere — a full diagnosis cycle to discover what one sentence could have said. Three surfaces now carry the reason: - the collision message names the exact unblocking command (git worktree remove --force <taskdir>) instead of a bare "already exists" - _record_prepare_error stores the reason on the task record as setup_error (included in the run-state payload for the HUD and post-mortems) and appends it to the worker log, which is where log_tail, activity, and humans look first - the run summary prints a "setup failures (no worker was spawned)" section listing each affected task with its reason Test: end-to-end worktrees run against a temp git repo with a pre-existing stale taskdir — asserts the ERROR verdict now travels with the collision message in the summary, the worker log, and the run-state record's setup_error field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Only print the worktree-remove command when it will actually work A linked worktree has a .git file; only then is `git worktree remove` the right recovery, now printed repo-qualified and shell-quoted so it is paste-safe from anywhere. A plain directory in the way gets named as exactly that. Tests cover both branches with a real registered stale worktree and a bare directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Jonathan Edwards <justfinethanku@gmail.com>
…ateBJones-Projects#38) * Add run --baseline: execute every check against the unmodified tree Two of twelve lanes in a real fix swarm failed on verify assertions that were wrong about the PRE-change tree — one demanded a status its own lane spec forbade, one was unsatisfiable on a pristine repo. An honest worker burned ~100k tokens against the latter before anyone knew the check was the bug. run --baseline makes that question answerable in one command, before any worker spawns: every task's check runs against a fresh scratch taskdir (a detached worktree of the repo when the manifest uses worktrees), and the report shows pass/FAIL with the check's own output excerpt. Reading it is the orchestrator's judgment call, which the report spells out: a FAIL is expected for assertions that demand the new behavior workers will build; a FAIL on an assertion about unchanged behavior means the check itself is broken and will burn worker attempts against something no model can satisfy. Design notes: - spawns nothing: runs before preflight_engine_bins on purpose, so a missing engine binary cannot block baseline - writes no model-log rows and touches neither the manifest workdir nor the repo (scratch worktrees are removed; the scratch root is deleted) - checks run for real, including any exports they perform — same semantics as at run time - exits 0 regardless of failures: baseline reports, the orchestrator judges Test: end-to-end manifest with a deliberately nonexistent engine binary and two checks — one asserting current tree state (must pass baseline), one demanding a worker-built file (expected FAIL) — asserting the report lines, the excerpt, the guidance, exit 0, an absent model log, and zero leaked taskdirs or worktrees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Baseline hardening: key containment + leaked-worktree reporting Resolve each baseline taskdir and refuse keys that escape the scratch root (same rule as the real run path), and surface failed worktree removals per task and in the summary instead of letting a clean report hide leaked state. Merge with current main keeps --baseline alongside --allow-noncanonical-route. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Jonathan Edwards <justfinethanku@gmail.com>
macOS and Linux run the suite on every push/PR. The windows-latest job is continue-on-error until native-Windows portability PRs (invited in NateBJones-Projects#17) turn it green, then it flips to required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every merged community contributor must appear in README Contributors — tests/test_contributors.py audits git history (squash authors + merged-PR handles) against the section and fails on any omission; CI checks out full history for it. README refresh: all four merged contributors credited (NateBJones-Projects#24 NateBJones-Projects#30 #4 NateBJones-Projects#37 NateBJones-Projects#38), --baseline documented, canonical-route enforcement mentioned in the taxonomy section, CI badge added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verify outputs not confine workers (no safety theater), not a model-testing harness, honest displays, small scoped PRs, executed proof, Ringside UI innovation encouraged under the data contract, authorship preserved and credit test-enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Jones-Projects#77) * templates/bakeoff-kit: model-only research bakeoff starter kit The downloadable companion to the "How to Run a Chinese-Model Bakeoff in Ringer" guide. Distinct from templates/bakeoff/, which drives an external product and needs a session validator to prove model identity; this kit is the simpler model-only lane where the worker writes result.json directly. Contents: a stdlib-only validator (check_result.py) enforcing the eight rules the guide documents, a five-claim source packet built so one claim is genuinely unsupported and one is legitimately ambiguous, known-good and known-bad fixtures, a two-cell model-only manifest with per-task pinned models, and the accepted-result score sheet. Verified by execution, not inspection: the known-good fixture is accepted, the known-bad fixture is rejected naming all four defects the guide promises (invented quote, nonexistent source, invalid status, omitted claim), and an empty payload is rejected so the check cannot pass vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * bakeoff-kit: point the score sheet at the Ringside Models view Eight of score-sheet.csv's fourteen columns are already computed on the Models view in Ringside from the user's own attempt history. The kit now says so, and names the six that are genuinely manual — case type, Ringer commit, human repair minutes, provider charge, tool/infra cost, failure type. Reframes the sheet as the layer on top of the scoreboard (the money and the minutes) rather than a manual re-entry of the machine's own record. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…07-28 # Conflicts: # tests/test_design_reference.py # tests/test_scoreboard_page.py
…efault Per Ramy: drop the fork's worker-lane apparatus and use upstream's model configuration, with GPT-5.6 rather than GPT-5.5 as the model. - config.sample.toml: back to upstream verbatim — removes the fork-only [engines.sol] and [engines.luna] lanes, the codex model_default pin, and the model_reasoning_effort=medium arg. Live machine lanes are unaffected; they live in ~/.config/ringer/config.toml, outside the repo. - registry/model-identity.toml: upstream's file, with codex default_model_key = gpt-5.6-terra. Upstream already defines all three 5.6 models, so this is a one-line pointer change and drops the fork's 'GPT-5.5 (historical)' relabel. - README: removes the entry-environment contract section documenting the retired lanes. - test_model_db.py: back to upstream's fixture. - test_scoreboard_page.py: back to upstream. The fork's date-agnostic regex fixed a real rot in July, but upstream PR NateBJones-Projects#51 fixed the same rot by mocking the clock, which keeps the assertion exact. Upstream's is stronger. - test_contributors.py: fork maintainer and automation identities join the existing exclusion sets, mirroring how upstream excludes its own maintainers. Community PRs to the fork still must be credited. Verification: python3.12 -m unittest discover -s tests → 242 tests, OK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the fork up to date with
NateBJones-Projects/ringermain (2c2b599c), 12 commits behind at the start.What comes in from upstream
run --baseline— execute every check against the unmodified tree (Add run --baseline: execute every check against the unmodified tree NateBJones-Projects/ringer#38)templates/bakeoff-kit/— model-only research bakeoff starter kit (templates/bakeoff-kit: model-only research bakeoff starter kit NateBJones-Projects/ringer#77)CONTRIBUTING.md, the contributor-credit audit test, and the committed design-reference fixture (fix: commit design-reference fixture instead of hardcoded local path NateBJones-Projects/ringer#30)What the fork keeps
The TRIP lifecycle kit (
templates/trip-lifecycle/, its 363-linetrip_contract.pyand 555-line test suite) and the/bin/shpolyglot launcher inringer.py. The launcher is load-bearing on macOS, where the systempython3is 3.9 and upstream's#!/usr/bin/env python3fails ontomllib.Model configuration: upstream's, pointed at 5.6
Per the repo owner, the fork's worker-lane apparatus is retired in favour of upstream's configuration, with GPT-5.6 as the model instead of GPT-5.5:
config.sample.tomlreturns to upstream verbatim — the fork-only[engines.sol]and[engines.luna]lanes, the codexmodel_defaultpin, and themodel_reasoning_effort=mediumarg are all removed. Machine-local lanes are unaffected; they live in~/.config/ringer/config.toml, outside this repo.registry/model-identity.tomlis upstream's file with a single change: codexdefault_model_key = "gpt-5.6-terra". Upstream already defines all three 5.6 models, so keeping 5.6 costs one line, and the fork'sGPT-5.5 (historical)relabel is dropped.Conflicts
Two, both in tests, both resolved toward upstream:
tests/test_design_reference.py— upstream's fixture-driven token test replaces the fork's hard-coded duplicate of the same assertions, now that the fixture is committed.tests/test_scoreboard_page.py— reverted to upstream. The fork's date-agnostic regex fixed a real rot in July, but upstream's Self-update mechanism + scoreboard signal contract NateBJones-Projects/ringer#51 fixed the same rot by mocking the clock, which keeps the assertion exact rather than fuzzy. Upstream's is the stronger guard.One deliberate divergence
tests/test_contributors.py— the fork's maintainer and automation identities join the guard's existing exclusion sets, mirroring how upstream already excludes its own maintainers (jonathan edwards,nate jones) and bot prefixes. Without this the guard demands that the fork credit its own owner and CI bot as community contributors. Community PRs to the fork are still caught.Verification
Matches CI's invocation and Python version.
upstream/mainis confirmed an ancestor of this branch.Tracked as OPE-447.