Add GPT-5.6 capability sheets (sol/terra/luna)#46
Conversation
|
Thanks for this — capability sheets for the 5.6 tiers are exactly the kind of contribution this registry needs, and the layout/sourcing discipline (URLs, access dates, quotations) is spot on. We'd like to merge this once a few things are tightened up. Some context on how these sheets get used, so the requested changes make sense: model selection in Ringer runs on local scoreboard evidence, not spec sheets. Sheets exist to give the orchestrator sourced operational facts — pricing, access routes, provenance, supported reasoning-effort levels, and tool-calling capabilities. They inform and filter; they never rank. Anything that reads as a recommendation or an unsourced number is signal pollution for that use. Requested changes:
Nice to have, not blocking: a small test that parses every sheet in Happy to answer questions on any of these. Once the unsourced items are fixed this is a clear merge. |
|
All five addressed in ee7ca6e (rebased on current main per CONTRIBUTING.md — the conflict was two dated entries landing at the top of the same MODEL-NOTES section; both kept, newest first), plus the sheet-structure test. Two items came out stronger than requested once re-verified against live sources:
One observation for your call, not acted on here: the kimi sheet's Full suite: 185 tests, same 3 pre-existing failures as clean |
model-identity.toml already registers the three GPT-5.6 tiers, but registry/model-capabilities/ had no sheets for them — gpt-5.5.toml was the newest OpenAI entry. This fills that gap so the family Ringer can already name is also one it can reason about. Each sheet follows the gpt-5.5.toml section layout (model/api/caching/ tool_calling/reasoning/limits/openrouter/pricing/sources) and carries a [[sources]] entry per claim with URL, quote, and access date. Sourcing is deliberately conservative — where OpenAI does not document something, the sheet says so rather than guessing: - Per-tier reasoning effort ceilings (Sol alone exposing `max`/`ultra`) are marked harness-observed via the Codex CLI config surface, not OpenAI docs, which confirm reasoning token support but never enumerate the effort scale. - Luna's [limits].long_context_caveat records that no OpenAI or third-party tier-by-tier long-context recall benchmark could be found (searched 2026-07-14), so the conservative slot for Luna is short-context work. It is flagged as an operational caveat, explicitly not a sourced benchmark. - The GA date (2026-07-09) is third-party; the model pages carry the Feb 16 2026 knowledge cutoff but no in-page GA date. Pricing is as_of 2026-07-14 from the OpenAI pricing page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e, per-surface effort scales - cache-write prices now cite the pricing page's Cache writes column and the prompt-caching guide's 1.25x rule directly (replaces the search-relayed quote from the 403'd overview page) - reasoning effort scales stated per surface with quotes (Codex CLI enum minimal|low|medium|high|xhigh; OpenRouter supported_efforts incl. max on all three tiers); removed 'ultra' and the Sol-only 'max' claim, which both cited surfaces contradict — the harness observation moved to docs/MODEL-NOTES.md as a dated entry - OpenRouter listings for all three tiers sourced via the models API - Luna limits.long_context_caveat removed; dated observation added to docs/MODEL-NOTES.md with fact and routing judgment explicitly separated - positioning language replaced with plain facts throughout - tests/test_capability_sheets.py: every sheet parses; required sections; unique model.key; sources carry claim/quote/http(s) url/ISO accessed date; mirrored caching/pricing values must agree when numeric Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
10a443a to
ee7ca6e
Compare
registry/model-identity.tomlalready registers the three GPT-5.6 tiers, butregistry/model-capabilities/has no sheets for them —gpt-5.5.tomlis the newest OpenAI entry. This adds one sheet per tier, so the family Ringer can already name is also one it can reason about.Each sheet follows the
gpt-5.5.tomlsection layout (model/api/caching/tool_calling/reasoning/limits/openrouter/pricing/sources) and carries a[[sources]]entry per claim with URL, quote, and access date.Sourcing
Conservative by design — where OpenAI doesn't document something, the sheet says so instead of guessing:
as_of = 2026-07-16from the OpenAI pricing page, including the explicit Cache writes column on the GPT-5.6 rows and the >272K-input 2x/1.5x rule. The prompt-caching guide's "cache writes are billed at 1.25× the uncached input token rate" statement and OpenRouter endpoint metadata (input_cache_write) corroborate the cache-write figures.model_reasoning_effortasminimal|low|medium|high|xhigh; OpenRouter endpoint metadata listssupported_efforts[max, xhigh, high, medium, low, none]for all three tiers. OpenAI publishes no per-tier effort ceiling and the sheets make no per-tier exclusivity claim; a local harness observation that disagrees with the config reference lives indocs/MODEL-NOTES.mdas a dated entry, not in the sheets.docs/MODEL-NOTES.mdas a dated observation — the fact (no tier-by-tier long-context benchmark found as of 2026-07-14) and the routing judgment are explicitly separated — not in the sheet.Test
tests/test_capability_sheets.pyparses every sheet inregistry/model-capabilities/, requires the standard sections on model sheets, a uniquemodel.key, source records with a non-empty claim and quote, an http(s) URL and a valid ISO accessed date, and agreement between mirrored caching/pricing price fields when numeric. Passes against all existing sheets.Verification
python3 -m unittest discover -s tests→ 185 tests, same 3 pre-existing failures as a cleanorigin/main(test_design_reference×2,test_scoreboard_page×1 — the one tracked in #29). All registry-adjacent suites green; each sheet round-trips throughtomllib.Data files plus one new test — no code paths touched.
🤖 Generated with Claude Code