Skip to content

Add GPT-5.6 capability sheets (sol/terra/luna)#46

Open
rafaelpini wants to merge 2 commits into
NateBJones-Projects:mainfrom
rafaelpini:contrib/gpt-5.6-capability-sheets
Open

Add GPT-5.6 capability sheets (sol/terra/luna)#46
rafaelpini wants to merge 2 commits into
NateBJones-Projects:mainfrom
rafaelpini:contrib/gpt-5.6-capability-sheets

Conversation

@rafaelpini

@rafaelpini rafaelpini commented Jul 15, 2026

Copy link
Copy Markdown

registry/model-identity.toml already registers the three GPT-5.6 tiers, but registry/model-capabilities/ has no sheets for them — gpt-5.5.toml is the newest OpenAI entry. This adds one sheet per tier, so the family Ringer can already name is also one it can reason about.

Each sheet follows the gpt-5.5.toml section layout (model / api / caching / tool_calling / reasoning / limits / openrouter / pricing / sources) and carries a [[sources]] entry per claim with URL, quote, and access date.

Sourcing

Conservative by design — where OpenAI doesn't document something, the sheet says so instead of guessing:

  • Pricing is as_of = 2026-07-16 from the OpenAI pricing page, including the explicit Cache writes column on the GPT-5.6 rows and the >272K-input 2x/1.5x rule. The prompt-caching guide's "cache writes are billed at 1.25× the uncached input token rate" statement and OpenRouter endpoint metadata (input_cache_write) corroborate the cache-write figures.
  • Reasoning effort scales are stated per surface, with quotes: the Codex CLI config reference types model_reasoning_effort as minimal|low|medium|high|xhigh; OpenRouter endpoint metadata lists supported_efforts [max, xhigh, high, medium, low, none] for all three tiers. OpenAI publishes no per-tier effort ceiling and the sheets make no per-tier exclusivity claim; a local harness observation that disagrees with the config reference lives in docs/MODEL-NOTES.md as a dated entry, not in the sheets.
  • GA date (2026-07-09) is third-party, single source. The model pages carry the Feb 16 2026 knowledge cutoff but no in-page GA date.
  • Long-context guidance for Luna lives in docs/MODEL-NOTES.md as a dated observation — the fact (no tier-by-tier long-context benchmark found as of 2026-07-14) and the routing judgment are explicitly separated — not in the sheet.

Test

tests/test_capability_sheets.py parses every sheet in registry/model-capabilities/, requires the standard sections on model sheets, a unique model.key, source records with a non-empty claim and quote, an http(s) URL and a valid ISO accessed date, and agreement between mirrored caching/pricing price fields when numeric. Passes against all existing sheets.

Verification

python3 -m unittest discover -s tests → 185 tests, same 3 pre-existing failures as a clean origin/main (test_design_reference ×2, test_scoreboard_page ×1 — the one tracked in #29). All registry-adjacent suites green; each sheet round-trips through tomllib.

Data files plus one new test — no code paths touched.

🤖 Generated with Claude Code

@justfinethanku

Copy link
Copy Markdown
Contributor

Thanks for this — capability sheets for the 5.6 tiers are exactly the kind of contribution this registry needs, and the layout/sourcing discipline (URLs, access dates, quotations) is spot on. We'd like to merge this once a few things are tightened up.

Some context on how these sheets get used, so the requested changes make sense: model selection in Ringer runs on local scoreboard evidence, not spec sheets. Sheets exist to give the orchestrator sourced operational facts — pricing, access routes, provenance, supported reasoning-effort levels, and tool-calling capabilities. They inform and filter; they never rank. Anything that reads as a recommendation or an unsourced number is signal pollution for that use.

Requested changes:

  1. Cache-write prices ($6.25/M Sol etc.): the cited pages support input/output/cache-read but not these. Please either add a source that states them explicitly or remove them — no extrapolated pricing.
  2. ultra mode: this is documented under reasoning as if it were an effort tier above max. Codex's ultra mode is its multi-agent orchestration feature, not a reasoning level — please remove it from the reasoning section (a sourced note about it under a separate key would be fine).
  3. Terra on OpenRouter: unsourced — add a source or drop the claim.
  4. Luna's limits.long_context_caveat: this is routing advice expressed as a capability fact, in an ad-hoc field no other sheet uses. Please remove it from the sheet — that kind of observed-behavior guidance belongs in docs/MODEL-NOTES.md as a dated observation, and we'd welcome it there.
  5. Tier framing: where the sheets say "flagship / deep-reasoning tier" etc., prefer the plain fact ("accepts low/medium/high/max reasoning effort via Codex") over the positioning language.

Nice to have, not blocking: a small test that parses every sheet in registry/model-capabilities/ and checks required sections, a unique model key, and that source records carry url + accessed date — it would protect your work (and future sheets) from drift.

Happy to answer questions on any of these. Once the unsourced items are fixed this is a clear merge.

@rafaelpini

rafaelpini commented Jul 16, 2026

Copy link
Copy Markdown
Author

All five addressed in ee7ca6e (rebased on current main per CONTRIBUTING.md — the conflict was two dated entries landing at the top of the same MODEL-NOTES section; both kept, newest first), plus the sheet-structure test. Two items came out stronger than requested once re-verified against live sources:

  1. Cache writes: the pricing page itself now settles this — the GPT-5.6 rows carry an explicit "Cache writes" column (Sol $6.25 / Terra $3.125 / Luna $1.25; older families show "-"), and the prompt-caching guide states the rule directly: "On GPT-5.6 models and later model families, cache writes are billed at 1.25× the uncached input token rate." Both quoted in the sheets (accessed 2026-07-16), with OpenRouter endpoint metadata (input_cache_write) as corroboration. Numbers unchanged, now explicitly sourced — this also let the sheets drop the search-relayed quote from the 403'd overview page in favor of the directly fetchable guide.
  2. ultra: removed. Re-verifying this surfaced that the max-tier claim didn't survive sourcing either: the Codex config reference enumerates minimal|low|medium|high|xhigh (no max), while OpenRouter lists supported_efforts including max for all three tiers. The reasoning sections now state the per-surface facts with quotes and make no per-tier exclusivity claim; the local harness observation (codex accepted max for Sol on one machine, 2026-07-13) moved to docs/MODEL-NOTES.md as a dated entry.
  3. Terra on OpenRouter: sourced via the OpenRouter models API (accessed 2026-07-16). All three tiers are listed there, so Sol and Luna carry the same sourced line.
  4. Luna long_context_caveat: removed from the sheet (field + its source record); added under ## gpt-5.6-luna in docs/MODEL-NOTES.md as a dated observation, with the fact (no tier-by-tier benchmark found) and the routing judgment explicitly separated.
  5. Tier framing: positioning language replaced with plain facts throughout — including comparative leftovers the first pass missed ("Sol's deeper reasoning…", a cross-tier price comparison in Luna's identity paragraph, and an endpoint inventory that had crept into Sol's streaming field).
  6. Test: tests/test_capability_sheets.py — every sheet parses; model sheets need the standard sections and a unique model.key; source records must carry a non-empty claim and quote, an http(s) url, and a valid ISO accessed date; mirrored caching/pricing price fields must agree when numeric. Platform sheets (no [model]) are an explicit allowlist, not inferred. Passes on all existing sheets.

One observation for your call, not acted on here: the kimi sheet's model.key (moonshotai/kimi-k2.7-code) lacks the openrouter/ prefix that both model-identity.toml and the glm sheet use. If you settle a key-naming convention, an identity-join assertion could be added to the test later.

Full suite: 185 tests, same 3 pre-existing failures as clean origin/main.

rafaelpini and others added 2 commits July 16, 2026 18:59
model-identity.toml already registers the three GPT-5.6 tiers, but
registry/model-capabilities/ had no sheets for them — gpt-5.5.toml was the
newest OpenAI entry. This fills that gap so the family Ringer can already
name is also one it can reason about.

Each sheet follows the gpt-5.5.toml section layout (model/api/caching/
tool_calling/reasoning/limits/openrouter/pricing/sources) and carries a
[[sources]] entry per claim with URL, quote, and access date.

Sourcing is deliberately conservative — where OpenAI does not document
something, the sheet says so rather than guessing:

- Per-tier reasoning effort ceilings (Sol alone exposing `max`/`ultra`) are
  marked harness-observed via the Codex CLI config surface, not OpenAI docs,
  which confirm reasoning token support but never enumerate the effort scale.
- Luna's [limits].long_context_caveat records that no OpenAI or third-party
  tier-by-tier long-context recall benchmark could be found (searched
  2026-07-14), so the conservative slot for Luna is short-context work. It is
  flagged as an operational caveat, explicitly not a sourced benchmark.
- The GA date (2026-07-09) is third-party; the model pages carry the Feb 16
  2026 knowledge cutoff but no in-page GA date.

Pricing is as_of 2026-07-14 from the OpenAI pricing page.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e, per-surface effort scales

- cache-write prices now cite the pricing page's Cache writes column and
  the prompt-caching guide's 1.25x rule directly (replaces the
  search-relayed quote from the 403'd overview page)
- reasoning effort scales stated per surface with quotes (Codex CLI enum
  minimal|low|medium|high|xhigh; OpenRouter supported_efforts incl. max on
  all three tiers); removed 'ultra' and the Sol-only 'max' claim, which
  both cited surfaces contradict — the harness observation moved to
  docs/MODEL-NOTES.md as a dated entry
- OpenRouter listings for all three tiers sourced via the models API
- Luna limits.long_context_caveat removed; dated observation added to
  docs/MODEL-NOTES.md with fact and routing judgment explicitly separated
- positioning language replaced with plain facts throughout
- tests/test_capability_sheets.py: every sheet parses; required sections;
  unique model.key; sources carry claim/quote/http(s) url/ISO accessed
  date; mirrored caching/pricing values must agree when numeric

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rafaelpini
rafaelpini force-pushed the contrib/gpt-5.6-capability-sheets branch from 10a443a to ee7ca6e Compare July 16, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants