Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/MODEL-NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,13 @@ checks and raw logs support — no vibes, no worker self-reports.

## gpt-5.6-sol (codex)
- 2026-07-15 ringer-self-update run (3 serial tasks, direct-repo-edit mode): code-fix baseline-test repair 1/1 first-try (61k tokens, 1.6m); code-feature self-update mechanism (git fetch/ff-pull/re-exec + HUD staleness restart + 20-test suite) 1/1 first-try at high effort (153k, 8.1m); code-feature signal-contract (all 3 scoreboard surfaces + canonical-route lint enforcement) passed on retry (358k, 13.7m) — attempt 1 died on stale old-column assertions in pre-existing tests it hadn't finished updating; the retry prompt's injected FAIL list was enough to close it out. Lesson: when a task rewrites a display contract, name every test file asserting the old contract in the spec's ownership list AND tell it to update them FIRST.
- 2026-07-13 reasoning-effort surfaces (dated observation): codex CLI on this
box accepted `model_reasoning_effort=max` for gpt-5.6-sol. As of 2026-07-16
the Codex config reference types the field `minimal|low|medium|high|xhigh`
(no `max`), while OpenRouter metadata lists supported_efforts including
`max` for ALL three 5.6 tiers. The surfaces disagree — per-tier effort
ceilings are unsettled; the capability sheets carry only the per-surface
documented facts.
- 2026-07-09 code-feature/code-fix (ringside-overhaul): 4/4 first-try — a ringer.py logging change with tests, a 265-line stdlib backfill CLI (atomic rewrite, dry-run, idempotence all check-verified), a ~1500-line single-file HTML redesign (running-now pills + worker-card grid + multi-expansion refactor, 30KB patch, node --check + contract greps + unittest), and a render-gating change where it correctly UPDATED tests asserting the old behavior instead of gaming the check. Medium/high reasoning, 65–120k tokens/task.
- Same day, different session (bench-harness-patches, code-fix): 0.29 first-try over 7 tasks on a Next.js/Turbopack harness. Spec and check quality dominate model choice — see the scoreboard before generalizing either number.

Expand All @@ -309,6 +316,13 @@ checks and raw logs support — no vibes, no worker self-reports.

## gpt-5.6-luna (codex)
- 2026-07-09 code-feature (unlock-ai guide-format conversion, strict type-contract check): 1/1 first-try, 42.6k tokens, 80s. Followed a multi-file TS pattern precisely at $1/$6 pricing. Good candidate for mechanical codegen/docs lanes; audition in adjacent types.
- 2026-07-14 long-context (dated observation, no run): Luna's nominal window
is 1.05M, same as Sol/Terra, but a search that day found no tier-by-tier
long-context recall benchmark from OpenAI or any third party — per-tier
long-context behavior is unverified in both directions. Routing judgment,
not scoreboard evidence: until run evidence accumulates, the conservative
slot for Luna is short-context, low-stakes, latency-sensitive work; treat
long-context Luna assignments as exploration, not proven routing.

## opencode / z-ai glm-5.2 (via openrouter)
- 2026-07-09 (aicred-invoice-downloads, 4 code-fix tasks + 1 follow-up, worktrees+npm ci checks): systematic attempt-1 NO-OP — all 4 parallel workers produced zero edits and no summary on first attempt, then completed cleanly on attempt 2 after retry-prompt injection (34k-69k tokens each). Follow-up single task passed attempt 1. Suspect first-invocation session warm-up in opencode-sandboxed under parallel spawn; budget for 2 attempts on parallel GLM batches. Output quality on Next.js/Stripe route+test work: solid, spec-faithful, one boss-caught design gap (used user-scoped supabase client where RLS demanded service role — spec didn't say explicitly; say it explicitly).
139 changes: 139 additions & 0 deletions registry/model-capabilities/gpt-5.6-luna.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
[model]
key = "gpt-5.6-luna"
display = "GPT-5.6 Luna"
vendor = "OpenAI"
release_date = "2026-07-09"
identity_verification = "OpenAI's developer model page for gpt-5.6-luna lists it as a member of the GPT-5.6 family (Sol/Terra/Luna) with a 1,050,000 context window, 128,000 max output tokens, reasoning token support, and a Feb 16, 2026 knowledge cutoff (fine-tuning listed as Not supported). GA of the GPT-5.6 family is reported as 2026-07-09 by third-party coverage (single source); the model page carries the Feb 16 2026 knowledge cutoff but no in-page GA date."

[api]
endpoint_families = ["responses", "chat-completions", "batch"]
auth = "In this setup, Codex CLI signs in with ChatGPT/OAuth plan credentials and uses ChatGPT workspace permissions, plan access, and ChatGPT-managed Codex entitlements. Codex CLI can also sign in with an OpenAI API key; API-key usage is billed through the OpenAI Platform account at standard API rates instead of included ChatGPT plan credits."
streaming = "Supported. API streaming is available with stream=true on Responses and Chat Completions; streaming responses use server-sent events. Family-level (GPT-5.6) behavior; the per-tier model page does not separately enumerate a streaming flag."

[caching]
supported = true
mode = "implicit"
how_enabled = "Prompt caching works automatically for eligible requests with no code changes; prompt_cache_options.mode defaults to implicit (the mode field records this default). For GPT-5.6, explicit cache breakpoints are additionally available in both the Responses and Chat Completions APIs, and prompt_cache_key must be set to get the more reliable cache matching. Caching starts at 1024 prompt tokens (cached_tokens is zero for requests under 1024); repeated prompt prefixes, tools, and structured-output schemas can contribute to the cached prefix."
ttl = "For GPT-5.6, prompt_cache_options.ttl sets a MINIMUM cache lifetime; the only supported value is 30m (also the default) — a cached prefix remains eligible for reuse for at least 30 minutes and may be retained longer. The prompt_cache_retention field used by earlier families (24h on GPT-5.5) is deprecated for GPT-5.6 and later."
read_pricing_per_m = 0.10
write_pricing_per_m = 1.25
min_cacheable_prefix_tokens = 1024

[tool_calling]
format = "OpenAI Responses API tools and function calls with JSON Schema tool parameters. Chat Completions function calling is also supported. Structured Outputs are supported, including strict JSON Schema mode with strict=true. Family-level GPT-5.6 behavior inherited from the OpenAI platform; not separately re-verified on the per-tier model page."
parallel = true
strict_json_schema = true
quirks = [
"For reasoning models using function calling through the Responses API, pass back reasoning items, function call items, and function call output items during tool loops.",
"Prompt-only JSON mode can still produce incomplete JSON when output is cut off by max_output_tokens or a content filter; use Structured Outputs for schema adherence.",
]

[reasoning]
params = ["reasoning.effort", "text.verbosity", "max_output_tokens", "model_reasoning_effort", "model_reasoning_summary", "model_verbosity"]
defaults = "The gpt-5.6-luna model page confirms reasoning token support but does not enumerate an effort scale. Effort scales are surface-specific: the Codex CLI config reference types model_reasoning_effort as minimal|low|medium|high|xhigh (xhigh is model-dependent), and OpenRouter endpoint metadata lists supported_efforts [max, xhigh, high, medium, low, none] with default medium for openai/gpt-5.6-luna. OpenAI publishes no per-tier effort ceiling; Ringer passes effort per-task via engine_args [\"-c\",\"model_reasoning_effort=...\"]."
max_output_interaction = "max_output_tokens limits total generated tokens, including reasoning tokens, visible output tokens, and non-visible formatting tokens. If the cap is exhausted during reasoning, the response can be incomplete before any visible output is produced."

[limits]
context_window = 1050000
max_output_tokens = 128000
rate_limits = "Prompts with >272K input tokens are priced at 2x input and 1.5x output for the full request (per the gpt-5.6-luna model page). ChatGPT-plan Codex OAuth usage is governed by plan entitlements, not API billing tiers; per-tier long-context API rate-limit tiers are not published on the model page."

[openrouter]
upstream_providers = ["openai"]
require_parameters_behavior = "not applicable; this registry entry is for OpenAI direct/Codex OAuth access. gpt-5.6-luna is also listed on OpenRouter as openai/gpt-5.6-luna (see sources); OpenRouter routing would pass the model slug through the opencode engine's model field."

[pricing]
prompt_per_m = 1.00
completion_per_m = 6.00
cache_read_per_m = 0.10
cache_write_per_m = 1.25
as_of = "2026-07-16"

[[sources]]
claim = "model: gpt-5.6-luna context window"
url = "https://developers.openai.com/api/docs/models/gpt-5.6-luna"
accessed = "2026-07-14"
quote = "1,050,000 context window"

[[sources]]
claim = "model: gpt-5.6-luna max output tokens"
url = "https://developers.openai.com/api/docs/models/gpt-5.6-luna"
accessed = "2026-07-14"
quote = "128,000 max output tokens"

[[sources]]
claim = "pricing: gpt-5.6-luna input, cached input, cache writes, and output per 1M"
url = "https://developers.openai.com/api/docs/pricing"
accessed = "2026-07-16"
quote = "Input Cached input Cache writes Output ... gpt-5.6-luna $1.00 $0.10 $1.25 $6.00"
note = "The pricing page's GPT-5.6 rows carry an explicit Cache writes column (older families show '-'). The $1.25/M cache-write price is independently corroborated by OpenRouter endpoint metadata (input_cache_write = 0.00000125 for openai/gpt-5.6-luna, accessed 2026-07-16)."

[[sources]]
claim = "limits: gpt-5.6-luna large-prompt (>272K) pricing rule"
url = "https://developers.openai.com/api/docs/models/gpt-5.6-luna"
accessed = "2026-07-14"
quote = "Prompts with >272K input tokens are priced at 2x input and 1.5x output for the full request"

[[sources]]
claim = "model: gpt-5.6-luna knowledge cutoff"
url = "https://developers.openai.com/api/docs/models/gpt-5.6-luna"
accessed = "2026-07-14"
quote = "Feb 16, 2026 knowledge cutoff"

[[sources]]
claim = "model: gpt-5.6-luna fine-tuning not supported"
url = "https://developers.openai.com/api/docs/models/gpt-5.6-luna"
accessed = "2026-07-16"
quote = "Fine-tuning Not supported"

[[sources]]
claim = "reasoning: Codex CLI reasoning-effort scale"
url = "https://developers.openai.com/codex/config-reference"
accessed = "2026-07-16"
quote = "minimal | low | medium | high | xhigh"
note = "The config reference types model_reasoning_effort with this enum ('xhigh is model-dependent') and does not include 'max'."

[[sources]]
claim = "reasoning: OpenRouter supported reasoning efforts for openai/gpt-5.6-luna"
url = "https://openrouter.ai/api/v1/models"
accessed = "2026-07-16"
quote = "\"supported_efforts\":[\"max\",\"xhigh\",\"high\",\"medium\",\"low\",\"none\"],\"default_effort\":\"medium\""

[[sources]]
claim = "tool_calling: strict structured outputs use JSON Schema strict true (platform, version-independent)"
url = "https://developers.openai.com/api/docs/guides/structured-outputs"
accessed = "2026-07-16"
quote = "strict: true"

[[sources]]
claim = "caching: GPT-5.6 cache writes are billed at 1.25x the uncached input rate"
url = "https://developers.openai.com/api/docs/guides/prompt-caching"
accessed = "2026-07-16"
quote = "On GPT-5.6 models and later model families, cache writes are billed at 1.25× the uncached input token rate and reported in `cache_write_tokens`."
note = "Direct OpenAI source (replaces an earlier search-relayed quote from openai.com/index/gpt-5-6, which 403s to direct fetch). 1.25x of $1.00 input = $1.25/M, matching the pricing page's Cache writes column."

[[sources]]
claim = "caching: explicit cache breakpoints and 30m minimum cache life"
url = "https://developers.openai.com/api/docs/guides/prompt-caching"
accessed = "2026-07-16"
quote = "use `prompt_cache_options.ttl` to set the minimum lifetime of all breakpoints written by the request. The only supported value is `30m`, which is also the default."
note = "The guide also states: 'you can mark the end of a reusable prompt prefix with an explicit cache breakpoint. Breakpoints are available in both the Responses API and Chat Completions API.', documents `prompt_cache_options.mode` with implicit as the default, requires prompt_cache_key for the more reliable matching, and deprecates prompt_cache_retention for GPT-5.6 and later."

[[sources]]
claim = "caching: 1024-token floor for cached_tokens"
url = "https://developers.openai.com/api/docs/guides/prompt-caching"
accessed = "2026-07-16"
quote = "For requests under 1024 tokens, `cached_tokens` is zero."

[[sources]]
claim = "release: GPT-5.6 family (Sol/Terra/Luna) GA date (third-party, single source)"
url = "https://www.digitalapplied.com/blog/gpt-5-6-sol-terra-luna-public-ga"
accessed = "2026-07-14"
quote = "GPT-5.6 went to general availability on July 9, 2026 — across ChatGPT, Codex, and the OpenAI API"
note = "release_date records the public GA date, not first availability (pre-GA access via Codex existed for the family; see docs/MODEL-NOTES.md, 'GPT-5.5 (codex) — attribution caveat'). Do not use release_date as a lower bound for scoreboard row attribution."

[[sources]]
claim = "openrouter: gpt-5.6-luna listing and cache-write price metadata"
url = "https://openrouter.ai/api/v1/models"
accessed = "2026-07-16"
quote = "\"id\": \"openai/gpt-5.6-luna\", \"name\": \"OpenAI: GPT-5.6 Luna\" ... \"input_cache_read\": \"0.0000001\", \"input_cache_write\": \"0.00000125\""
Loading