fix(eval): agent cost was the wrong model's price, and the budget was the wrong unit

changelogs/v0.18-current.md

@@ -2,6 +2,129 @@
 
 > Current changelog (v0.18.0 onward). Earlier series: [v0.10–v0.17](v0.10-v0.17-bytecode-vm.md). Index: [CHANGELOG.md](../CHANGELOG.md).
 
+## [Unreleased]
+
+### Added
+
+- `eval-suite` now prints a **cost tally** when a run finishes — per model/mode,
+  then totals split by provenance. The split is the point: a single grand total
+  is misleading on a rig where agent-mode `codex` and `claude` authenticate by
+  subscription, so their `cost_usd` is real arithmetic over real tokens that
+  nobody paid. `METERED (actually billed)` is the only line that answers "what
+  did this cost us"; list-price-equivalent and unlabelled rows are surfaced
+  alongside it and never folded in. Best-effort by construction — a malformed
+  result file is skipped and a cost report never fails an otherwise-complete run
+  (`eval_harness.TallyCosts`).
+
+### Changed
+
+- Agent-eval budgets are now gated on **work**, not dollars. A dollar ceiling
+  buys tokens in inverse proportion to model price, so the nominally-uniform
+  `$0.30`/`$0.50` caps across `agent_suite` spanned **24x** in actual work
+  (audited 2026-07-31): `opencode-or-deepseek-v4-flash` got 3.40M tokens while
+  `claude-sonnet-4-6` — the longitudinal **anchor** every other model is scored
+  against — got 0.14M, the tightest gate in the suite. It had no `budgets:`
+  block at all, so it fell through to the default formula, which at `$3/$15`
+  buys almost nothing. A suite asking "does the agent loop rescue weak models"
+  was handing the weakest models the most iteration and the reference the least.
+
+  `budgets.max_tokens_per_bench` (new, per-model, overrides the global
+  `--max-tokens-per-bench` flag) sets one shared **3.0M blended-token** ceiling
+  across the suite. Subscription lanes — `codex` on `auth_mode: chatgpt`,
+  `claude` on OAuth — gate on tokens alone and now sit at an identical 3.00M;
+  their dollar caps were raised purely so the token gate is the one that binds
+  (`claude-sonnet-4-6` `$0.50` → `$10.80`, `claude-haiku-4-5` `$0.30` → `$3.60`,
+  `gpt5-6-luna` `$0.30` → `$0.75`). This is **not** a spend increase: those lanes
+  are subscription-authenticated, so the figure is a list-price equivalent.
+  ⚠️ Under `AILANG_AUTH_MODE=apikey` (cloud dispatch) `claude` *is* metered and
+  those numbers become a real per-benchmark exposure ceiling — noted inline.
+
+  Metered `opencode` lanes keep their existing dollar ceiling as a genuine spend
+  control and take the token gate as a *second* bound, whichever binds first.
+  Their comparison stays cost-limited, but now for a stated reason rather than
+  by accident. Net effect: the metered lane that was over-favoured
+  (`deepseek-v4-flash`, 3.40M) is pulled into line at 3.00M with no spend change.
+
+- `Task.MaxTokensPerBench` is now enforced by `codex` and `claude`. The field has
+  been plumbed to `executor.Task` since M-EVAL-OS-LONGITUDINAL Phase 1, but only
+  `opencode` ever honoured it — the other executors silently ignored it. Both now
+  abort on the cumulative-token ceiling, report `ThrashKilledAt`, and set
+  `FinishThrashAborted`, which outranks `FinishCostExhausted` because on a
+  subscription lane the dollar figure is notional and "did too much work" is the
+  truthful reason a run stopped.
+
+### Added
+
+- Cost provenance: every cost figure now records whether anyone was actually
+  charged. `executor.CostProvenance` is one of `metered` (API key, OpenRouter
+  credits, Vertex ADC), `list-price-equivalent` (real arithmetic over real
+  tokens on a subscription/OAuth lane — never billed), `free-local` (on-device,
+  no marginal cost), or `unknown`. Each executor classifies its own auth lane:
+  `codex` reads `~/.codex/auth.json` (`auth_mode: chatgpt` → subscription),
+  `claude` mirrors the `AILANG_AUTH_MODE` branch, `managed_agents` is metered
+  via ADC, and `opencode`/`motoko`/`pi` report provider-billed cost. Zero
+  resolved rates win over the lane, so a free local model is never labelled as
+  spend. Standard mode reaches metered HTTP APIs and is labelled accordingly.
+
+  This closes the gap where a subscription run and a metered run were
+  indistinguishable once both landed in the same column — the v1.0
+  `cost-per-verified-success` KPI defines its numerator as attributable
+  **metered** dollars, and on this rig most agent-mode dollars are not that.
+
+  Surfaced end to end: banked as `cost_provenance` on eval result rows; stored
+  on chain stages (schema v17, `chain_stages.cost_provenance`); classified by
+  `ClassifyStageCost` into a new `subscription` status that `CostRollup` keeps
+  **out** of `TotalKnownCost` and reports separately; and exported as a
+  `costProvenance` breakdown alongside `totalCostUSD`. An absent label — every
+  row banked before 2026-07-30 — reads as `unknown`, never as metered; no
+  provenance is backfilled by guess. `dollars_per_pass`, `p90_cost_per_success`
+  and `cost_overhead_vs_best` are documented as list-price figures rather than
+  silently re-scoped.
+
+### Fixed
+
+- Agent-mode `cost_usd` now bills the model that actually ran. Executors that
+  compute cost from token counts were using their own hardcoded table — the
+  codex executor billed **every** model it ran at gpt-5-codex's `$1.25/$10` per
+  1M, so `gpt5-6-luna` and `gpt5-4-mini` were banked at identical rates despite
+  really costing `$0.20/$1.20` and `$0.75/$4.50`. `Task.Pricing` now carries the
+  per-model rates from `models.yml` and `executor.ResolveCostModel` prefers them,
+  falling back to the executor's table only when a caller supplies none (a
+  present-but-zero rate is honoured, so free local Ollama models are not billed
+  at cloud prices). Applied to the two executors that compute cost — `codex` and
+  `managed_agents`; `claude`, `pi`, `motoko` and `opencode` bank the figure their
+  CLI reports and were never affected, and their `CostModel()` doc comments now
+  say so.
+
+  The cost-**kill** path was already correct — `executor.CostBudget` is built
+  from `models.yml` pricing in `agent_runner_multi.go` — which means banked cost
+  and kill threshold were computed from two different price tables. A v0.30.0
+  row shows the split directly: `gpt5-6-luna`/`graph_bfs` banked `$0.34259` at
+  codex rates while the `$0.30` budget that spared it saw `$0.26980` at luna's.
+  Do not compare `cost_usd` against `max_cost_usd` on any pre-fix agent row.
+
+- `codex` HealthCheck no longer treats a missing `OPENAI_API_KEY` as a warning.
+  Codex authenticates from `~/.codex/auth.json` (`codex login`); an env key does
+  not override it — probe-verified against codex-cli 0.145.0 with
+  `auth_mode: "chatgpt"`, where a deliberately invalid key in the environment
+  still ran clean. Its presence was never evidence that runs are metered.
+
+### Changed
+
+- Eval pricing: OpenAI cut the GPT-5.6 family on 2026-07-30 and `models.yml` now
+  matches. `gpt5-6-luna` -80% (`$1.00/$6.00` → `$0.20/$1.20` per 1M) and
+  `gpt5-6-terra` -20% (`$2.50/$15.00` → `$2.00/$12.00` per 1M); `gpt5-6-sol` is
+  unchanged at `$5/$30`. Verified against
+  `developers.openai.com/api/docs/models/gpt-5.6-{sol,terra,luna}`. Two
+  consequences are recorded inline: Luna's explicit `max_cost_usd: $0.30` is now
+  load-bearing (the default formula resolves to `$0.051` at the new prices, down
+  from `$0.256`), and every banked Terra/Luna cost figure dated before 2026-07-30
+  was measured at the old prices, so cross-boundary cost comparisons need
+  normalizing. Luna is now 3.75x cheaper than the `gpt5-4-mini` incumbent it lost
+  to on 2026-07-11 — the quality half of that gate (23/26 vs 25/26, N=1) still
+  stands, so the `dev_models`/`lang_harness_suite` slot is unchanged pending an
+  N≥3 `--tier core` re-gate.
+
 ## [v0.31.0] - 2026-07-29
 
 ### Added

cmd/ailang/eval_benchmark.go

@@ -291,6 +291,10 @@ func runSingleBenchmark(ctx context.Context, model, benchmarkID, lang, condition
 			// NOT recomputed from tokens: agent CLIs report their own billed cost,
 			// which already includes reasoning. Deriving it here would double-count.
 			CostUSD: result.Cost,
+			// Whether that cost was actually billed. The rig's codex and claude
+			// lanes authenticate by subscription, so a non-zero cost there is a
+			// list-price equivalent, not spend (see executor.CostProvenance).
+			CostProvenance: result.CostProvenance,
 			// Use standard validation fields from agent runner
 			CompileOk:  result.CompileOk,
 			RuntimeOk:  result.RuntimeOk,
@@ -417,7 +421,7 @@ func runSingleBenchmark(ctx context.Context, model, benchmarkID, lang, condition
 			_ = evalChain.Store.UpdateStageEvalAssessment(ctx, stageID, assessment)
 
 			tokensIn := result.Usage.InputTokens + result.Usage.CacheCreationInputTokens + result.Usage.CacheReadInputTokens
-			_ = evalChain.Store.UpdateStageMetrics(ctx, stageID, result.Cost, tokensIn, result.Usage.OutputTokens, result.NumTurns, result.ToolCallCount, int64(result.DurationMS))
+			_ = evalChain.Store.UpdateStageMetrics(ctx, stageID, result.Cost, tokensIn, result.Usage.OutputTokens, result.NumTurns, result.ToolCallCount, int64(result.DurationMS), result.CostProvenance)
 
 			stageStatus := observatory.StageStatusCompleted
 			if !result.Success {
@@ -693,7 +697,7 @@ func runSingleBenchmark(ctx context.Context, model, benchmarkID, lang, condition
 		}
 		_ = evalChain.Store.UpdateStageEvalAssessment(ctx, stageID, assessment)
 		_ = evalChain.Store.UpdateStageMetrics(ctx, stageID, metrics.CostUSD,
-			metrics.InputTokens, metrics.OutputTokens, 0, 0, metrics.DurationMs)
+			metrics.InputTokens, metrics.OutputTokens, 0, 0, metrics.DurationMs, metrics.CostProvenance)
 
 		stageStatus := observatory.StageStatusCompleted
 		if !metrics.StdoutOk {

cmd/ailang/eval_suite_finalize.go

@@ -79,6 +79,17 @@ func finalizeSuiteRun(p suiteSummaryParams) {
 	}
 	fmt.Println()
 
+	// Cost tally, split by whether the dollars were actually charged. A single
+	// grand total would be misleading here: agent-mode codex and claude run on
+	// subscriptions, so their cost_usd is real arithmetic over real tokens that
+	// nobody paid. Best-effort — a cost report must never fail a completed run.
+	if tally, err := eval_harness.TallyCosts(p.outputDir); err == nil {
+		if s := tally.Format(); s != "" {
+			fmt.Print(s)
+			fmt.Println()
+		}
+	}
+
 	// M-EVAL-OS-LONGITUDINAL Phase 3: write summary.json that aggregates
 	// per-(benchmark, model, lang, condition) pass rate and token distribution
 	// across trials. Required for Phase 4 candidates command + Phase 5

internal/coordinator/daemon_tasks_chain.go

@@ -48,7 +48,7 @@ func (d *Daemon) updateStageMetrics(ctx context.Context, task *TaskRecord, resul
 		return
 	}
 	durationMs := result.Duration.Milliseconds()
-	if err := d.obsBackend.UpdateStageMetrics(ctx, task.StageID, result.Cost, result.InputTokens, result.OutputTokens, result.NumTurns, result.ToolCallCount, durationMs); err != nil {
+	if err := d.obsBackend.UpdateStageMetrics(ctx, task.StageID, result.Cost, result.InputTokens, result.OutputTokens, result.NumTurns, result.ToolCallCount, durationMs, result.CostProvenance); err != nil {
 		d.logger.Printf("Warning: Failed to update stage %s metrics: %v", task.StageID, err)
 	}
 }

internal/coordinator/provider.go

@@ -105,13 +105,18 @@ func DefaultExecuteOptions() *ExecuteOptions {
 
 // ExecuteResult contains the result of task execution
 type ExecuteResult struct {
-	Success    bool
-	Output     string
-	Error      string
-	Provider   string
-	Duration   time.Duration
-	Cost       float64
-	TokensUsed int // Total tokens (InputTokens + OutputTokens)
+	Success  bool
+	Output   string
+	Error    string
+	Provider string
+	Duration time.Duration
+	Cost     float64
+	// CostProvenance says whether Cost was actually billed — "metered",
+	// "list-price-equivalent" (subscription lane), "free-local", or "" for
+	// unclassified. Carried through to the chain stage so cost rollups can
+	// keep notional and real dollars apart.
+	CostProvenance string
+	TokensUsed     int // Total tokens (InputTokens + OutputTokens)
 	// Detailed token breakdown
 	InputTokens  int
 	OutputTokens int

internal/coordinator/provider_executor.go

@@ -144,6 +144,7 @@ func (p *ExecutorProvider) Execute(ctx context.Context, task *AnalyzedTask, opts
 	result.Output = execResult.Output
 	result.Error = execResult.Error
 	result.Cost = execResult.CostUSD
+	result.CostProvenance = string(execResult.CostProvenance)
 	result.InputTokens = execResult.InputTokens
 	result.OutputTokens = execResult.OutputTokens
 	result.TokensUsed = execResult.InputTokens + execResult.OutputTokens

internal/eval_analysis/export_json.go

@@ -130,7 +130,11 @@ func ExportBenchmarkJSON(matrix *PerformanceMatrix, history []*Baseline, results
 		"cacheCreationTokens": matrix.Aggregates.CacheCreationTokens,
 		"cacheHitRate":        matrix.Aggregates.CacheHitRate,
 		"totalCostUSD":        matrix.Aggregates.TotalCostUSD,
-		"avgDurationMs":       matrix.Aggregates.AvgDurationMs,
+		// How those dollars arose. Absent/"unknown" dominates pre-2026-07-30
+		// baselines; a large list-price-equivalent share means the cohort's cost
+		// column is notional, not spend.
+		"costProvenance": matrix.Aggregates.CostProvenance,
+		"avgDurationMs":  matrix.Aggregates.AvgDurationMs,
 		// Agent metrics (M-EVAL-AGENT)
 		"agentRuns":        len(agentResults),
 		"agentSuccessRate": agentSuccessRate,

internal/eval_analysis/loader_chains_test.go

@@ -129,7 +129,7 @@ func TestLoadResultsFromChain(t *testing.T) {
 		if err := store.UpdateStageEvalAssessment(ctx, stage.ID, assessment); err != nil {
 			t.Fatalf("failed to update assessment: %v", err)
 		}
-		if err := store.UpdateStageMetrics(ctx, stage.ID, 0.003, 500, 200, 3, 5, 8000); err != nil {
+		if err := store.UpdateStageMetrics(ctx, stage.ID, 0.003, 500, 200, 3, 5, 8000, ""); err != nil {
 			t.Fatalf("failed to update metrics: %v", err)
 		}
 	}

internal/eval_analysis/matrix.go

@@ -58,6 +58,7 @@ func calculateAggregates(results []*BenchmarkResult) Aggregates {
 	totalDuration := int64(0)
 	cacheRead := 0
 	cacheCreate := 0
+	costProvenance := map[string]int{}
 
 	for _, r := range results {
 		if r.FirstAttemptOk {
@@ -78,6 +79,13 @@ func calculateAggregates(results []*BenchmarkResult) Aggregates {
 		totalDuration += r.DurationMs
 		cacheRead += r.CacheReadInputTokens
 		cacheCreate += r.CacheCreationInputTokens
+
+		// An absent label is unknown provenance, never an assumption of metering.
+		prov := r.CostProvenance
+		if prov == "" {
+			prov = "unknown"
+		}
+		costProvenance[prov]++
 	}
 
 	agg.ZeroShotSuccess = safeDiv(float64(firstAttemptSuccess), float64(len(results)))
@@ -86,6 +94,7 @@ func calculateAggregates(results []*BenchmarkResult) Aggregates {
 	agg.RepairSuccessRate = safeDiv(float64(repairSuccess), float64(repairUsed))
 	agg.TotalTokens = totalTokens
 	agg.TotalCostUSD = totalCost
+	agg.CostProvenance = costProvenance
 	agg.AvgDurationMs = safeDiv(float64(totalDuration), float64(len(results)))
 	agg.CacheReadTokens = cacheRead
 	agg.CacheCreationTokens = cacheCreate

internal/eval_analysis/sweet_spot.go

@@ -56,12 +56,18 @@ type SweetSpotRow struct {
 	// Efficiency (mirrors EfficiencyAggregates, but flattened for table output).
 	MedianTTSMs        float64 `json:"median_tts_ms"`
 	MedianTokensPerSec float64 `json:"median_tokens_per_sec"`
-	P90CostPerSuccess  float64 `json:"p90_cost_per_success"`
-	SpeedEfficiency    float64 `json:"speed_efficiency"`
+	// P90CostPerSuccess is list-price, same caveat as DollarsPerPass below.
+	P90CostPerSuccess float64 `json:"p90_cost_per_success"`
+	SpeedEfficiency   float64 `json:"speed_efficiency"`
 
 	// DollarsPerPass is total $ across all runs / number of passes. The
 	// headline economic metric for the dashboard. 0 when no passes.
 	// M-EVAL-SWEET-SPOT-WEBSITE-INTEGRATION (v0.19.0).
+	//
+	// LIST-PRICE dollars, not spend. Agent-mode rows from a subscription lane
+	// (codex auth_mode chatgpt, claude OAuth) contribute a cost nobody was
+	// billed. Read alongside Aggregates.CostProvenance before calling this
+	// money; the v1.0 metered-dollars KPI must not take it at face value.
 	DollarsPerPass float64 `json:"dollars_per_pass"`
 
 	// CostOverheadVsBest is the median ratio (this_model_cost / best_passer_cost)
@@ -71,6 +77,9 @@ type SweetSpotRow struct {
 	// per-benchmark min(CostUSD) across all models that passed THAT benchmark.
 	// Captures "if a perfect router picked the cheapest model per benchmark, how
 	// much more would this model cost than that router?"
+	//
+	// Ratio of LIST-PRICE figures. It stays meaningful across provenances (both
+	// sides are list price), but it does NOT say the overhead was paid.
 	CostOverheadVsBest float64 `json:"cost_overhead_vs_best"`
 
 	// TokenOverheadVsBest is the same shape but for TotalTokens. Distinguishes

internal/eval_analysis/types.go

@@ -25,13 +25,18 @@ type BenchmarkResult struct {
 	CacheReadInputTokens     int     `json:"cache_read_input_tokens,omitempty"`
 	CacheCreationInputTokens int     `json:"cache_creation_input_tokens,omitempty"`
 	CostUSD                  float64 `json:"cost_usd"`
-	CompileOk                bool    `json:"compile_ok"`
-	RuntimeOk                bool    `json:"runtime_ok"`
-	StdoutOk                 bool    `json:"stdout_ok"`
-	DurationMs               int64   `json:"duration_ms"`
-	CompileMs                int64   `json:"compile_ms"`
-	ExecuteMs                int64   `json:"execute_ms"`
-	ErrorCategory            string  `json:"error_category"`
+	// CostProvenance says whether CostUSD was actually billed: "metered",
+	// "list-price-equivalent" (subscription lane — real arithmetic, zero spend),
+	// "free-local", or "unknown". Absent in baselines banked before 2026-07-30,
+	// where it reads "" — unmeasured, NOT metered.
+	CostProvenance string `json:"cost_provenance,omitempty"`
+	CompileOk      bool   `json:"compile_ok"`
+	RuntimeOk      bool   `json:"runtime_ok"`
+	StdoutOk       bool   `json:"stdout_ok"`
+	DurationMs     int64  `json:"duration_ms"`
+	CompileMs      int64  `json:"compile_ms"`
+	ExecuteMs      int64  `json:"execute_ms"`
+	ErrorCategory  string `json:"error_category"`
 
 	// Validity marks whether this row is a MEASUREMENT at all (vs a failure to
 	// measure: dead subject, harness error, wrong config). NIL means valid —
@@ -172,8 +177,16 @@ type Aggregates struct {
 	RepairUsed        int     `json:"repair_used"`         // Number of repairs attempted
 	RepairSuccessRate float64 `json:"repair_success_rate"` // Repair success rate
 	TotalTokens       int     `json:"total_tokens"`
-	TotalCostUSD      float64 `json:"total_cost_usd"`
-	AvgDurationMs     float64 `json:"avg_duration_ms"`
+	// TotalCostUSD sums cost_usd across EVERY run in the cohort regardless of
+	// provenance, so it is a LIST-PRICE total, not spend. On a subscription rig
+	// most agent-mode dollars here were never billed. Use CostProvenance to see
+	// the split before quoting this as money.
+	TotalCostUSD float64 `json:"total_cost_usd"`
+	// CostProvenance counts runs by how their cost arose (metered /
+	// list-price-equivalent / free-local / unknown). "unknown" includes every
+	// row banked before 2026-07-30, when the label did not exist.
+	CostProvenance map[string]int `json:"cost_provenance,omitempty"`
+	AvgDurationMs  float64        `json:"avg_duration_ms"`
 	// Prompt-cache aggregates. CacheHitRate is cache reads as a share of ALL
 	// input tokens the model saw (reads + writes + uncached), so it answers "how
 	// much of our input did we avoid paying full price for". 0 means either no