prove value

[macanderson]

thesis: determinism where determinism will do produces more reliable and efficiernt intelligence

i have never seen the list below of features make a positive impact yet

self driving mode
tool foundry
code graph
testing tool foundry with cc traces
repo overview
triage agent
verifier agent
context record retrieval
adaptive context lifecycle
pipeline mode
subagents
stella fleet
skill mining
fact rule promotion

--

where we are very weak: software developer lifecycle, creating prs/worktrees and working in rythym with developer drivers.

[macanderson]

The prove-value plan

Thesis under test: determinism where determinism will do produces more reliable and efficient intelligence.

Standing rule while this plan runs — FEATURE FREEZE. No new features until every feature below either proves its value or is deliberately cut. In scope: fixing, hardening, instrumenting, measuring, demonstrating. Out of scope: anything that adds surface area. The single exception is named in Phase 0 — the ablation switches and attribution counters that make measurement possible at all — and it is called out there explicitly rather than smuggled in.

This plan is the tracking artifact. Each phase ends with a verdict per feature written back into this issue.

---

1. Why nothing has shown value yet

The honest diagnosis is not "these features are worthless." It is that most of them have never been measured, and several are provably broken or starved in ways that make their value invisible even when it is real. Four patterns, all evidenced:

Silent starvation. The learning plane — skill mining, fact/rule promotion, context lifecycle — hangs off one JSON parse. Reflection routes to the triage model with reasoning unset; a reasoning-pinned triage model blows the 2048-token cap, truncates, yields zero lessons, and reflect_and_record returns early before anything is written. Measured frozen nine days: 8 reflection rows against 85 executions. The failure is detected and then thrown away to eprintln!, so the execution row still reads outcome='completed' and the dashboard shows a healthy loop. A workspace that has learned nothing for nine days is indistinguishable from a fresh one. (#2174, #2175, #2277)

Measuring the wrong panel. graph_query showed 0 of 1,243 tool calls on Terminal-Bench 2.1 — read as "the model ignores the tool." The trace said otherwise: TB tasks are green-field puzzles in tiny fresh workspaces with no existing codebase to symbol-hunt, so the steer conditions never trigger. On an actually-indexed repo the same tool measured 4.6% adoption with a 100% resolve rate — 5 of 5, zero unresolved. The retrieval half is not the problem; discoverability is. A benchmark where zero effect is the correct answer cannot disprove a feature.

Proof machinery that cannot prove itself. The verifier's flip oracle never observes a baseline tree, so every recorded "fail→pass" is candidate-only (#2272), and verifier_independent is unstamped on the deterministic rungs, so every real flip reports unmeasured (#2273). The feature whose entire job is producing evidence is currently not producing admissible evidence about itself.

Feedback loops that were never closed. cite_memory has been called zero times across 85 executions and 2,720 tool calls. It is registered everywhere with availability Always; the model simply is never asked. So selection_health is permanently empty, efficacy-based retirement can never fire, and the loop can learn but never unlearn (#2195). Recall force-filled 5 irrelevant frames every turn until the evidence gate landed (PR #2298), and its domain rung still re-admits every tagged node in any stella init-ed workspace (#2342).

So the plan is not "run a benchmark." It is, in order: make each feature honest → measure it where it should matter → keep, fix, or cut on the result.

---

2. The contract every feature must satisfy

Identical for all fourteen. A feature that skips a step has not been proven, whatever the number says.

1 — Pre-register before running. Metric, panel, minimum effect worth keeping, and kill criterion go into the feature's tracking issue before the experiment runs. This is the discipline doc:self-driving-missions already spells out: pre-registration bounds credit, and baselines are measured, never asserted. A threshold chosen after seeing the data is not a threshold.

2 — Paired ablation on a panel where the feature should matter. Same commit, same tasks, same models, feature ON vs OFF, repeats per task. Never a panel where zero effect is the correct result. Where no off-switch exists, Phase 0 builds one — the alternative is comparing across commits, which is confounded by construction.

3 — Mechanism metrics from the trace, never a surface signal. stella-events.jsonl is ground truth; result.json and the UI are projections with their own bugs. Census (role, model) from step_usage before believing any claim about which model ran. Join tool_start → tool_result for real wall clock. Read full proof reason strings — they are truncated everywhere else and the truncated half is usually the part that matters. Any two runs differing by more than one commit are confounded and must be reported as such.

4 — Reliability gate before the value experiment. A feature enters measurement only when: no crashes or operational aborts on the panel; every failure is detected and persisted, never silently swallowed; defaults fail closed; and anything claiming determinism replays twice and byte-compares. A feature that cannot be trusted to report its own failure cannot be measured — that is the nine-day lesson.

5 — Verdicts. PROVEN — keep, default on, with the number in the issue. FIX-AND-RETEST — defect found, fix it, re-run once. NO-VALUE — default off or delete. The third verdict is a success of this plan, not a failure of it. Fourteen features is a lot of surface area to carry on faith; cutting the ones that do not earn their keep is the point.

Two anti-patterns, both already stepped in once. Do not quote turns-to-first-artifact while #2359 is open — it is a placeholder for skills and absent for rules, and it renders like a measurement. Do not run a five-task solve rate to resolve anything smaller than a catastrophe; treat it as a guardrail and let the mechanism metrics do the measuring.

---

3. Phase 0 — unblock and instrument

Nothing downstream is trustworthy until these are true. Each is a fix or a test seam, not a feature.

#	Blocker	Why it blocks	Issue
0.1	main is red — cargo doc -D warnings fails on a private intra-doc link in stella-tools/src/shell_touch.rs	Every PR is red from an already-broken base; no measurement lands	#2375
0.2	Reflection parse failure is never persisted; triage reasoning=off still burns the whole budget	Starves features 9, 13, 14 invisibly. The single highest-leverage fix in this plan	#2175, #2277, #2174
0.3	Flip oracle never observes a baseline tree; verifier_independent unstamped	Feature 7 cannot produce admissible evidence about itself	#2272, #2273
0.4	Recall evidence gate's domain rung admits every tagged node	Feature 8's force-fill survives in any inited workspace	#2342
0.5	cite_memory never called	Feature 8/9's feedback half is dead; no efficacy signal exists	#2195
0.6	No independent off-switch for triage or verifier	The largest ablation gap. --no-pipeline removes triage and plan and scope and witness and verify together, so nothing attributes an effect to one stage	#2381
0.7	loop-bench counts only project_overview + graph_query as attributable	Features 8, 11, 13 have events but no per-feature counter in the comparison report	#2382

0.6 and 0.7 are the freeze exception, and I am saying so out loud. Both add code. Neither is a user-facing capability — 0.6 is a per-stage off switch in agent_engine_config so a stage can be ablated without deleting the pipeline, and 0.7 extends an existing counter table in the bench harness. If either grows past that, it has stopped being test infrastructure and needs a separate decision.

Two seams already exist and should be extended rather than rebuilt: context.retrieval.ab_recall_rate (durable, cross-process, counted in ab_control_counter in context.db; suppresses memories, skills, and records together on the off-turn) and loop-bench --compare a,b (emits stella-core's ComparisonReport, already consumed by stella tune effort). Neither is per-feature today; both are the right place to make it per-feature.

---

4. Feature cards

Fourteen cards. Each gives the reference files an agent must read, the exact on/off control, the panel, the metric, and the pre-registered kill criterion. Phase assignment follows evidence cost — deterministic and free first, paid and end-to-end last.

Phase 1 — deterministic, zero model calls

The strongest and cheapest evidence in the plan. make replay-learning drives the entire learning loop from recorded traces with zero model calls, and the Claude Code transcript adapter landed on main as PR #2367. Determinism means an ablation here is exact, not statistical. Read the Phase 1 kickoff comment before starting: the two corpora grade different features — the real CC corpus feeds the tool foundry only, and the synthetic corpus is all that can reach skill/rule mining.

---

F13 · skill mining

• Read: crates/stella-cli/src/memory/learning.rs, learning/dedupe.rs, learning/guarantees.rs, crates/stella-cli/src/memory/skill_files.rs, crates/stella-core/src/skills.rs, crates/stella-core/src/mining.rs
• Control: STELLA_DISABLE_REFLECTION=1 (kills the whole reflection loop, not just promotion — note the coupling)
• Panel: the committed simulated-months.json corpus. Not the CC corpus — Claude Code transcripts carry no Stella reflection JSON, so the adapter emits ScriptedReflection::NotRecorded and never reaches the reflection path (see the Phase 1 kickoff comment)
• Metric: skills mined per 100 turns; human-graded precision (is the skill correct and non-obvious?); and the half that actually matters — selection rate, how often a mined skill is injected into a later turn where it applies
• Kill: fewer than 3 skills per 100 turns, or graded precision below 50%, or selection rate ~0. A skill nobody selects is a file, not a feature.
• Blocked by: nothing — replay scripts the Provider, so the parse that starves the live system never runs. But a Phase 1 verdict here proves the machinery over supplied lessons, not delivered value; the live yield stays unmeasured until 0.2 lands, and the verdict must say so. Also note the far tail is unminable — same-fact pairs at 0.28 sit below the 0.36 cross-fact noise ceiling (core: lexical clustering still misses the far tail of natural variation — same-fact pairs at 0.28 sit below the cross-fact noise ceiling at 0.36 #2369), so measure inside the declared band and report the tail as known-missing.

F14 · fact rule promotion

• Read: crates/stella-cli/src/context_cmd/govern.rs, crates/stella-core/src/records/promotion.rs, crates/stella-cli/src/context_records.rs, crates/stella-cli/src/memory/rules_mining.rs
• Control: context.governance.mode; context.promotion.inferred_directive.*; promotion is opt-in per rule
• Panel: same replay corpus; verify the hash-chained .stella/rules/promotions.jsonl survives a tamper attempt
• Metric: rules promoted; precision (would a human have written this rule?); enforcement correctness — does a promoted must rule actually change a later turn's behavior?
• Kill: a promoted rule never changes downstream behavior. That is the whole claim; if it fails, the ledger is bookkeeping.
• Watch: records: inferred_rule_record stamps no precedence, so a human-promoted rule is the floor — first dropped by any budget #2338 — inferred_rule_record stamps no precedence, so a human-promoted rule is the floor and is first dropped by any budget. Measure with that fixed or the result is upside-down.

F9 · adaptive context lifecycle

• Read: crates/stella-core/src/context_record/lifecycle.rs, crates/stella-cli/src/settings/context.rs, crates/stella-cli/src/memory/{observations,proposals,appraisals,retirement}.rs
• Control: context.lifecycle.enabled = false — a genuine master switch that restores every pre-adaptive behavior
• Panel: the replay corpus over simulated months; the time_lapse_certification suite in crates/stella-core/src/records/tests.rs
• Metric: observation → proposal → record conversion rate; retirement correctness (does a never-useful record actually get retired?); cached→volatile→gone migration integrity
• Kill: the lifecycle advances no record through a full cycle on a months-long corpus, or retirement never fires.
• Blocked by: 0.2 and 0.5 — retirement is currently impossible because cite_memory is never called, so the unlearn half cannot be measured until that is fixed. Do not grade this feature before then.

F2 · tool foundry

• Read: crates/stella-core/src/tool_foundry.rs, crates/stella-tools/src/{foundry_author,foundry_gate,foundry_witness}.rs, crates/stella-store/src/foundry.rs, crates/stella-cli/src/tool_foundry/adopt.rs
• Control: stella tools --disable <NAME>; tools ship disabled (adopt ≠ enable), which is the correct fail-closed posture
• Panel: the real Claude Code corpus via the adapter — 3,822 transcripts, 485 project dirs, ~30 days (STELLA_REPLAY_CC_CORPUS=1, local-only, privacy-gated). The only card in Phase 1 with real data behind it, because shell history is exactly what option (b) supplies
• Metric: tools authored; adoption rate; Store::foundry_reuse — the existing fold that joins each adoption against tool_calls with ts >= adopted_at, so a name used before adoption cannot inflate it. Report adopted / reused / never-used together; false starts are the cost column.
• Kill: reuse rate near zero — an authored tool nobody calls again is pure overhead.
• Known gap: the third success limb from Self-improvement: Tool Foundry — stella authors, tests, and installs its own tools at runtime #830 — "converts a previously-failing task class to passing" — still needs a bench harness, not a ledger column. That is Phase 3 work, not a Phase 1 claim.
• Foundry gotcha: the detector only holes a path or number argument. cargo test -p <crate> varies a bareword, which stays in the skeleton and never clusters. Design corpora accordingly or you will measure the tokenizer, not the feature.

F4 · testing tool foundry with CC traces

• Read: crates/stella-cli/src/memory/replay.rs, replay/trace.rs, replay/summary.rs, replay/cc_adapter.rs (on main via feat(stella-cli): the Claude Code transcript adapter — local-only, option (b), honest about what the corpus cannot supply (#2304) #2367 — not in every worktree), docs/spec/trace-replay-learning-harness.md
• Control: n/a — #[cfg(test)]-only, driven by make replay-learning. Not ablatable in a live run by construction.
• Panel: itself. This feature's value is that it makes F2/F9/F13/F14 measurable at zero model cost.
• Metric: corpus turns replayed per second; determinism (replay twice, byte-compare); coverage of the learner surface
• Kill: if the harness cannot reproduce a known-good learning outcome deterministically, everything in Phase 1 is unfounded. This is the load-bearing piece — validate it first.
• Gotchas that cost an hour each: open_with_clock must be called with include_workspace_skills: true, or write_candidates/induce_rules record proposals and decline to write files, and a whole corpus replays correctly while building nothing. Fixture lessons must land inside the mining band (dedup 0.5 in forget::tokens space, clustering 0.4 in mining::terms space — different numbers in different token spaces, on purpose; PR fix(stella-cli): mine the recurrence the store deduplicates, and measure the clustering threshold in its own token space (#2358) #2371). And cli(replay): turns-to-first-artifact is a placeholder for skills and absent for rules, but renders like a measurement #2359 means turns-to-first-artifact is not yet quotable.
• Structural note: the learning loop is trapped in a bin-only crate (cli: the learning loop is trapped in a bin-only crate — extract it so anything but an in-crate test module can drive it #2321). That is what forces the harness to be a test module.

Phase 2 — live mechanism A/B, cheap

Few model calls, real sessions, mechanism metrics from the trace. Every one of these needs a panel chosen so that zero effect would be wrong.

---

F8 · context record retrieval

• Read: crates/stella-context/src/retrieval.rs, retrieval/{ranking,evidence,scope}.rs, crates/stella-cli/src/memory/recall.rs, crates/stella-cli/src/context_records.rs
• Control: context.retrieval.ab_recall_rate (durable, counted, already built for exactly this); require_evidence; ann_enabled
• Panel: a repo with real accumulated context — this workspace — on tasks whose answers depend on prior sessions. A fresh workspace cannot show recall value and must not be used.
• Metric: AgentEvent::ContextRecall { frames, provider_mix, tokens, usage, latency_ms } — the richest per-feature event in the tree. Precision of admitted frames (human-graded relevance); no_evidence_cut refusal count; turns-to-answer with recall on vs off.
• Kill: admitted-frame precision below chance, or no measurable difference in turns-to-answer on context-dependent tasks.
• Blocked by: 0.4 (context: the recall evidence gate's domain rung admits every tagged node on every turn — the force-fill survives in any stella init-ed workspace (#2298) #2342 domain rung force-fill) and 0.5. Also honest caveat: the only Embedder in the tree is HashEmbedder, character-trigram hashing — the whole vector pipeline runs on lexical surface overlap, not semantics (context: ship a real semantic embedder — the 'tracked follow-up' in embed.rs (R14, ONNX bge-small default) is tracked nowhere #2288). Grade the feature that ships, and report that a semantic embedder is untested rather than implied.

F3 · code graph

• Read: crates/stella-graph/src/{graph,queries,walk,watch}.rs, crates/stella-tools/src/graph.rs, crates/stella-cli/src/contextgraph.rs
• Control: needs both graph_query:off and project_overview:off — project_overview reads the graph directly via CodeGraph::entry_points, so disabling the query alone leaks the feature. Alternative clean control: an un-indexed workspace.
• Panel: a navigation/comprehension panel on an indexed repo — who-calls-this, what-imports-that, blast radius of a signature change. Explicitly not TB2.1, where 0 usage is the correct answer.
• Metric: stella stats graph --scan adoption and resolve rate (INCONCLUSIVE below 30 structural searches; RESOLVE-REGRESSED never grades as a win); tool calls to reach a correct answer, graph on vs off
• Kill: on a panel built for it, the graph arm needs no fewer steps than the grep arm.
• Prior: adoption 4.6%, resolve rate 100% (5/5). The retrieval half is likely fine; the gap is discoverability, so measure whether the agent reaches for it, not whether it answers.

F5 · repo overview

• Read: crates/stella-tools/src/overview.rs, crates/stella-tools/src/scripts.rs, crates/stella-cli/src/agent/prompt.rs
• Control: project_overview:off
• Panel: cold-start orientation tasks in an unfamiliar repo
• Metric: loop-bench already counts project_overview per trial — the cleanest existing per-feature attribution in the tree. Measure exploration tool calls before first productive edit, on vs off.
• Kill: no reduction in orientation cost. This is a pure-assembly tool with no model call, so its cost is near zero and the bar is correspondingly low — but "cheap" is not "valuable."

F6 · triage agent

• Read: crates/stella-pipeline/src/triage.rs, pipeline/triage_stage.rs
• Control: does not exist independently — blocked on 0.6
• Panel: a mixed panel spanning trivial lookups through multi-file changes, where routing should visibly differ
• Metric: classification accuracy against human labels; cost saved on the SimpleLookup fast path; misroute cost — how expensive is a wrong class?
• Kill: accuracy no better than a length heuristic, or fast-path savings smaller than the triage call itself.
• Watch: protocol: ProofStep::Assurance is emitted at triage and states intent, in a rail of outcomes #2274 — ProofStep::Assurance is emitted at triage and states intent, in a rail of outcomes. A metric read off that rail is reading a promise as a result.

F7 · verifier agent

• Read: crates/stella-pipeline/src/verify.rs, verify/mutation.rs, witness.rs, witness/warrant.rs, flip_halt.rs, docs/spec/witness-protocol.md
• Control: partial only — pipeline_verifier_evidence_demand and pipeline_require_independent_verifier tune strictness; neither removes the stage. Blocked on 0.6.
• Panel: tasks with known-correct and known-subtly-wrong solutions — the verifier's job is telling them apart
• Metric: true-flip rate; false-credit rate (the expensive error); tamper-exclusion effectiveness; witness authoring cost per verified task
• Kill: false-credit rate materially above zero. A verifier that credits wrong work is worse than none, because the contract this whole project sells is "verified done, not claimed done."
• Blocked by: 0.3. And the standing caution: a WITNESS CONFIRMED has already turned out to be a false proof off a ModuleNotFoundError. Read the full proof reason string, every time.

F11 · subagents

• Read: crates/stella-core/src/subagent.rs, crates/stella-tools/src/subagent.rs, crates/stella-cli/src/subagent.rs, crates/stella-cli/src/subsession.rs
• Control: task:off. Note STELLA_LEAN_TOOLS hides task from advertisement but still executes it if called — it is a prompt-size lever, not an ablation switch.
• Panel: tasks with genuinely parallel independent subtasks
• Metric: AgentEvent::SubAgent { phase } + the agent_uses table; parent context saved; spend accounting correctness — does every child's spend reach a ledger, including an orphaned child's?
• Kill: no context saving versus doing the work inline, or any spend leak. The accounting bug here was historically sharper than the runtime one.

Phase 3 — end-to-end, paid

Real money, real benchmark, run last because it is the most expensive and the least diagnostic.

---

F10 · pipeline mode

• Read: crates/stella-pipeline/src/lib.rs, pipeline.rs, crates/stella-cli/src/agent.rs::run_prompt (the use_pipeline fork), crates/stella-cli/src/agent/goal.rs (the raw-loop alternative)
• Control: --no-pipeline — the cleanest ablation switch in the repo; ArenaBench declares it as engine.bare_loop = true
• Panel: the 8-task TB2.1 panel, repeats per task, same commit both arms
• Metric: solve rate as a guardrail only; the real measurement is steps to solve, priced cost per solve, and cache hit rate. Sum priced_cost — self-reported cost undercounts, and a timed-out trial never writes its final cost line at all ($4.35 self-reported against $20.25 priced, on one real 8-trial run).
• Kill: pipeline mode costs materially more per solve without a solve-rate gain. This is the most consequential card in the plan — pipeline mode is the default path, so a NO-VALUE here is a large architectural finding, and it must be run with the same rigor either way.
• Comparator: Claude Code on Fable 5 = 5/8, match 41c7e447666a, 93% cache hit. Stella measured 76% on a comparable run; at any provider's rates that gap is worth more than any token-count reduction.

F1 · self driving mode

• Read: crates/stella-core/src/self_driving.rs, crates/stella-cli/src/self_driving_cmd.rs, scripts/self-driving.sh, scripts/test-self-driving.sh
• Control: no toggle — it is a command, so "off" is not running it
• Panel: a fixed backlog slice, self-driving vs a human-driven session on the same slice
• Metric: from ~/.stella/self-driving/<slug>/ledger.jsonl — defects fixed per cycle, PRs landed green on first push, AIMD convergence behavior, dry-streak honesty
• Kill: the loop produces PRs a human must redo, or the governor oscillates instead of converging.
• Important scope note: missions are spec-only. docs/spec/self-driving-missions.md and self-driving-foundry.md are both status proposed, and a word-boundary search for mission/campaign across crates/ returns zero hits. Grade the shipped verb set (plan/cycle/aperture/calibrate/…). Implementing mission schema: generalize the campaign manifest into system-agnostic missions (doc:self-driving-missions) #2345 is a new feature and is frozen.

F12 · stella fleet

• Read: crates/stella-fleet/src/{fleet,ledger,plan,git,gc}.rs, crates/stella-cli/src/fleet_cmd.rs
• Control: no toggle — a separate command
• Panel: a batch of 5–10 independent tasks, fleet vs sequential
• Metric: wall clock versus sequential; worktree isolation correctness (zero cross-contamination); spend ledger accuracy; merge conflict rate on the produced branches — the number that decides whether fan-out is real leverage or deferred work
• Kill: conflict-resolution cost exceeds the parallelism saved.

Phase 4 — the named weakness: SDLC rhythm

Mac's own words: "where we are very weak: software developer lifecycle, creating prs/worktrees and working in rhythm with developer drivers."

This phase is deliberately characterization, not construction. The freeze holds. The deliverable is an honest account of what the shipped path does when a human is driving alongside it, and a set of issues written as handoffs for whatever it cannot do.

• Drive a real change end to end with a human in the loop: worktree → commits → PR → review → green gate → merge. Record where the agent needs a human and where it should not have.
• Instrument the friction: how many human interventions per landed PR, and what each one was for.
• Known process hazards to reproduce or rule out, all of them already bitten: admin merges bypass the required check — five PRs landed in 2m32s with no green CI and two composed into a red main (gate: admin merges bypass the required check — five PRs landed in 2m32s with no green CI, two composed into a red main #2368); the deleted-test guard tells you to name the test in a commit message that CI's fetch-depth: 2 cannot read (ci: check-deleted-tests tells you to name the test in a commit message, but CI's fetch-depth: 2 cannot read one #2331); the file-size gate reads a local trunk merge as a PR checkout and goes false-green on your own growth (gate: resolve_base_commit reads a local trunk merge as a PR checkout, so the pre-push file-size gate goes false-green on your own growth (#2267) #2317).
• Every gap found becomes an issue written as a handoff — problem, paths, repro, constraints, definition of done — not a feature built in the same breath.

---

5. Reference files

The plan's own inputs, for an agent picking this up cold:

The contract and the invariants — AGENTS.md (ports/no-I/O/typed-errors invariants, the witness-test definition of done, the gate, the "nothing left behind" rule), CLAUDE.md (the reference-grade bar; the "read the trace, never a surface signal" rule).

Specs — docs/spec/trace-replay-learning-harness.md, docs/spec/witness-protocol.md, docs/spec/verification-gate.md, docs/spec/adaptive-context/adaptive-context.md, docs/spec/adaptive-context/context-pr.md, docs/spec/self-driving-missions.md (proposed), docs/spec/self-driving-foundry.md (proposed), docs/spec/schema-graph.md.

Measurement — crates/stella-protocol/src/event.rs (the AgentEvent variants that carry per-feature attribution: ContextRecall, ContextWrite, SubAgent, Stage, ToolStart/ToolResult, StepUsage, Proof, Verdict), bench/loop-bench/ (src/artifacts.rs distills stella-events.jsonl; --compare a,b emits ComparisonReport), bench/telemetry_store/schema.sql + ingest.py (the offline warehouse), bench/evidence/<experiment>/analyze.py (the established per-experiment writeup pattern — copy one), arenabench/config.py (where engine.bare_loop is declared).

Telemetry tables — store.db: skill_usage, agent_uses, foundry_tools, memory_citations, reflections, tool_calls, step_manifest, step_receipt. context.db: context_records, context_extraction_cursor, ab_control_counter. Plus codegraph.db and fleet.db.

Baselines to beat or reproduce — Claude Code + Fable 5 = 5/8 on the 8-task TB2.1 panel (match 41c7e447666a, zero detections, 93% cache hit). graph_query = 4.6% adoption / 100% resolve on an indexed repo. Reflection: 8 rows against 85 executions during the frozen window.

---

6. Execution order

1. Phase 0 — every blocker above, main green first. Nothing else starts.
2. Phase 1 — validate the replay harness itself, then F13, F14, F9, F2, F4. Free and deterministic; run it to completion before spending a dollar.
3. Phase 2 — F8, F3, F5, then F6, F7 once 0.6 lands, then F11.
4. Phase 3 — F10 first (it is the default path and the biggest decision), then F1, F12.
5. Phase 4 — SDLC characterization; file everything found.

After each phase: write the verdict per feature into this issue with the number and the trace path behind it. A NO-VALUE verdict gets a follow-up issue proposing default-off or deletion, and the freeze stays in force until every card carries a verdict.

---

7. Kickoff prompt

Paste this to start a session against this plan.

Read issue #2374 and its prove-value plan comment in full before doing anything.

You are working under a FEATURE FREEZE: no new features. Only fixing,
hardening, instrumenting, measuring, and proving what already exists.
If the work you are about to do adds user-facing surface area, stop and
say so instead of doing it.

Your task: <PHASE / FEATURE ID, e.g. "Phase 0 blocker 0.2" or "Phase 1, F13 skill mining">

Follow the plan's contract exactly:

1. Pre-register before you run anything. Post a comment on the feature's
   tracking issue stating the metric, the panel, the minimum effect worth
   keeping, and the kill criterion. A threshold chosen after seeing the
   data is not a threshold.
2. Read the card's reference files first. Verify each claim against the
   current code — the plan cites file paths and issue numbers that may
   have moved.
3. Run the ablation paired: same commit, same tasks, same models, feature
   ON vs OFF, repeats per task. If no off-switch exists, that is Phase 0
   work and it comes first.
4. Take every conclusion from stella-events.jsonl, never from result.json,
   the UI, or a solve-rate cell. Census (role, model) from step_usage
   before believing which model ran. Read full proof reason strings.
   Sum priced_cost, never self-reported cost.
5. Report the unflattering number when it is the true one. A benchmark
   that flatters Stella because of a measurement artifact spends the exact
   reputation this project exists to build.
6. Write the verdict — PROVEN / FIX-AND-RETEST / NO-VALUE — into issue
   #2374 with the number and the trace path behind it. NO-VALUE is a
   legitimate and expected outcome; do not argue a feature into surviving.
7. Everything you notice and do not fix becomes a GitHub issue written as
   a handoff before you finish.

Definition of done for your task: a verdict backed by a trace someone else
could re-read, plus any issues filed. Not a claim in a PR description.

[macanderson]

Phase 1 kickoff — the deterministic proofs

Two corrections to the plan's Phase 1, both found while writing this prompt by reading cc_adapter.rs rather than trusting the card.

Correction 1 — Phase 1 is not blocked by 0.2, and the reason matters. The card for F13 says "blocked by 0.2" (the reflection starvation fix). That is wrong for the replay path: the replayer scripts the Provider, so the reflection JSON comes from the fixture and the parse that starves the live system never runs. Phase 1 can start today, on a red main, with no model key.

The consequence is a limit on what a Phase 1 verdict may claim. Replay proves the learner machinery — given lessons, does the miner cluster, promote, and write files correctly. It does not prove delivered value, because in production those lessons arrive through the path that has been starving. So a Phase 1 PROVEN is conditional and must be written that way: machinery proven, live yield unmeasured pending #2175/#2277. Anyone who reads a Phase 1 green as "skill mining works in production" has been misled by us.

Correction 2 — the Claude Code corpus grades the tool foundry, and only the tool foundry. The plan implies the adapter feeds skill mining. It does not. cc_adapter.rs implements option (b) of §7.2 deliberately: Claude Code transcripts contain no Stella reflection JSON, so rather than invent lessons — which would measure the adapter's invention heuristic instead of Stella's learning — every adapted turn emits ScriptedReflection::NotRecorded, and the replayer never reaches the model boundary at all.

So the corpora split cleanly, and the split is the most important thing to hold in your head this phase:

Corpus	Grades	Strength of claim
~/.claude/projects via the adapter — 3,822 transcripts, 485 project dirs, ~30 days	F2 tool foundry (real shell history, real recurrence density)	Strong: real data, real recurrence no synthetic fixture matches
simulated-months.json (committed)	F13 skill mining, F14 rule promotion, F9 lifecycle	Weaker: proves the machinery over authored lessons; says nothing about real-world yield

The corpus is thirty days, not months — the months axis comes from the replayer's dilated clock, not the data — and it is a rolling window under Claude Code's own retention, so it is not reproducible across machines or across time. Say it that way in the report. Claiming a months-long corpus we do not have is exactly the small dishonesty that costs more than it buys.

---

The prompt

Read issue #2374 and its prove-value plan comment in full, plus the Phase 1
kickoff comment, before doing anything.

You are working under a FEATURE FREEZE: no new features. Only fixing,
hardening, instrumenting, measuring, and proving what already exists. If the
work you are about to do adds user-facing surface area, stop and say so
instead of doing it.

Your task: Phase 1 — grade the four learning-plane features deterministically,
with zero model calls, in this order. F4 first; it is load-bearing and
everything after it is unfounded if it fails.

Work from a fresh worktree off origin/main. main is currently red on a
cargo-doc gate (#2375) — that does not block cargo test, so proceed, but do
not attempt to land anything until it is green.

--- F4: validate the harness itself ---

    make replay-learning

Read the printed §6 report. Then prove the harness is what it claims:

  1. Determinism: run it twice and byte-compare the report. The summary is
     documented as deterministic by construction (sorted lists, no
     process-global sources, workspace-relative paths). If two runs differ,
     stop — every number in this phase is worthless until that is fixed, and
     that finding alone is worth the session.
  2. Zero model calls: confirm the scripted Provider is the only provider on
     the path and nothing reaches a network boundary.
  3. Coverage: which of the learner surfaces does the corpus actually
     exercise? Name the ones it does not.

If F4 fails, file it and stop. Do not grade F2/F13/F14/F9 on a harness that
has not proven itself.

--- F2: tool foundry, against the real corpus ---

This is the only card in Phase 1 with real data behind it.

    STELLA_REPLAY_CC_CORPUS=1 cargo test -p stella-cli --bin stella \
      memory::replay::cc_adapter::tests::the_real_corpus_adapts_when_opted_in \
      -- --exact --nocapture

The variable is a boolean opt-in; the path is anchored at
~/.claude/projects and is never read without it. This is LOCAL-ONLY and
PRIVACY-GATED under invariant 3: nothing derived from it may be committed,
the committed CI corpus stays synthetic permanently, and every command is
redacted before it is written. A derived trace belongs in gitignored scratch.
Do not paste transcript content into an issue, a PR, or your final report —
report counts and shapes, never commands.

Pre-register, then measure:
  - tool proposals minted from the whole shell history (ReplaySummary
    reports this directly)
  - human-graded precision: of the proposals, how many are tools you would
    actually want? Read them. This number cannot be automated.
  - Store::foundry_reuse is the honest reuse metric — it joins each adoption
    against tool_calls rows with ts >= adopted_at, so a name the workspace
    used before adoption cannot inflate it. Report adopted / reused /
    never-used together; false starts are the cost column.

Kill criterion: proposals near zero over 3,822 real transcripts, or graded
precision below 50%.

Gotcha that will cost you an hour: the detector only holes a PATH or a
NUMBER argument. `cargo test -p <crate>` varies a bareword, which stays in
the skeleton and never clusters. If proposals come back low, check whether
you are measuring the tokenizer rather than the feature, and say which.

--- F13, F14, F9: the synthetic corpus ---

Same harness, the committed simulated-months.json. ReplaySummary already
reports what you need: lessons_offered vs lessons_stored, distinct memory
texts, SKILL.md files on disk, published rule records on disk, and the
per-turn model-error record.

Grade each on:
  - yield (artifacts built per 100 turns)
  - human-graded precision — read the mined skills and rules; are they
    correct and non-obvious?
  - selection: is a mined artifact actually loaded and used by a later turn?
    A skill nobody selects is a file, not a feature.

For F9 specifically, the retirement half CANNOT be graded: cite_memory has
never been called in production (#2195), so efficacy-based retirement can
never fire. Grade the learn half, and report the unlearn half as
unmeasurable-by-construction rather than passing it silently.

Do NOT quote turns-to-first-artifact. The field exists in ReplaySummary and
renders like a measurement, but #2359 says it is a placeholder for skills and
absent for rules. Quoting it would be reporting a placeholder as a result.

Two gotchas from the last session in this code:
  - open_with_clock must be called with include_workspace_skills: true. With
    false, write_candidates/induce_rules record proposals and decline to
    write FILES (#737) — a whole corpus replays correctly and builds nothing.
  - Dedup runs at 0.5 in forget::tokens space; clustering at 0.4 in
    mining::terms space. Different numbers in different token spaces, ON
    PURPOSE (PR #2371). Do not "align" them. The far tail — same-fact pairs
    at 0.28, below the 0.36 cross-fact noise ceiling — is known-unminable and
    tracked as #2369; measure inside the band and report the tail as missing.

--- Reporting ---

Write one verdict per feature into #2374: PROVEN / FIX-AND-RETEST / NO-VALUE,
with the number and the command that produced it.

Every Phase 1 verdict is CONDITIONAL and must say so: replay proves the
learner machinery over supplied lessons, not delivered value, because the
live path that supplies those lessons is the one starving behind
#2175/#2277. Do not let a green replay report be read as "this works in
production."

NO-VALUE is a legitimate and expected outcome. Do not argue a feature into
surviving. Everything you notice and do not fix becomes a GitHub issue
written as a handoff before you finish.