Freeze the Understanding Alpha study runner

.github/workflows/ci.yml

@@ -35,6 +35,8 @@ jobs:
           cargo test --workspace --doc --locked
       - name: Tests
         run: cargo test --workspace --all-targets --locked
+      - name: Understanding study runner tests
+        run: python3 scripts/test-understanding-study.py
 
   msrv:
     name: MSRV 1.88

.gitignore

@@ -24,6 +24,8 @@
 # OS / editor cruft.
 .DS_Store
 Thumbs.db
+__pycache__/
+*.py[cod]
 *.swp
 .idea/
 .vscode/

CHANGELOG.md

@@ -17,6 +17,14 @@ project uses version-gated milestones (see ROADMAP.md), not dates.
   and erase prototype rather than complete continuity.
 
 ### Added
+- A dependency-free frozen runner and 20-probe bank for 0.4 Understanding
+  Alpha. It derives the complete balanced 24-pair allocation from the declared
+  seed, emits oracle-free public study packets, redacts forbidden host data,
+  enforces equal call budgets and identical Reveal payloads, handles ordered
+  reserves and one schema repair, computes the predeclared stratified paired
+  percentile bootstrap, and refuses to report incomplete cohorts. Fifteen
+  focused regressions run in CI and the release gate. No qualifying response or
+  study result is claimed.
 - Formula Jam discovery is shared across App, CLI, and MCP: curated recipe bank
   in core; MCP `plot_expression` supports manual expr, `recipe`, `seed`,
   `auto_step`, and `list_recipes` with structured discovery results; CLI `plot`

README.md

@@ -172,11 +172,14 @@ and green CI. Human stranger sessions, accessibility panels, musician
 long-listening, and broad hardware certification are real gates for **0.8 /
 1.0**, not stop-the-world work before 0.2.
 
-**Current milestone:** 0.3 Tactile Alpha meets its agent-and-machine exit on
-the five flagships, including the consented Watch Agent path and the declared
-reference performance budget. The next incomplete milestone is 0.4
-Understanding Alpha. Soft-thin densify, bulk rooms, and Phase B glow stay background unless
-measurement binds. Ordered criteria live in
+**Current milestone:** 0.4 Understanding Alpha. The 0.3 Tactile Alpha
+agent-and-machine exit is met on the five flagships, including the consented
+Watch Agent path and the declared reference performance budget. The frozen
+0.4 headless runner and held-out probe bank now exist and are CI-gated; no
+qualifying cohort result is claimed. Returning-journal sovereignty, independent
+pre-collection review, and the complete matched cohort remain open. Soft-thin
+densify, bulk rooms, and Phase B glow stay background unless measurement binds.
+Ordered criteria live in
 [`docs/ROADMAP.md`](docs/ROADMAP.md) (**Critical path right now**).
 The predeclared contract and evidence boundary for the next study live in
 [`docs/UNDERSTANDING_STUDY.md`](docs/UNDERSTANDING_STUDY.md); no qualifying

VERIFY.md

@@ -17,6 +17,7 @@ contributors and the curious.
   fresh shell does not see `cargo`, add that to `PATH`.
 - Optional, for the local coverage gate: `cargo install cargo-llvm-cov`.
 - Optional, for the local supply-chain gate: `cargo install cargo-deny`.
+- **Python 3.11 or newer**, for the frozen 0.4 study-runner regressions.
 - The Linux build needs the ALSA, xkbcommon, and libudev headers (the packages
   CI installs): `sudo apt-get install -y libasound2-dev libxkbcommon-dev libudev-dev`.
 
@@ -27,7 +28,8 @@ Run the full gate and regenerate every artifact:
 - Windows: `scripts\verify.ps1`
 - macOS / Linux: `bash scripts/verify.sh`
 
-It runs format, clippy and rustdoc with warnings denied, tests, locked build,
+It runs format, clippy and rustdoc with warnings denied, Rust and 0.4 study
+runner tests, locked build,
 coverage (if `cargo-llvm-cov` is present), supply-chain policy (if `cargo-deny`
 is present), the house-style guard, and the native installer safety self-test,
 then writes images and audio into `renders/`.
@@ -42,6 +44,8 @@ RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --locked  # macOS / L
 RUSTDOCFLAGS="-D warnings" cargo test --workspace --doc --locked     # macOS / Linux
 cmd /d /c "set RUSTDOCFLAGS=-D warnings&& cargo doc --workspace --no-deps --locked && cargo test --workspace --doc --locked"  # Windows
 cargo test --workspace --all-targets --locked
+python scripts/test-understanding-study.py          # Windows
+python3 scripts/test-understanding-study.py         # macOS / Linux
 cargo build --workspace --locked
 cargo +1.88.0 check --workspace --all-targets --locked
 cargo llvm-cov --workspace --fail-under-lines 80 --ignore-filename-regex '(crates[\\/](gpu|audio)[\\/]|faces[\\/]app[\\/]src[\\/]main\.rs)'
@@ -54,9 +58,10 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts/install.ps1 -SelfTes
 ```
 
 
-Expected right now: **format and clippy clean, 3,180 all-target test cases pass,
-one screenshot diagnostic is ignored, 95.15% region coverage, and 95.33% line
-coverage**. The `gpu` and `audio` crates plus the app event
+Expected right now: **format and clippy clean, 3,180 all-target Rust test cases
+and 15 frozen study-runner regressions pass, one screenshot diagnostic is
+ignored, 95.15% region coverage, and 95.33% line coverage**. The `gpu` and
+`audio` crates plus the app event
 loop are excluded from the coverage gate and have dev-machine integration
 evidence, see `docs/QUALITY.md`. Controller routing is pure-tested. Sessions
 with representative physical controller models remain open.

docs/ENGINEERING.md

@@ -188,8 +188,9 @@ git config core.hooksPath scripts/hooks
 It runs the house-style guard on every commit (instant, and it applies to docs
 as much as code), and the cargo gate (fmt, Clippy and rustdoc with warnings
 denied, plus doctests and the full test suite) only when the commit touches
-Rust, `Cargo.*`, or a shader, so a docs-only commit stays fast. Coverage, the
-locked build, and artifact regeneration stay in
+Rust, `Cargo.*`, or a shader, so a docs-only commit stays fast. Changes to the
+frozen 0.4 study runner, probe bank, or its test file run the focused Python
+regressions. Coverage, the locked build, and artifact regeneration stay in
 `scripts/verify.sh` (the release gate); they are too slow for every commit.
 Emergency bypass is `git commit --no-verify`, after which you must run
 `scripts/verify.sh` before pushing.

docs/ROADMAP.md

@@ -44,15 +44,16 @@ stop-the-world work.
 
 | Priority | What | Why |
 | --- | --- | --- |
-| **1. Build the frozen 0.4 study runner** | Deterministic allocation, held-out transfer probes, active control, scoring, redaction, and incomplete-cohort refusal from `UNDERSTANDING_STUDY.md` | The method must exist before qualifying data is collected, or the result can drift toward the desired outcome |
+| **1. Audit and freeze the 0.4 collection** | Independently review the implemented runner and 20-probe bank, then track its generated 24-pair allocation before the first qualifying response | The runner now blocks method drift mechanically; pre-collection review must catch a flawed probe or oracle before results can influence a correction |
 | **2. Complete returning-journal sovereignty** | Add provenance-preserving correction, bounded structured export, two-process clean-clone acceptance, and managed-path residue evidence | Current read, record, and erase tools are only the prototype slice and do not yet satisfy the 0.4 return-session exit |
 | **3. Run and publish the 0.4 cohort** | Execute 20 matched pairs, publish every planned outcome and deviation, and obtain independent methodology and math review | A tracked runner without a complete, reviewable result is preparation, not milestone evidence |
 | **Standing gate** | Keep 0.2 and 0.3 proof plus public CI green | Regressions reopen completed milestones and invalidate new evidence |
 | **Deferred past 0.2** | Human stranger hallway, a11y sessions, musician long-listening, soft-thin densify, bulk rooms, Phase B glow | Humans before 1.0; densify/glow only if a sensory ceiling binds |
 
 The 0.3 agent-and-machine exit is met. The next incomplete milestone is 0.4
-understanding and retention. Its protocol and some mechanisms exist, but the
-qualifying study result and returning-player exit evidence remain open.
+understanding and retention. Its protocol, deterministic study runner, held-out
+probe bank, and some mechanisms exist, but pre-collection independent review,
+the qualifying study result, and returning-player exit evidence remain open.
 Detail below and in the version sections.
 
 - **Done:** the headless core (`Room` trait with `reveal()`, deterministic ASCII `Canvas`, seeded RNG, registry, `verb`, `render_poked`, and variation); the CLI face (`numinous`), the MCP face (`numinous-mcp`), and the windowed app; **354 catalog rooms** plus hidden content; 6 lever-driven sims; 11+ games; the full engineering harness (edition-2024 workspace, pinned toolchain, `-D warnings`, cargo-deny, house-style guard, an 80% line coverage gate, three-OS CI). Current local evidence: fmt, Clippy, 3,180 passing all-target test cases plus one ignored screenshot diagnostic, locked build, Windows release gate, 95.15% region coverage, and 95.33% line coverage all pass.
@@ -777,10 +778,19 @@ read-only App viewer, with no private host or protocol data in the stream.
   sample, frozen outcomes, exclusions, pass rule, privacy boundary, and honest
   distinction between immediate transfer, within-context retention, and durable
   human learning. No qualifying result is claimed.
-- **Incomplete:** implement the deterministic runner and held-out probe bank,
-  then run and publish the complete cohort with all outcomes and deviations.
-  Private working notes and scripted conclusions do not satisfy this evidence
-  gate.
+- **Done (2026-07-27):** Implement the dependency-free headless study runner
+  and frozen 20-probe bank. The runner derives a balanced 24-pair allocation
+  from the declared seed, emits one oracle-free probe at a time, enforces equal
+  public tool-call budgets and identical Reveal payloads, permits one
+  schema-only repair, redacts forbidden host data, scores with independent math
+  oracles, runs the declared stratified paired percentile bootstrap, accounts
+  for ordered reserves, and refuses to report an incomplete cohort. Fifteen
+  focused regressions run in CI and the release gate. No qualifying response
+  has been collected.
+- **Incomplete:** obtain independent pre-collection review, commit the exact
+  generated allocation before accepting a response, then run and publish the
+  complete cohort with all outcomes and deviations. Private working notes and
+  scripted conclusions do not satisfy this evidence gate.
 - **Done (cycle 152):** Add source provenance and an independent math-review checklist to every
   flagship Reveal.
 - **Done (architectural invariant):** Keep progression subordinate to autonomy: no streak loss, required grind, or

docs/UNDERSTANDING_STUDY.md

@@ -1,8 +1,49 @@
 # Understanding Alpha Study Protocol
 
-Status: predeclared design, reviewed 2026-07-26. No qualifying cohort has
-started and no result is claimed. Earlier private scripted notes are exploratory
-only, are excluded from this study, and cannot satisfy the 0.4 evidence gate.
+Status: predeclared design and implemented frozen runner, reviewed 2026-07-27.
+No qualifying cohort has started and no result is claimed. Earlier private
+scripted notes are exploratory only, are excluded from this study, and cannot
+satisfy the 0.4 evidence gate.
+
+## Implemented runner boundary
+
+`scripts/understanding-study.py` and `scripts/understanding-probes.json` now
+implement the first dependency in this protocol. The dependency-free runner
+derives all 24 pairs from the declared allocation seed, balances condition
+order and cyclic room order within each model family, and binds each manifest
+to the canonical probe-bank hash. The bank contains 10 immediate and 10 late
+probes, two per flagship per phase, plus one five-item distractor sequence.
+
+Public commands emit one condition, probe, repair schema, or distractor at a
+time without the oracle. Analysis requires an exact runner-generated manifest,
+explicit consent metadata, 20 public tool events per session, identical Reveal
+payloads across each pair, ordered probe delivery, and all 20 qualifying pairs.
+The sanitizer removes forbidden fields, absolute host paths, and named host
+identifiers before tracked evidence is analyzed. Withdrawn records retain no
+response content. The scorer uses independent mathematical oracles and the
+declared 100,000-resample stratified paired percentile interval, implemented
+with a stable SHA-256 counter stream and frozen linear percentile interpolation.
+
+The current canonical hashes are:
+
+- probe bank: `4ac647fdfc4559b26ab417ece0eb01a021fc53d0decda35d8d5c798fd000cbc0`
+- generated allocation: `8a241287e91589d18e877f75d07d9fc03cb98dbd518e63b7cd26aa8922fd1a01`
+
+Before any qualifying response, independently review the probes and oracles,
+generate the manifest into the tracked evidence directory, validate it with the
+same runner revision, and commit it. A dry run stays gitignored:
+
+```text
+python scripts/understanding-study.py allocate --output .agent/understanding-allocation.json
+python scripts/understanding-study.py validate --manifest .agent/understanding-allocation.json
+python scripts/test-understanding-study.py
+```
+
+The 15 focused regressions prove deterministic balance and hashes, oracle
+coverage, oracle-free public packets, redaction, ordered reserve replacement,
+single schema repair, equal call ordering, identical Reveal delivery, and hard
+incomplete-cohort refusal. They do not constitute participant evidence or
+independent methodology or math review.
 
 ## Decision and scope
 
@@ -240,10 +281,10 @@ successful journal read does not satisfy this evidence contract.
 
 ## Implementation order and acceptance
 
-1. Build the deterministic allocation, probe, scoring, redaction, and report
-   runner. Prove allocation balance, answer-key independence, no condition
-   leakage, deterministic scoring, complete failure accounting, and refusal to
-   report an incomplete cohort.
+1. The deterministic allocation, probe, scoring, redaction, and report runner
+   is implemented and CI-gated. Independently review its probe content and
+   oracles, then commit its exact generated allocation before the first
+   qualifying response.
 2. Complete the journal correction, structured export, and residue receipt, then
    prove the two-process return path with an isolated journal location.
 3. Run the frozen cohort, publish all bounded evidence, obtain independent math
@@ -260,15 +301,19 @@ does not belong in Numinous.
 ## Current sources
 
 - [Center for Open Science preregistration guidance](https://www.cos.io/initiatives/prereg),
-  reviewed 2026-07-26. It separates planned confirmatory work from exploratory
+  reviewed 2026-07-27. It separates planned confirmatory work from exploratory
   work and calls for methods detailed enough to replicate.
 - [Retrieval Practice Versus Elaborative Encoding, 2025 systematic and
   meta-analytic review](https://doi.org/10.1007/s10648-025-10076-6), reviewed
   2026-07-26. It supports an active elaborative control, corrective feedback,
   and separate retention and transfer outcomes.
 - [NIST Privacy Framework](https://www.nist.gov/privacy-framework), reviewed
-  2026-07-26. Version 1.0 remains final; version 1.1 is an initial public draft
+  2026-07-27. Version 1.0 remains final; version 1.1 is an initial public draft
   as of this review.
 - [MCP 2025-11-25 security and trust principles](https://modelcontextprotocol.io/specification/2025-11-25),
-  reviewed 2026-07-26. The current specification requires explicit consent,
+  reviewed 2026-07-27. The current specification requires explicit consent,
   user control, and appropriate data protections.
+- [SciPy bootstrap reference](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html),
+  reviewed 2026-07-27. It documents paired resampling, repeatable random input,
+  and two-sided percentile intervals. The runner implements the predeclared
+  stratified variant without adding SciPy as a runtime dependency.

scripts/check.ps1

@@ -28,5 +28,6 @@ Invoke-Step "docs"        {
     }
 }
 Invoke-Step "test"        { cargo test --workspace --all-targets --locked }
+Invoke-Step "understanding study runner" { python scripts/test-understanding-study.py }
 Invoke-Step "house style" { powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-style.ps1 }
 Write-Host "All checks passed."

scripts/check.sh

@@ -12,6 +12,8 @@ RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --locked
 RUSTDOCFLAGS="-D warnings" cargo test --workspace --doc --locked
 echo "== test =="
 cargo test --workspace --all-targets --locked
+echo "== understanding study runner =="
+python3 scripts/test-understanding-study.py
 echo "== house style =="
 bash scripts/check-style.sh
 echo "All checks passed."

scripts/hooks/pre-commit

@@ -13,6 +13,8 @@
 #     full test suite) runs only when this commit actually touches Rust or the
 #     build manifest, so a docs-only commit stays fast while a code commit
 #     cannot land red.
+#   - The frozen 0.4 runner regressions run only when their runner, bank, or
+#     test file changes.
 #
 # Deliberately NOT here: coverage, the locked build, and artifact regeneration.
 # Those live in scripts/verify.sh (the release/CI gate); they are too slow to
@@ -43,4 +45,18 @@ if git -c core.quotePath=false diff --cached --name-only --diff-filter=ACMRD -z
   cargo test --workspace --all-targets --locked
 fi
 
+if git -c core.quotePath=false diff --cached --name-only --diff-filter=ACMRD -z \
+    | grep -zqE '^scripts/(understanding-study\.py|understanding-probes\.json|test-understanding-study\.py)$'; then
+  if command -v python3 >/dev/null 2>&1; then
+    study_python=python3
+  elif command -v python >/dev/null 2>&1; then
+    study_python=python
+  else
+    echo "pre-commit: Python 3.11 or newer is required for the 0.4 runner gate" >&2
+    exit 1
+  fi
+  echo "== understanding study runner =="
+  "$study_python" scripts/test-understanding-study.py
+fi
+
 echo "pre-commit: all checks passed."