PsychQuant · kiki830621 · Jul 5, 2026 · Jul 5, 2026 · Jul 5, 2026 · Jul 5, 2026
diff --git a/.github/workflows/freshness.yml b/.github/workflows/freshness.yml
@@ -0,0 +1,64 @@
+# vs-context7 snapshot freshness (#49): the PR-gate guards (ci.yml) lock README ⟷ sample,
+# but nothing there checks sample ⟷ REALITY. This workflow is that reality check:
+#   1. --verify-live   — re-fetches the registry ground truth (LiveDocs' side; network)
+#                        and exits 1 on drift OR when any library couldn't be verified;
+#   2. --check-staleness — exits 1 when captured_at exceeds the age budget (default 120d).
+#
+# Trigger design (#48/#49 verify round 1):
+# - schedule: monthly reality check. Drift on the fast-mover corpus is EXPECTED over time,
+#   so on scheduled runs the drift step is informational (continue-on-error) and the HARD
+#   red is the staleness budget — one deliberate nag per budget, not a permanent alarm
+#   that trains the maintainer to ignore it.
+# - pull_request (paths): any PR touching the snapshot/corpus runs the LIVE check as a
+#   hard PR gate — a tampered or mis-refreshed sample (fabricated ground_truth) goes red
+#   on the PR itself, closing the "edit the snapshot source and regenerate" hole (#48).
+#   Network cost only on the rare sample-touching PR, never on unrelated PRs.
+# - workflow_dispatch: manual re-run / reproduction.
+#
+# ⚠ KNOWN PLATFORM LIMIT (#49 verify): GitHub auto-disables scheduled workflows after
+# ~60 days of repository inactivity — i.e. the cron dies quietly exactly when the repo
+# goes dormant (which is also when the snapshot rots). If returning to a dormant repo:
+# check the Actions tab (re-enable if disabled) or run --check-staleness by hand.
+# The refresh procedure lives in evals/look-up/README.md § Refreshing the snapshot.
+#
+# Injection surface: the only ${{ }} is `github.event_name` (a trusted context value in
+# a YAML field, not event-payload text in a shell). No secrets; least-privilege token.
+name: vs-context7 freshness
+
+on:
+  schedule:
+    - cron: "17 3 1 * *"   # monthly, 1st @ 03:17 UTC (off-peak, non-round minute)
+  pull_request:
+    paths:
+      - "evals/look-up/compare_context7_sample.json"
+      - "evals/look-up/compare_corpus.yaml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  freshness:
+    name: snapshot reality check
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install eval-harness dependencies
+        run: pip install -r evals/look-up/requirements.txt
+
+      - name: Registry drift check (LiveDocs-side ground truth)
+        # Informational on SCHEDULED runs (drift on fast-movers is expected over time;
+        # the staleness budget below is the hard signal). Hard-fails on PR runs — a
+        # sample-touching PR must match live reality at merge time.
+        continue-on-error: ${{ github.event_name == 'schedule' }}
+        run: python3 evals/look-up/compare_context7.py --verify-live
+
+      - name: Staleness budget (captured_at age)
+        if: always()   # surface both axes in one run, even if the drift step failed
+        run: python3 evals/look-up/compare_context7.py --check-staleness
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -19,7 +19,7 @@ Swift MCP server (engine) + `look-up` skill (routing) + Python evals.
 
 ```bash
 swift test                                   # 144 Swift tests (no network)
-python3 -m pytest evals/look-up/tests/   # 66 Python tests (no API calls)
+python3 -m pytest evals/look-up/tests/   # 72 Python tests (no API calls)
 python3 evals/look-up/compare_context7.py --verify-live   # freshness snapshot drift check
 make release-signed VERSION=vX.Y.Z           # build + sign + notarize + mcpb (needs DEVELOPER_ID/NOTARY_PROFILE)
 bash scripts/sync-wiki.sh                    # mirror docs/wiki/ -> GitHub wiki (after merge)

diff --git a/README.md b/README.md
@@ -104,7 +104,7 @@ than hardcoding a fact), and it's statistical (N runs, rate thresholds).
 
 ```bash
 pip install -r evals/look-up/requirements.txt
-python3 -m pytest evals/look-up/tests/     # 66 harness unit tests (no API calls)
+python3 -m pytest evals/look-up/tests/     # 72 harness unit tests (no API calls)
 python3 evals/look-up/run_eval.py --dry-run
 python3 evals/look-up/run_eval.py --runs 5 # live baseline (real `claude -p` calls)
 ```
@@ -113,13 +113,14 @@ The same directory carries the **vs-context7 freshness comparison** — a dated,
 head-to-head (registry = neutral ground truth) whose numbers the wiki homepage cites:
 
 ```bash
-python3 evals/look-up/compare_context7.py                # table + tally
-python3 evals/look-up/compare_context7.py --verify-live  # warn if the snapshot drifted
+python3 evals/look-up/compare_context7.py                    # table + tally
+python3 evals/look-up/compare_context7.py --verify-live      # exit 1 if the registry drifted past the snapshot
+python3 evals/look-up/compare_context7.py --check-staleness  # exit 1 if the snapshot is over its age budget
 ```
 
 See [`evals/look-up/README.md`](evals/look-up/README.md) for the design of both, and
 the [Testing wiki page](https://github.com/PsychQuant/livedocs/wiki/Testing) for the full
-suite breakdown (210 tests: 144 Swift + 66 Python).
+suite breakdown (216 tests: 144 Swift + 72 Python).
 
 Status: shipped (v0.8.0; engine features below shipped v0.7.0, the skill's `look-up` rename +
 explicit invocation landed in v0.8.0). 9-ecosystem registry resolution, version pinning (npm/pypi/crates/go/rubygems), OpenAPI/GraphQL/CLI +

diff --git a/docs/wiki/Home-zh-TW.md b/docs/wiki/Home-zh-TW.md
@@ -42,7 +42,7 @@ context7 是以覆蓋率排序、週期性重爬的索引; LiveDocs 直接抓 li
 
 - [版本協調流程](Version-Reconciliation-zh-TW): 自動偵測更新流程。
 - [Primary-Source 光譜](Primary-Source-Spectrum-zh-TW): LiveDocs 是什麼、不是什麼。
-- [測試](Testing-zh-TW): 測試套件（210 個 test）與各自覆蓋範圍。
+- [測試](Testing-zh-TW): 測試套件（216 個 test）與各自覆蓋範圍。
 
 ## 其他
 

diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md
@@ -57,7 +57,7 @@ Then use the `look-up` skill; it routes questions to the right tool.
 
 - [Version Reconciliation](Version-Reconciliation): the auto-detect update flow.
 - [Primary-Source Spectrum](Primary-Source-Spectrum): what LiveDocs is and isn't for.
-- [Testing](Testing): the test suites (210 tests) and what each covers.
+- [Testing](Testing): the test suites (216 tests) and what each covers.
 
 ## More
 

diff --git a/docs/wiki/Testing-zh-TW.md b/docs/wiki/Testing-zh-TW.md
@@ -2,13 +2,13 @@
 
 # 測試
 
-LiveDocs 有兩套獨立 test suite —— **210 個 test,全綠**。Swift suite 驗證引擎;Python suite
+LiveDocs 有兩套獨立 test suite —— **216 個 test,全綠**。Swift suite 驗證引擎;Python suite
 驗證 `look-up` *skill* 是否真的會激起 LiveDocs 查詢並給出現行答案, 外加 vs-context7 比較
 harness。以下計數是快照;真正的 source of truth 是實際跑一次。
 
 ```bash
 swift test                                  # 144 個 Swift test
-python3 -m pytest evals/look-up/tests/  # 66 個 Python eval test
+python3 -m pytest evals/look-up/tests/  # 72 個 Python eval test
 ```
 
 ## Swift — 144 個 test（`swift test`）
@@ -43,7 +43,7 @@ python3 -m pytest evals/look-up/tests/  # 66 個 Python eval test
 | `ProcessRunnerTests` | 4 | 大輸出不 deadlock(並發讀 pipe)、exit code 浮現、timeout 回報、SIGTERM→SIGKILL escalation。 |
 | `LatestVersionEncodingTests` | 14 | `latest_version` 工具邊界:pin_honored true/false/nil surfacing(含 nothing-resolved 路徑的 false、無 pin 時缺席)、Go `v`-prefix 已確認 pin(不可誤判成 not-applied)、v-prefix 輸入建出正確 confirm URL、npm 找不到→latest+false、npm dist-tag/exact-pin 區分(移動 tag 絕不假裝成 pin)、RubyGems 端到端 pin(honored + not-found)。 |
 
-## Python — 66 個 test（`pytest evals/look-up/`）
+## Python — 72 個 test（`pytest evals/look-up/`）
 
 `look-up` **skill eval harness** —— 不是測 Swift 引擎,而是測 skill 對多種 prompt 會不會
 激起 LiveDocs 查詢並答對現行 —— 外加 vs-context7 比較 harness。見
@@ -52,7 +52,7 @@ python3 -m pytest evals/look-up/tests/  # 66 個 Python eval test
 | 檔案 | 數 | 測什麼 |
 |------|---:|--------|
 | `test_run_eval.py` | 17 | rate 門檻判定、N=3 門檻塌縮 guard、failed-run / inconclusive 處理、explicit route 比對、`args_match`——對的工具但錯的參數(mis-split `name@version` / 掉 ecosystem pin)判為不正確。 |
-| `test_compare.py` | 20 | vs-context7 新鮮度 harness:版本 token 比對(邊界形式)、對稱計分、表格 render、README 圖片嵌入區塊漂移守衛(唯一*完整*區塊含 orphaned sentinel、ISO 日曆日期驗證、`main()` `--check`/`--write` exit code + write round-trip)、corpus↔sample 同步。 |
+| `test_compare.py` | 26 | vs-context7 新鮮度 harness:版本 token 比對(邊界形式)、對稱計分、表格 render、README 圖片嵌入區塊漂移守衛(唯一*完整*區塊含 orphaned sentinel、ISO 日曆日期驗證、`main()` `--check`/`--write` exit code + write round-trip)、corpus↔sample 同步、snapshot 時效預算(`--check-staleness` 注入 `--today` + pinned sample fixture —— 純數學 / fresh / 超額 / 邊界 / 未來 / 格式錯 / 寬鬆 ISO 拒絕)、`--verify-live` fail-loud exit code(drift 與 unverified 都紅;registry stub)、wiki Home 手寫競品數字對 sample 守衛(表格列 + 散文比分 + react 版本對 + 擷取日期,雙語)。 |
 | `test_chart.py` | 8 | 逐庫 vs-context7 SVG 圖:`chart_rows` 是 snapshot 的 verbatim 投影、committed SVG **語意**忠實(逐庫 cell 以 `version_matches` 綁對欄位、headline == `tally()`、證明能抓欄位對調)——matplotlib 版本無關的漂移守衛、render 確定性、`--emit-chart` `--check`/`--write` CLI(drift / write round-trip / flag 互斥)。 |
 | `test_oracle.py` | 8 | `self_check`(eval 時打 registry —— 防腐爛)/ `structural` / `golden` oracle。 |
 | `test_detect.py` | 9 | `claude -p` stream-json 解析、`is_error` 偵測、LiveDocs 觸發訊號辨識、逐 call 工具輸入捕捉(short name + input args)。 |

diff --git a/docs/wiki/Testing.md b/docs/wiki/Testing.md
@@ -2,14 +2,14 @@
 
 # Testing
 
-LiveDocs has two independent test suites — **210 tests, all green**. The Swift suite verifies
+LiveDocs has two independent test suites — **216 tests, all green**. The Swift suite verifies
 the engine; the Python suite verifies that the `look-up` *skill* actually triggers a
 LiveDocs query and answers currently, plus the vs-context7 comparison harness. Counts below
 are a snapshot; the source of truth is running the suites.
 
 ```bash
 swift test                                  # 144 Swift tests
-python3 -m pytest evals/look-up/tests/  # 66 Python eval tests
+python3 -m pytest evals/look-up/tests/  # 72 Python eval tests
 ```
 
 ## Swift — 144 tests (`swift test`)
@@ -45,7 +45,7 @@ and file-system layer.
 | `ProcessRunnerTests` | 4 | Large output doesn't deadlock (concurrent pipe drain), exit code surfaces, timeout reported, SIGTERM→SIGKILL escalation. |
 | `LatestVersionEncodingTests` | 14 | `latest_version` tool boundary: `pin_honored` true/false/nil surfacing (incl. `false` in the nothing-resolved path, and absent when no pin asked), the Go `v`-prefix confirmed-pin case (must not read as not-applied), v-prefixed input building a valid confirm URL, npm not-found → latest+false, npm dist-tag / exact-pin discrimination (a moving tag is never a faked pin), RubyGems end-to-end pin (honored + not-found). |
 
-## Python — 66 tests (`pytest evals/look-up/`)
+## Python — 72 tests (`pytest evals/look-up/`)
 
 The `look-up` **skill eval harness** — not a test of the Swift engine, but of whether
 the skill fires a LiveDocs query for varied prompts and answers currently — plus the
@@ -55,7 +55,7 @@ vs-context7 comparison harness. See
 | File | Tests | Covers |
 |------|------:|--------|
 | `test_run_eval.py` | 17 | Rate-threshold judging, the N=3 threshold-collapse guard, failed-run / inconclusive handling, explicit-invocation route matching, and `args_match` — the right tool with the wrong args (a mis-split `name@version` / dropped ecosystem pin) scores incorrect. |
-| `test_compare.py` | 20 | vs-context7 freshness harness: version-token matching (boundary forms), symmetric scoring, table render, README image-embed block drift guard (exactly-one *well-formed* block incl. orphaned sentinel, ISO calendar-date validation, `main()` `--check`/`--write` exit codes + write round-trip), corpus↔sample sync. |
+| `test_compare.py` | 26 | vs-context7 freshness harness: version-token matching (boundary forms), symmetric scoring, table render, README image-embed block drift guard (exactly-one *well-formed* block incl. orphaned sentinel, ISO calendar-date validation, `main()` `--check`/`--write` exit codes + write round-trip), corpus↔sample sync, snapshot-staleness budget (`--check-staleness` with injected `--today` + pinned sample fixture — pure math / fresh / over-budget / boundary / future / malformed / lenient-ISO), `--verify-live` fail-loud exit codes (drift AND unverified both red; registry stubbed), and the wiki Home pages' hand-written competitive numbers vs the sample (table row + prose score + react pair + capture date, both languages). |
 | `test_chart.py` | 8 | Per-library vs-context7 SVG chart: `chart_rows` is a verbatim snapshot projection; the committed SVG is **semantically** faithful (each per-library cell bound to the right column via `version_matches`, headline == `tally()`, proven to catch a column swap) — a matplotlib-version-independent drift guard; deterministic render; `--emit-chart` `--check`/`--write` CLI (drift / write round-trip / flag mutual-exclusion). |
 | `test_oracle.py` | 8 | `self_check` (fetch registry at eval time — rot-proof) / `structural` / `golden` oracles. |
 | `test_detect.py` | 9 | `claude -p` stream-json parsing, `is_error` detection, LiveDocs trigger-signal recognition, and per-call tool-input capture (short name + input args). |

diff --git a/evals/look-up/README.md b/evals/look-up/README.md
@@ -104,7 +104,8 @@ registry as neutral ground truth.
 ```bash
 python3 evals/look-up/compare_context7.py            # print the table + tally
 python3 evals/look-up/compare_context7.py --json     # machine-readable
-python3 evals/look-up/compare_context7.py --verify-live   # warn if the snapshot drifted
+python3 evals/look-up/compare_context7.py --verify-live       # exit 1 if the registry moved past the snapshot
+python3 evals/look-up/compare_context7.py --check-staleness   # exit 1 if captured_at is over the age budget (120d)
 ```
 
 - Corpus: [`compare_corpus.yaml`](compare_corpus.yaml) — fast-moving libraries across
@@ -135,9 +136,9 @@ harness lets the data show it and **concedes** what context7 is good at:
   answer, not "context7 can't find the version." Per library the default is either
   *behind* (react, vite) or *version-less* (fastapi, pydantic, tokio, serde) — the
   sample records which.
-- **`--verify-live` is one-sided:** it re-fetches the registry (LiveDocs' side) and warns
-  on drift, but does not re-query context7. The context7 column is a dated snapshot;
-  refresh it by re-running both MCP tools.
+- **`--verify-live` is one-sided:** it re-fetches the registry (LiveDocs' side) and exits
+  non-zero on drift (#49), but does not re-query context7. The context7 column is a dated
+  snapshot; refresh it by re-running both MCP tools.
 - **D1 (this harness):** a dated recorded head-to-head. **D2 (future, not built):**
   a fully-automated live A/B that shells `claude -p` per tool — stronger but
   costly/flaky, so it stays out of the default path.
@@ -146,6 +147,51 @@ Result at capture (2026-07-02): context7's top-ranked default match reflected th
 release on **0 of 6** (behind for react/vite, version-less for fastapi/pydantic/tokio/serde);
 LiveDocs returned the live registry version each time.
 
+### Refreshing the snapshot (#49)
+
+The front-page claim is only as honest as the snapshot's `captured_at`. Two mechanical
+signals tell you it's time (run monthly by `.github/workflows/freshness.yml`, and either
+can be run by hand):
+
+- `--verify-live` exits 1 → the registry has moved past a recorded `ground_truth`, **or**
+  some library could not be verified (registry unreachable / unsupported ecosystem —
+  never silently counted as "no drift"). On the *scheduled* run this step is
+  **informational** (drift on a deliberately fast-moving corpus is expected over time);
+  on a PR that touches the sample it is a **hard gate** — a tampered or mis-refreshed
+  `ground_truth` reddens that PR against live reality.
+- `--check-staleness` exits 1 → `captured_at` is over the age budget — the **hard**
+  monthly signal. Default 120 days ≈ one typical release cycle for this project — long
+  enough that refreshes aren't busywork, short enough that a dated front-page claim
+  can't quietly become ancient; tunable via `--max-age-days`.
+
+Two honesty notes on the machinery itself:
+
+- **GitHub platform limit**: scheduled workflows are auto-disabled after ~60 days of
+  repository inactivity — the cron dies quietly exactly when the repo goes dormant. If
+  returning after a quiet stretch, check the Actions tab (re-enable if disabled) or run
+  `--check-staleness` by hand.
+- **The date is an attestation, not a proof.** Bumping `captured_at` asserts you actually
+  re-judged context7's side (step 1 below). Bumping the date without re-judging is
+  fabricating a capture — no tool can catch it (context7's column is a recorded human
+  judgment); the procedure is trust-based by design.
+
+Refresh procedure (deliberately a **human** act — context7's column is a recorded
+judgment, not queryable):
+
+1. Re-run both MCP tools per library in `compare_corpus.yaml`: LiveDocs `latest_version`,
+   and context7's resolve→query flow; judge whether context7's **top-ranked default**
+   match reflects the current release.
+2. Update `compare_context7_sample.json`: `ground_truth` + `livedocs.answer` verbatim,
+   `context7.answer`/`is_current`/`default_version`/`default_status` per the new
+   judgment, and bump `captured_at` to today.
+3. Regenerate the committed artifacts:
+   `python3 evals/look-up/compare_context7.py --emit-chart --write` then
+   `--emit-readme --write`.
+4. `python3 -m pytest evals/look-up/tests/` — the drift guards confirm README ⟷ sample
+   ⟷ SVG (and the wiki Home numbers) are in sync again; commit sample + SVG + README
+   together. The staleness tests pin their own sample fixture, so a refresh never
+   breaks them.
+
 ## Scope / caveats
 
 - Measures a **sample**, not a proof of universal correctness — the corpus must