vs-context7 snapshot can silently rot: guard locks README==sample but nothing checks sample==registry

[kiki830621]

Problem

The README/wiki competitive claim is only as honest as compare_context7_sample.json's captured_at. The drift guard (--emit-readme --check + test_emit_readme_block_matches_committed_readme) verifies README ⟷ sample internal consistency, but nothing verifies sample ⟷ registry reality. The only reality check is --verify-live (re-fetches the registry ground truth for LiveDocs' side), and it is not wired into any CI / script / release gate.

The devil's-advocate lens (#46 verify) put it sharply: once the sample ages, the front-page captured 2026-07-02 · context7 0/6 silently rots into an asserted number — the exact failure mode the "measured, never asserted" iron rule (#27/#29) exists to prevent — while all tests + CI stay green.

Mitigations already in place (why this is low-urgency, not zero)

• The emitted block prints its captured_at, so a stale date is visible on the front page (not hidden). README: promote vs-context7 freshness stat + a harness-generated Unicode bar chart to the top #46 added captured_at validation so a missing/malformed date fails loud rather than shipping "captured None".
• context7's side is a recorded human judgment (its answer is prose, not a queryable version), so it structurally cannot be auto-re-captured — a refresh is inherently a deliberate human act.

Scope note

This is broader than #46 (README chart) — it's the snapshot-refresh discipline for the whole vs-context7 comparison (the wiki Home number has the same dependency). #46 doesn't make it worse.

Options

• A periodic --verify-live run (CI cron / release checklist) that warns when the LiveDocs-side ground truth has drifted from the recorded sample.
• A staleness budget: warn/fail if captured_at is older than N months.
• Document the refresh procedure (re-judge context7 + re-run + --emit-readme --write) as a release-checklist step.

---

Source: surfaced during /idd-verify #46 (devil's-advocate lens). Non-blocking follow-up — beyond #46's locked scope.

Current Status

Phase: closed (merged + verified; see Closing Summary)

[kiki830621]

Diagnosis

Type

feature (snapshot-freshness discipline: staleness budget + scheduled reality check)

Analysis

The guard chain locks README ⟷ sample, but sample ⟷ registry reality is checked only by --verify-live, which nothing runs. Two distinct failure clocks:

1. LiveDocs-side drift — the registry moves past the recorded ground_truth (machine-checkable via --verify-live, network).
2. Age itself — even if nothing drifted, a captured 2026-07-02 claim eventually rots into an asserted number (deterministic date math).

Design constraint: neither belongs in PR CI — a time/network-based failure would redden PRs unrelated to the chart (the exact "time-bomb red" anti-pattern), and #48 deliberately keeps PR CI deterministic. The right home is a scheduled workflow that fails loudly to the maintainer, not to PR authors.

Decision (documented, unattended):

• Add --check-staleness [--max-age-days N] (default 120) to compare_context7.py — pure date math on captured_at vs an injectable today (testable, no network), exit 1 when over budget.
• New .github/workflows/freshness.yml: monthly cron + manual dispatch, ubuntu, runs --verify-live (registry reality, LiveDocs side) then --check-staleness. A red scheduled run is the maintainer's refresh signal. Separate file → zero merge conflict with Wire the vs-context7 drift check into CI so req-4's mechanical guarantee fires at the PR gate #48's ci.yml job.
• Document the refresh procedure (re-judge context7 by hand → update sample + captured_at → --emit-chart/--emit-readme --write) in evals/look-up/README.md.
• context7's side stays a recorded human judgment (structurally not auto-recapturable — conceded in the issue).

Strategy

• compare_context7.py: staleness_days() pure helper + --check-staleness / --max-age-days CLI (TDD).
• Tests: staleness math (fresh/over-budget/boundary/malformed date) — count changes → run scripts/sync-test-counts.sh.
• .github/workflows/freshness.yml (cron monthly + workflow_dispatch).
• evals/look-up/README.md: refresh-procedure section.

Conflict Class

C_shared_module_coord — shares .github/workflows/ + compare_context7.py + Testing docs with #48; serialized after #48.

Complexity

Simple — small pure function + CLI flag + one workflow + docs; options were enumerated in the issue.

Risks

Registry flakiness in the cron (acceptable: a scheduled red is a signal, retryable). Staleness default (120d) is a judgment call — documented, tunable via flag.

Residue

The context7-side of the snapshot cannot be mechanically re-verified (human judgment by design) — the cron covers age + LiveDocs-side reality only. Stated, not hidden.

[kiki830621]

Implementation Complete

Checklist

• staleness_days() pure helper + --check-staleness / --max-age-days / --today CLI (TDD RED→GREEN) → 3f1e3e7
• --verify-live behavior change: warn → fail-loud exit 1 on drift (TDD; registry stubbed) → 3f1e3e7
• .github/workflows/freshness.yml: monthly cron + manual dispatch (NOT PR CI — time/network failures must never redden unrelated PRs) → 3f1e3e7
• evals/look-up/README.md § Refreshing the snapshot (the human procedure both error messages point to) → 3f1e3e7
• Count sync 210→215 via sync-test-counts.sh (+4 staleness, +1 verify-live) → d5db1d0

Design notes

• Injected --today: a wall-clock test would itself become the time bomb this feature catches — every staleness test pins its own today.
• Boundary semantics: age == budget is within; fail only when age > budget. A future captured_at fails loud (clock/snapshot error), never silently "fresh".
• Residue (from diagnosis, unchanged): context7's column is a recorded human judgment — the cron covers age + LiveDocs-side registry reality only; re-capture is deliberately a human act.

Sister Bugs Filed

(none) — the PR-gate half is #48 (PR #54, same batch).

PR #55. Next: /idd-verify #49. Stop at verified.

[kiki830621]

Verify: #49 — Round 1 (opus ensemble) + fixes

Verdict: FINDINGS · dispatch=opus · 4 lenses + devil's-advocate, daOk (Codex disabled — session limit; non-blocking). All four strategy items confirmed delivered; logic lens actively verified boundary math / monkeypatch / handler placement correct. Findings + dispositions (all fixed in a4363b8):

Sev	Finding	Fix
HIGH (DA)	GitHub auto-disables scheduled workflows after ~60d of repo inactivity — the cron dies quietly exactly when the repo goes dormant (= when rot happens); unacknowledged anywhere	Acknowledged in freshness.yml header + README (recovery path: Actions tab / manual run); the new pull_request: paths: trigger + workflow_dispatch are the activity-driven backstops. Honest residual: a fully dormant repo has no GitHub-side signal — stated, not hidden
MED (logic)	--verify-live treated fetch-failure (None) as "no drift" → false GREEN on registry outage + permanent silent-skip for unsupported ecosystems	Per-lib ⚠ unverified warning + verified X of N summary + exit 1 on drift OR unverified (distinct messages). Test extended: None stub → 1
MED (DA)	Fast-mover corpus → drift ~every month → permanent red trains the maintainer to ignore it; 120d unargued	Scheduled runs: drift step continue-on-error (informational); staleness budget is the hard red (one deliberate nag per budget). PR runs keep both hard. 120d rationale documented (≈ one release cycle; tunable)
MED (regression)	Staleness tests hardcoded dates against the committed sample — the documented refresh procedure would break them (empirically demonstrated by the reviewer)	Pinned _pinned_sample fixture — refresh-proof; README step 4 notes it
MED (regression)	Root README still said --verify-live "warns" (+ the module docstring, per the DA)	Both fixed; --check-staleness documented alongside
LOW	--today lenient-ISO; no permissions:; step-2 "bump the date" trust-based without warning	Strict ISO round-trip (tested: 20261030 rejected); permissions: contents: read + timeout; "the date is an attestation, not a proof" honesty note added

Also landed here (from #48's ensemble, by design): the sample-tamper PR gate + the Home-pages guard test. Tests 216 (144 Swift + 72 Python), counts synced.

Round 2: blocked this session (model-tier doctrine hard-denies new Workflow fan-outs) — fixes are finding-targeted + test-covered; flagged honestly.

[kiki830621]

Closing Summary

Problem. The guard chain locked README ⟷ sample, but nothing checked sample ⟷ reality: --verify-live was wired to nothing and only printed warnings, and nothing bounded the snapshot's age — captured 2026-07-02 would silently rot into an asserted number (#27/#29) with every test green. Surfaced by the #46 verify devil's-advocate.

Solution — split along the determinism boundary. Time/network checks must never redden unrelated PRs, so they live outside PR CI:

• --check-staleness [--max-age-days 120] [--today ISO] — pure date math with an injected today (a wall-clock test would itself be the time bomb); over-budget / future / malformed / lenient-ISO dates all exit 1. 120d ≈ one release cycle, documented + tunable.
• --verify-live → fail-loud: exits 1 on drift or when any library couldn't be verified (verified X of N summary — a registry outage is no longer a false GREEN).
• .github/workflows/freshness.yml — monthly cron + workflow_dispatch + (per verify) pull_request: paths: on the sample/corpus, making the live check a hard PR gate against snapshot tampering (closing Wire the vs-context7 drift check into CI so req-4's mechanical guarantee fires at the PR gate #48's HIGH). On scheduled runs the drift step is informational (drift on a fast-mover corpus is expected — no alarm fatigue); the staleness budget is the hard red.
• evals/look-up/README.md § Refreshing the snapshot — the human procedure both error messages point to, including two honesty notes: the 60-day GitHub auto-disable of scheduled workflows on dormant repos (acknowledged with recovery paths — no mechanism survives full dormancy), and "the date is an attestation, not a proof" (context7's column is a recorded human judgment; the procedure is trust-based by design).
• test_home_pages_match_sample — the wiki Home pages' hand-written competitive numbers, published to the public wiki, now derive their guard from the sample (both languages).

Verification. TDD throughout (RED→GREEN, injected clocks, stubbed registry); R1 opus ensemble (4 lenses + DA, daOk) — 1 HIGH + 4 MED all fixed in a4363b8 (staleness tests decoupled from the committed sample via a pinned fixture so the refresh procedure can't break them; stale "warn" wording fixed in root README + module docstring; matplotlib capped <4). 216 tests (144 Swift + 72 Python) green on merged main; --check-staleness live: ✓ snapshot age 4d within the 120d budget.

Changes. PR #55 (4 commits), merged ad0e04c.

Residue (honest). context7's column structurally cannot be auto-re-verified (recorded human judgment) — the machinery covers age + LiveDocs-side registry reality only. A fully dormant repo has no GitHub-side signal (60-day limit acknowledged, not solved). R2 confirmation ensemble not run this session (model-tier doctrine) — flagged in the verify comment.