Wire the vs-context7 drift check into CI so req-4's mechanical guarantee fires at the PR gate

[kiki830621]

Problem

The compare_context7.py --emit-readme --check drift guard (and its test_emit_readme_block_matches_committed_readme twin) is genuinely fail-loud when run, but .github/workflows/ci.yml runs only swift build + swift test — the Python suite is manual/periodic by project policy (CLAUDE.md: "CI runs swift build + swift test per push/PR"). So a reviewer who hand-edits the README competitive chart's numbers merges green through CI; the drift is caught only if a human independently remembers to run pytest or --check.

The verify ensemble (requirements + devil's-advocate lenses, #46) flagged this as the guard is least likely to fire precisely when it is needed — nothing prompts a --check after a prose/README edit, unlike sync-test-counts.sh which a dev naturally triggers when touching tests.

Why this is a separate issue (not part of #46)

#46's locked scope was README-top + chart + harness-generated, and its framing was explicitly "à la sync-test-counts.sh" — which is also manual. Making the Python suite (or a targeted --emit-readme --check step) part of CI is a pre-existing project CI-policy decision that predates #46 and affects the whole Python suite, not just the chart.

Options

• Add a lightweight python3 evals/look-up/compare_context7.py --emit-readme --check step to ci.yml (cheapest — just the chart drift, no claude -p cost).
• Or run the full pytest evals/look-up/tests/ in CI (broader, but the suite is deliberately kept off the critical path because run_eval.py makes real claude -p calls — note the unit tests don't).

Expected

A decision + (if yes) a CI step so the "measured, never asserted" guarantee is enforced mechanically at the PR gate, not just on a remembered manual run.

---

Source: surfaced during /idd-verify #46 (requirements + devil's-advocate lenses). Non-blocking follow-up — beyond #46's locked scope.

Current Status

Phase: closed (merged + verified; see Closing Summary)

[kiki830621]

Diagnosis

Type

feature (CI policy + workflow change)

Analysis

ci.yml runs only swift build + swift test. The Python suite (66 unit tests incl. the README-block drift guard test_emit_readme_block_matches_committed_readme, the CLI --emit-readme --check path, and the chart's semantic guard test_committed_svg_is_semantically_faithful) makes zero network / claude -p calls — the "keep Python off CI" policy protected against the live eval (run_eval.py), not the unit tests. So the cheapest complete fix is the issue's broader option: run the unit pytest suite in CI, which mechanically enforces both front-page guards (README block + SVG semantics) at the PR gate.

Decision (documented, unattended): full pytest evals/look-up/tests/ in CI rather than only the single --emit-readme --check step — same cost class (~1-2 min on ubuntu + pip), covers the chart guard and all harness regressions, and needs no bespoke step to keep in sync.

Critical exclusion: the byte-level --emit-chart --check must NOT run in CI — it byte-compares the committed SVG against a render by CI's matplotlib/freetype, which false-drifts across versions (#50 verify R1/R2 established this; the semantic guard is the version-independent CI-safe check, and it re-renders in CI's own env by design).

Strategy

• ci.yml: add a python job (ubuntu-latest, pip install -r evals/look-up/requirements.txt, python3 -m pytest evals/look-up/tests/) alongside the swift job.
• Update the CI-policy prose: CLAUDE.md ("CI runs swift build + swift test") + docs/wiki/Testing.md / Testing-zh-TW.md CI section — unit tests now per-PR; live evals (run_eval.py --runs N) remain manual (real claude -p calls).
• Verification is intrinsic: the PR adding the job must itself run it green.

Conflict Class

C_shared_module_coord — shared named resources with #49: .github/workflows/ + the CI-policy prose in CLAUDE.md/Testing.md. Serialized adjacently (#48 first; #49's scheduled workflow builds on this precedent in a separate file).

Complexity

Simple — one workflow job + doc prose; the decision space was already enumerated in the issue.

Risks

CI runtime creep (mitigated: unit tests run in <1s locally; pip install dominates). matplotlib install on ubuntu (wheel, fast).

Residue

(none) — the issue's ask is fully operationalizable.

[kiki830621]

Implementation Complete

Checklist

• ci.yml: new python-tests job (ubuntu, pip install requirements, pytest evals/look-up/tests/) → d718dbd
• CI-policy prose: CLAUDE.md + Testing.md/-zh-TW (live-eval wording scoped to run_eval.py) → d718dbd
• Verification intrinsic: PR CI: run the Python unit suite so the vs-context7 drift guards fire at the PR gate (#48) #54's own CI run must show the python unit tests check green

Design notes

• Included: the full 66-test unit suite (zero network / claude -p) — the "Python off CI" policy protected the live eval, not the unit tests. This fires the README-block drift guard AND the chart's matplotlib-version-independent semantic guard mechanically at the PR gate.
• Excluded (deliberate): --emit-chart --check (byte-compare — false-drifts across matplotlib/freetype per README vs-context7 chart: replace the ASCII block with a per-library matplotlib SVG (revises #46) #50 verify) and run_eval.py (real claude -p calls — stays manual).
• Injection surface: the new job has no ${{ }} expressions — static commands + stock actions only.

Sister Bugs Filed

(none) — the sample⟷reality half is #49 (PR #55, same batch).

PR #54. Next: PR's own CI check + /idd-verify #48. Stop at verified.

[kiki830621]

Verify: #48 — Round 1 (opus ensemble) + fixes

Verdict: FINDINGS · dispatch=opus · 4 lenses + devil's-advocate (Codex disabled — API session limit; process gap, non-blocking). Delivery confirmed (job runs the right guards; PR #54's own CI ran both checks green, incl. the SVG semantic guard re-rendering with Linux matplotlib). Findings + dispositions:

Sev	Finding	Fix
HIGH (DA)	Guards lock rendered artifacts ⟷ sample — editing the snapshot source + regenerating merges green; the live check was only monthly	FIXED via #55 (a4363b8): freshness.yml gains a pull_request: paths: trigger on the sample/corpus — any PR touching them runs the live registry check as a hard gate. Landed on #49's branch to avoid cross-PR conflicts; ci.yml comment cross-references it
HIGH (DA)	wiki Home pages' hand-written numbers (0 of 6, 6/6|0/6, react v18 vs 19.2.7) had zero guard yet publish to the public wiki via sync-wiki	FIXED via #55: test_home_pages_match_sample — every token derived from the sample (exact table row binds both scores in order; prose score; react pair; capture date; both languages)
MED (logic)	"matplotlib-version-independent by design" over-claims — a serialization change (entities/tspan) could false-fail	FIXED (71b02f5 + #55): prose → coordinate-independent + "false-FAILs loudly, never silently passes"; matplotlib <4 cap in requirements (via #55)
MED (security)	No permissions: block — pip+pytest run PR-influenced code with a default-scope token	FIXED (71b02f5): permissions: contents: read + timeout-minutes: 10
MED (regression)	Root README CI line left stale ("swift only")	FIXED (71b02f5)
LOW/INFO	header-x/overlay visual class beyond <text> (byte-check stays maintainer-local — accepted residual); actions tag-pinned (repo convention); fork-PR token read-only confirmed	noted

Round 2: a full re-ensemble is blocked this session (model-tier doctrine hard-denies new Workflow fan-outs); every fix above is finding-targeted, test-covered (216 green), and PR #54's CI re-ran green after the fix commit. Flagged honestly rather than skipped silently.

[kiki830621]

Closing Summary

Problem. The vs-context7 drift guards (README-block byte guard, chart semantic guard) were fail-loud but only when a human remembered to run pytest — ci.yml ran Swift only, so a hand-edited front-page competitive number merged green through CI. Surfaced by the #46 verify ensemble: the guard is least likely to fire precisely when it is needed.

Root cause. The "Python stays off CI" policy was protecting against the live eval (run_eval.py, real claude -p calls) but was over-applied to the unit suite, which makes zero network/API calls.

Solution. A python-tests job in ci.yml (ubuntu, pip install -r requirements.txt, full unit pytest) — the drift guards now fire mechanically at the PR gate. Deliberate exclusions, documented in the yml: --emit-chart --check (byte-compare, freetype-fragile across matplotlib versions per #50 verify — the semantic guard re-renders in CI's own env instead) and the live eval (stays periodic/manual). Hardened per verify: permissions: contents: read, timeout-minutes: 10, no ${{ }} event-payload interpolation.

Verification. PR #54's own CI ran both jobs green (python 15–18s — including the SVG semantic guard re-rendering with Linux matplotlib, empirically confirming its coordinate-independence). R1 opus ensemble (4 lenses + DA): delivery confirmed; 2 HIGHs (snapshot-source tampering passes the artifact guards; wiki Home numbers zero-guarded) closed via sibling #49/PR #55 — freshness.yml's pull_request: paths: trigger runs the live registry check as a hard gate on any sample-touching PR, and test_home_pages_match_sample derives the Home pages' numbers from the sample. All MEDIUMs fixed (71b02f5); prose de-overclaimed to "coordinate-independent".

Changes. PR #54 (2 commits), merged c525407. Docs: CLAUDE.md + Testing.md/-zh-TW + root README CI prose.

Residue (honest). The visual layer beyond <text> (coordinate/overlay tampering of the committed SVG) is guarded semantically for content/colour/column but not for arbitrary decoration — the byte-level --emit-chart --check remains the maintainer-local backstop. R2 confirmation ensemble not run this session (model-tier doctrine blocks new Workflow fan-outs) — flagged in the verify comment; all fixes finding-targeted + test-covered (216 green).