From d718dbdd140f0410cc0e6b7b8764a2de72772f0b Mon Sep 17 00:00:00 2001 From: che cheng Date: Sun, 5 Jul 2026 23:08:52 +0800 Subject: [PATCH 1/2] =?UTF-8?q?feat:=20run=20the=20Python=20unit=20suite?= =?UTF-8?q?=20in=20CI=20=E2=80=94=20vs-context7=20guards=20fire=20at=20the?= =?UTF-8?q?=20PR=20gate=20(#48)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The drift guards (README-block byte guard + the chart's matplotlib-version- independent SEMANTIC guard) were genuinely fail-loud but only when a human remembered to run pytest — CI ran swift only, so a hand-edited front-page competitive number merged green. The Python unit suite makes zero network / `claude -p` calls (that policy protected the LIVE eval, not the unit tests), so it now runs per push/PR on ubuntu. Deliberately excluded from CI: `--emit-chart --check` (byte-compare — false-drifts across matplotlib/freetype; the semantic guard re-renders in CI's own env and compares content instead) and run_eval.py (real `claude -p` calls — stays periodic/manual). Docs: CLAUDE.md + Testing.md/-zh-TW CI sections updated (live eval wording scoped to run_eval.py). No test-count changes. Injection surface: the new job contains no ${{ }} expressions at all — static commands + stock actions only. --- .github/workflows/ci.yml | 25 +++++++++++++++++++++++++ CLAUDE.md | 6 ++++-- docs/wiki/Testing-zh-TW.md | 9 ++++++--- docs/wiki/Testing.md | 11 ++++++++--- 4 files changed, 43 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6086e8b..aa7ae6a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,3 +33,28 @@ jobs: - name: Test run: swift test + + # Python unit suite (#48): no network, no `claude -p` — safe on the PR critical path. + # This is what makes the front-page vs-context7 guards fire MECHANICALLY at the PR + # gate: the README-block drift guard (test_emit_readme_block_matches_committed_readme, + # + the `--emit-readme --check` CLI path) and the chart's SEMANTIC guard + # (test_committed_svg_is_semantically_faithful — matplotlib-version-independent by + # design, it re-renders in THIS env and compares content, not bytes). + # Deliberately NOT run here: `--emit-chart --check` (byte-compare — false-drifts + # across matplotlib/freetype versions; #50 verify) and the LIVE eval + # (run_eval.py — real `claude -p` calls; stays periodic/manual per project policy). + python-tests: + name: python unit tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install eval-harness dependencies + run: pip install -r evals/look-up/requirements.txt + + - name: Run unit tests (incl. vs-context7 drift guards) + run: python3 -m pytest evals/look-up/tests/ -q diff --git a/CLAUDE.md b/CLAUDE.md index 7b7858c..bf56cd5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,8 +25,10 @@ make release-signed VERSION=vX.Y.Z # build + sign + notarize + mcpb (n bash scripts/sync-wiki.sh # mirror docs/wiki/ -> GitHub wiki (after merge) ``` -CI runs `swift build` + `swift test` per push/PR. `scripts/release.sh` gates on green tests + -version-source/tag match. Version source of truth: `Sources/LiveDocsCore/Version.swift`. +CI runs `swift build` + `swift test` AND the Python unit suite per push/PR (the Python job +is what fires the vs-context7 drift guards mechanically at the PR gate, #48 — it makes no +network / API calls). `scripts/release.sh` gates on green tests + version-source/tag match. +Version source of truth: `Sources/LiveDocsCore/Version.swift`. Live evals (`run_eval.py --runs N`) are periodic/manual — real `claude -p` calls; run from a clean non-nested directory with credits. diff --git a/docs/wiki/Testing-zh-TW.md b/docs/wiki/Testing-zh-TW.md index f30dd16..4ffb96f 100644 --- a/docs/wiki/Testing-zh-TW.md +++ b/docs/wiki/Testing-zh-TW.md @@ -60,11 +60,14 @@ python3 -m pytest evals/look-up/tests/ # 66 個 Python eval test ## CI 與 release gate -- **CI** 每個 push / PR 跑 `swift build` + `swift test`。 +- **CI** 每個 push / PR 跑 `swift build` + `swift test` **加上 Python unit suite**(#48)。 + Python job 不碰網路 / API;它讓 vs-context7 漂移守衛(README 區塊 + 圖的 matplotlib + 版本無關語意守衛)在 PR gate **機械性**觸發。byte-level `--emit-chart --check` 留在 + maintainer 本機(跨 matplotlib 版本 freetype-fragile)。 - **`scripts/release.sh`** 把 release gate 在:`swift test` 綠、版本 source 與 tag 相符、 Developer ID 簽章 + notarize。 -- **Python eval** 是 periodic / manual,不進 per-PR CI —— 它會真的呼叫 `claude -p`(有成本 + - stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。 +- **live Python eval**(`run_eval.py`)是 periodic / manual,不進 per-PR CI —— 它會真的呼叫 + `claude -p`(有成本 + stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。 ## 紀律 diff --git a/docs/wiki/Testing.md b/docs/wiki/Testing.md index 7ab8521..b7dbae5 100644 --- a/docs/wiki/Testing.md +++ b/docs/wiki/Testing.md @@ -63,11 +63,16 @@ vs-context7 comparison harness. See ## CI and the release gate -- **CI** runs `swift build` + `swift test` on every push and pull request. +- **CI** runs `swift build` + `swift test` **and the Python unit suite** on every push and + pull request (#48). The Python job makes no network / API calls; it is what fires the + vs-context7 drift guards (README block + the chart's matplotlib-version-independent + semantic guard) mechanically at the PR gate. The byte-level `--emit-chart --check` stays + maintainer-local (freetype-fragile across matplotlib versions). - **`scripts/release.sh`** gates a release on a green `swift test`, a version-source match against the tag, and Developer ID signing + notarization. -- The **Python eval** is periodic / manual, not per-PR CI — it makes real `claude -p` calls - (cost + stochastic), so it runs as a maintainer baseline rather than on the critical path. +- The **live Python eval** (`run_eval.py`) is periodic / manual, not per-PR CI — it makes + real `claude -p` calls (cost + stochastic), so it runs as a maintainer baseline rather + than on the critical path. ## Discipline From 71b02f523a7ab7a3dbba0786e6063e2eb0adc843 Mon Sep 17 00:00:00 2001 From: che cheng Date: Sun, 5 Jul 2026 23:33:31 +0800 Subject: [PATCH 2/2] fix: harden CI job per #48 verify round 1 (#48) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - permissions: contents: read at workflow level — pip install + pytest execute PR-influenced code; don't hand them a write-scoped GITHUB_TOKEN (MEDIUM). - timeout-minutes: 10 on the python job (hang guard). - De-overclaim "matplotlib-version-independent by design" → COORDINATE-independent: the semantic guard compares content, so a matplotlib serialization change (entities/tspan) would false-FAIL loudly, never silently pass; matplotlib gets a <4 upper bound via #49's requirements pin (MEDIUM, ci.yml comment + Testing ×2). - README.md CI line was left stale ("swift only") — now includes the Python suite (MEDIUM). - Guard-scope comment: the two R1 HIGHs (snapshot-SOURCE tampering passes the PR gate; wiki Home numbers zero-guarded) are closed by sibling #49/PR #55 — the freshness workflow gains a pull_request paths trigger on the sample (live registry check on any PR touching it) and a Home-pages guard test; landed there to avoid cross-PR count/file conflicts. Cross-referenced in both issues. --- .github/workflows/ci.yml | 16 ++++++++++++++-- README.md | 5 +++-- docs/wiki/Testing-zh-TW.md | 5 +++-- docs/wiki/Testing.md | 8 +++++--- 4 files changed, 25 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aa7ae6a..9ca0c61 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,6 +9,11 @@ concurrency: group: ci-${{ github.ref }} cancel-in-progress: true +# Least privilege (#48 verify): neither job uses the token — pip install + pytest +# execute PR-influenced code, so don't hand them a write-scoped GITHUB_TOKEN. +permissions: + contents: read + jobs: test: name: swift test @@ -38,14 +43,21 @@ jobs: # This is what makes the front-page vs-context7 guards fire MECHANICALLY at the PR # gate: the README-block drift guard (test_emit_readme_block_matches_committed_readme, # + the `--emit-readme --check` CLI path) and the chart's SEMANTIC guard - # (test_committed_svg_is_semantically_faithful — matplotlib-version-independent by - # design, it re-renders in THIS env and compares content, not bytes). + # (test_committed_svg_is_semantically_faithful — it re-renders in THIS env and + # compares content, not bytes/coordinates, so it is COORDINATE-independent; + # a future matplotlib changing text *serialization* would false-FAIL loudly, never + # silently pass — regenerate the asset to fix; requirements.txt caps matplotlib <4). # Deliberately NOT run here: `--emit-chart --check` (byte-compare — false-drifts # across matplotlib/freetype versions; #50 verify) and the LIVE eval # (run_eval.py — real `claude -p` calls; stays periodic/manual per project policy). + # Guard scope (#48 verify): these guards lock rendered artifacts ⟷ snapshot. + # Snapshot-SOURCE tampering is caught by freshness.yml's pull_request paths trigger + # (live registry check on any PR touching the sample), and the wiki Home numbers by + # test_home_pages_match_sample — both landed via #49 (same batch). python-tests: name: python unit tests runs-on: ubuntu-latest + timeout-minutes: 10 steps: - uses: actions/checkout@v4 diff --git a/README.md b/README.md index 19ce39c..d4b3107 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,9 @@ swift test # unit tests (pure logic + engine via injected HTTP fakes) swift build # builds the MCP executable ``` -CI runs `swift build` + `swift test` on every push/PR. `scripts/release.sh` gates the release on a -green `swift test`, a version-source match against the tag, and Developer ID signing + notarization. +CI runs `swift build` + `swift test` and the Python unit suite on every push/PR (#48). +`scripts/release.sh` gates the release on a green `swift test`, a version-source match +against the tag, and Developer ID signing + notarization. ### Skill eval (`evals/look-up/`) diff --git a/docs/wiki/Testing-zh-TW.md b/docs/wiki/Testing-zh-TW.md index 4ffb96f..08b4225 100644 --- a/docs/wiki/Testing-zh-TW.md +++ b/docs/wiki/Testing-zh-TW.md @@ -61,8 +61,9 @@ python3 -m pytest evals/look-up/tests/ # 66 個 Python eval test ## CI 與 release gate - **CI** 每個 push / PR 跑 `swift build` + `swift test` **加上 Python unit suite**(#48)。 - Python job 不碰網路 / API;它讓 vs-context7 漂移守衛(README 區塊 + 圖的 matplotlib - 版本無關語意守衛)在 PR gate **機械性**觸發。byte-level `--emit-chart --check` 留在 + Python job 不碰網路 / API;它讓 vs-context7 漂移守衛(README 區塊 + 圖的**座標無關** + 語意守衛 —— 比對 `` 內容,若未來 matplotlib 改變文字序列化只會**大聲 false-FAIL**、 + 絕不靜默通過)在 PR gate **機械性**觸發。byte-level `--emit-chart --check` 留在 maintainer 本機(跨 matplotlib 版本 freetype-fragile)。 - **`scripts/release.sh`** 把 release gate 在:`swift test` 綠、版本 source 與 tag 相符、 Developer ID 簽章 + notarize。 diff --git a/docs/wiki/Testing.md b/docs/wiki/Testing.md index b7dbae5..294abf3 100644 --- a/docs/wiki/Testing.md +++ b/docs/wiki/Testing.md @@ -65,9 +65,11 @@ vs-context7 comparison harness. See - **CI** runs `swift build` + `swift test` **and the Python unit suite** on every push and pull request (#48). The Python job makes no network / API calls; it is what fires the - vs-context7 drift guards (README block + the chart's matplotlib-version-independent - semantic guard) mechanically at the PR gate. The byte-level `--emit-chart --check` stays - maintainer-local (freetype-fragile across matplotlib versions). + vs-context7 drift guards (README block + the chart's coordinate-independent semantic + guard — it compares `` content, so a matplotlib serialization change would + false-FAIL loudly, never silently pass) mechanically at the PR gate. The byte-level + `--emit-chart --check` stays maintainer-local (freetype-fragile across matplotlib + versions). - **`scripts/release.sh`** gates a release on a green `swift test`, a version-source match against the tag, and Developer ID signing + notarization. - The **live Python eval** (`run_eval.py`) is periodic / manual, not per-PR CI — it makes