diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6086e8b..9ca0c61 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,6 +9,11 @@ concurrency: group: ci-${{ github.ref }} cancel-in-progress: true +# Least privilege (#48 verify): neither job uses the token — pip install + pytest +# execute PR-influenced code, so don't hand them a write-scoped GITHUB_TOKEN. +permissions: + contents: read + jobs: test: name: swift test @@ -33,3 +38,35 @@ jobs: - name: Test run: swift test + + # Python unit suite (#48): no network, no `claude -p` — safe on the PR critical path. + # This is what makes the front-page vs-context7 guards fire MECHANICALLY at the PR + # gate: the README-block drift guard (test_emit_readme_block_matches_committed_readme, + # + the `--emit-readme --check` CLI path) and the chart's SEMANTIC guard + # (test_committed_svg_is_semantically_faithful — it re-renders in THIS env and + # compares content, not bytes/coordinates, so it is COORDINATE-independent; + # a future matplotlib changing text *serialization* would false-FAIL loudly, never + # silently pass — regenerate the asset to fix; requirements.txt caps matplotlib <4). + # Deliberately NOT run here: `--emit-chart --check` (byte-compare — false-drifts + # across matplotlib/freetype versions; #50 verify) and the LIVE eval + # (run_eval.py — real `claude -p` calls; stays periodic/manual per project policy). + # Guard scope (#48 verify): these guards lock rendered artifacts ⟷ snapshot. + # Snapshot-SOURCE tampering is caught by freshness.yml's pull_request paths trigger + # (live registry check on any PR touching the sample), and the wiki Home numbers by + # test_home_pages_match_sample — both landed via #49 (same batch). + python-tests: + name: python unit tests + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install eval-harness dependencies + run: pip install -r evals/look-up/requirements.txt + + - name: Run unit tests (incl. vs-context7 drift guards) + run: python3 -m pytest evals/look-up/tests/ -q diff --git a/CLAUDE.md b/CLAUDE.md index 7b7858c..bf56cd5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,8 +25,10 @@ make release-signed VERSION=vX.Y.Z # build + sign + notarize + mcpb (n bash scripts/sync-wiki.sh # mirror docs/wiki/ -> GitHub wiki (after merge) ``` -CI runs `swift build` + `swift test` per push/PR. `scripts/release.sh` gates on green tests + -version-source/tag match. Version source of truth: `Sources/LiveDocsCore/Version.swift`. +CI runs `swift build` + `swift test` AND the Python unit suite per push/PR (the Python job +is what fires the vs-context7 drift guards mechanically at the PR gate, #48 — it makes no +network / API calls). `scripts/release.sh` gates on green tests + version-source/tag match. +Version source of truth: `Sources/LiveDocsCore/Version.swift`. Live evals (`run_eval.py --runs N`) are periodic/manual — real `claude -p` calls; run from a clean non-nested directory with credits. diff --git a/README.md b/README.md index 19ce39c..d4b3107 100644 --- a/README.md +++ b/README.md @@ -92,8 +92,9 @@ swift test # unit tests (pure logic + engine via injected HTTP fakes) swift build # builds the MCP executable ``` -CI runs `swift build` + `swift test` on every push/PR. `scripts/release.sh` gates the release on a -green `swift test`, a version-source match against the tag, and Developer ID signing + notarization. +CI runs `swift build` + `swift test` and the Python unit suite on every push/PR (#48). +`scripts/release.sh` gates the release on a green `swift test`, a version-source match +against the tag, and Developer ID signing + notarization. ### Skill eval (`evals/look-up/`) diff --git a/docs/wiki/Testing-zh-TW.md b/docs/wiki/Testing-zh-TW.md index f30dd16..08b4225 100644 --- a/docs/wiki/Testing-zh-TW.md +++ b/docs/wiki/Testing-zh-TW.md @@ -60,11 +60,15 @@ python3 -m pytest evals/look-up/tests/ # 66 個 Python eval test ## CI 與 release gate -- **CI** 每個 push / PR 跑 `swift build` + `swift test`。 +- **CI** 每個 push / PR 跑 `swift build` + `swift test` **加上 Python unit suite**(#48)。 + Python job 不碰網路 / API;它讓 vs-context7 漂移守衛(README 區塊 + 圖的**座標無關** + 語意守衛 —— 比對 `` 內容,若未來 matplotlib 改變文字序列化只會**大聲 false-FAIL**、 + 絕不靜默通過)在 PR gate **機械性**觸發。byte-level `--emit-chart --check` 留在 + maintainer 本機(跨 matplotlib 版本 freetype-fragile)。 - **`scripts/release.sh`** 把 release gate 在:`swift test` 綠、版本 source 與 tag 相符、 Developer ID 簽章 + notarize。 -- **Python eval** 是 periodic / manual,不進 per-PR CI —— 它會真的呼叫 `claude -p`(有成本 + - stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。 +- **live Python eval**(`run_eval.py`)是 periodic / manual,不進 per-PR CI —— 它會真的呼叫 + `claude -p`(有成本 + stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。 ## 紀律 diff --git a/docs/wiki/Testing.md b/docs/wiki/Testing.md index 7ab8521..294abf3 100644 --- a/docs/wiki/Testing.md +++ b/docs/wiki/Testing.md @@ -63,11 +63,18 @@ vs-context7 comparison harness. See ## CI and the release gate -- **CI** runs `swift build` + `swift test` on every push and pull request. +- **CI** runs `swift build` + `swift test` **and the Python unit suite** on every push and + pull request (#48). The Python job makes no network / API calls; it is what fires the + vs-context7 drift guards (README block + the chart's coordinate-independent semantic + guard — it compares `` content, so a matplotlib serialization change would + false-FAIL loudly, never silently pass) mechanically at the PR gate. The byte-level + `--emit-chart --check` stays maintainer-local (freetype-fragile across matplotlib + versions). - **`scripts/release.sh`** gates a release on a green `swift test`, a version-source match against the tag, and Developer ID signing + notarization. -- The **Python eval** is periodic / manual, not per-PR CI — it makes real `claude -p` calls - (cost + stochastic), so it runs as a maintainer baseline rather than on the critical path. +- The **live Python eval** (`run_eval.py`) is periodic / manual, not per-PR CI — it makes + real `claude -p` calls (cost + stochastic), so it runs as a maintainer baseline rather + than on the critical path. ## Discipline