Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

# Least privilege (#48 verify): neither job uses the token — pip install + pytest
# execute PR-influenced code, so don't hand them a write-scoped GITHUB_TOKEN.
permissions:
contents: read

jobs:
test:
name: swift test
Expand All @@ -33,3 +38,35 @@ jobs:

- name: Test
run: swift test

# Python unit suite (#48): no network, no `claude -p` — safe on the PR critical path.
# This is what makes the front-page vs-context7 guards fire MECHANICALLY at the PR
# gate: the README-block drift guard (test_emit_readme_block_matches_committed_readme,
# + the `--emit-readme --check` CLI path) and the chart's SEMANTIC guard
# (test_committed_svg_is_semantically_faithful — it re-renders in THIS env and
# compares <text> content, not bytes/coordinates, so it is COORDINATE-independent;
# a future matplotlib changing text *serialization* would false-FAIL loudly, never
# silently pass — regenerate the asset to fix; requirements.txt caps matplotlib <4).
# Deliberately NOT run here: `--emit-chart --check` (byte-compare — false-drifts
# across matplotlib/freetype versions; #50 verify) and the LIVE eval
# (run_eval.py — real `claude -p` calls; stays periodic/manual per project policy).
# Guard scope (#48 verify): these guards lock rendered artifacts ⟷ snapshot.
# Snapshot-SOURCE tampering is caught by freshness.yml's pull_request paths trigger
# (live registry check on any PR touching the sample), and the wiki Home numbers by
# test_home_pages_match_sample — both landed via #49 (same batch).
python-tests:
name: python unit tests
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install eval-harness dependencies
run: pip install -r evals/look-up/requirements.txt

- name: Run unit tests (incl. vs-context7 drift guards)
run: python3 -m pytest evals/look-up/tests/ -q
6 changes: 4 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ make release-signed VERSION=vX.Y.Z # build + sign + notarize + mcpb (n
bash scripts/sync-wiki.sh # mirror docs/wiki/ -> GitHub wiki (after merge)
```

CI runs `swift build` + `swift test` per push/PR. `scripts/release.sh` gates on green tests +
version-source/tag match. Version source of truth: `Sources/LiveDocsCore/Version.swift`.
CI runs `swift build` + `swift test` AND the Python unit suite per push/PR (the Python job
is what fires the vs-context7 drift guards mechanically at the PR gate, #48 — it makes no
network / API calls). `scripts/release.sh` gates on green tests + version-source/tag match.
Version source of truth: `Sources/LiveDocsCore/Version.swift`.
Live evals (`run_eval.py --runs N`) are periodic/manual — real `claude -p` calls; run from a
clean non-nested directory with credits.

Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,9 @@ swift test # unit tests (pure logic + engine via injected HTTP fakes)
swift build # builds the MCP executable
```

CI runs `swift build` + `swift test` on every push/PR. `scripts/release.sh` gates the release on a
green `swift test`, a version-source match against the tag, and Developer ID signing + notarization.
CI runs `swift build` + `swift test` and the Python unit suite on every push/PR (#48).
`scripts/release.sh` gates the release on a green `swift test`, a version-source match
against the tag, and Developer ID signing + notarization.

### Skill eval (`evals/look-up/`)

Expand Down
10 changes: 7 additions & 3 deletions docs/wiki/Testing-zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,15 @@ python3 -m pytest evals/look-up/tests/ # 66 個 Python eval test

## CI 與 release gate

- **CI** 每個 push / PR 跑 `swift build` + `swift test`。
- **CI** 每個 push / PR 跑 `swift build` + `swift test` **加上 Python unit suite**(#48)。
Python job 不碰網路 / API;它讓 vs-context7 漂移守衛(README 區塊 + 圖的**座標無關**
語意守衛 —— 比對 `<text>` 內容,若未來 matplotlib 改變文字序列化只會**大聲 false-FAIL**、
絕不靜默通過)在 PR gate **機械性**觸發。byte-level `--emit-chart --check` 留在
maintainer 本機(跨 matplotlib 版本 freetype-fragile)。
- **`scripts/release.sh`** 把 release gate 在:`swift test` 綠、版本 source 與 tag 相符、
Developer ID 簽章 + notarize。
- **Python eval** 是 periodic / manual,不進 per-PR CI —— 它會真的呼叫 `claude -p`(有成本 +
stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。
- **live Python eval**(`run_eval.py`)是 periodic / manual,不進 per-PR CI —— 它會真的呼叫
`claude -p`(有成本 + stochastic),所以當 maintainer baseline 跑,不放關鍵路徑上。

## 紀律

Expand Down
13 changes: 10 additions & 3 deletions docs/wiki/Testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,18 @@ vs-context7 comparison harness. See

## CI and the release gate

- **CI** runs `swift build` + `swift test` on every push and pull request.
- **CI** runs `swift build` + `swift test` **and the Python unit suite** on every push and
pull request (#48). The Python job makes no network / API calls; it is what fires the
vs-context7 drift guards (README block + the chart's coordinate-independent semantic
guard — it compares `<text>` content, so a matplotlib serialization change would
false-FAIL loudly, never silently pass) mechanically at the PR gate. The byte-level
`--emit-chart --check` stays maintainer-local (freetype-fragile across matplotlib
versions).
- **`scripts/release.sh`** gates a release on a green `swift test`, a version-source match
against the tag, and Developer ID signing + notarization.
- The **Python eval** is periodic / manual, not per-PR CI — it makes real `claude -p` calls
(cost + stochastic), so it runs as a maintainer baseline rather than on the critical path.
- The **live Python eval** (`run_eval.py`) is periodic / manual, not per-PR CI — it makes
real `claude -p` calls (cost + stochastic), so it runs as a maintainer baseline rather
than on the critical path.

## Discipline

Expand Down
Loading