diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 974664a..7f7b3ff 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "parallel-ai-agents", "source": "./plugins/parallel-ai-agents", "description": "平行派發任務給多個 AI agent（Claude + Codex），獨立執行後交叉比對結果。Codex 改走直接 HTTP wrapper（bin/codex-call，Swift script）取代 codex exec subprocess，解決 hang 問題且避開 Python 版本飄移", - "version": "2.14.0", + "version": "2.14.1", "author": { "name": "Che Cheng" }, diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index b6e6644..862d1eb 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -27,3 +27,6 @@ jobs: - name: bats run: bats test/ + + - name: harness regression (node) + run: node test/ensemble-workflow.test.mjs diff --git a/plugins/parallel-ai-agents/.claude-plugin/plugin.json b/plugins/parallel-ai-agents/.claude-plugin/plugin.json index 9976cea..3dec60b 100644 --- a/plugins/parallel-ai-agents/.claude-plugin/plugin.json +++ b/plugins/parallel-ai-agents/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "parallel-ai-agents", "description": "平行派發任務給多個 AI agent（Claude + Codex），獨立執行後交叉比對結果。Codex 改走直接 HTTP wrapper（bin/codex-call，Swift script）取代 codex exec subprocess，解決 hang 問題且避開 Python 版本飄移", - "version": "2.14.0", + "version": "2.14.1", "author": { "name": "Che Cheng" } diff --git a/plugins/parallel-ai-agents/CHANGELOG.md b/plugins/parallel-ai-agents/CHANGELOG.md index 3339cfd..f8cd12b 100644 --- a/plugins/parallel-ai-agents/CHANGELOG.md +++ b/plugins/parallel-ai-agents/CHANGELOG.md @@ -11,6 +11,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [2.14.1] - 2026-06-03 + +### Fixed +- **共用 harness `ensemble-workflow.js` 的 null-skip fail-OPEN**（dogfood lecture/academic/compose/harness 時抓到）：Workflow runtime 在「使用者中途 skip 某 agent」時讓 `agent()` 回 `null`。但 review / codex / devil's-advocate 三個 `.then` 只用 `(r && r.findings) || []` 處理 null 的 findings、**沒處理 `ok` flag** → 被 skip 的 reviewer 被當 `ok:true`（乾淨通過），**繞過 fail-closed integrity 檢查 → 可能假 PASS**（與 code-review 三輪一直在防的「假綠燈」同類，只是換成 JS null 路徑）。修：三處 `.then` 把 `r == null` 視為 `ok:false` → core lens／DA 被 skip 會如預期觸發 HIGH integrity finding。 + +### Added +- **`test/ensemble-workflow.test.mjs` —— 共用 harness 的 node regression 測試**（8 個）：unknown profile、空 lens 組合、core lens 被 skip(null)/error(throw)/DA 缺席 → HIGH integrity（鎖死上述 fail-open 修正）、codex 缺席 → INFO 非阻塞、mergeDedup 對 malformed severity 穩健。把 workflow script body 包成可 import 的 async 函式、注入 mock globals 實跑。已接進 `test/run.sh` 與 CI。 + +### Docs +- **plugin `CLAUDE.md` 修正 drift**：原本只列已不存在的單一 `/parallel-ai-agents:ensemble-review` + 「4 teammates + 1 Codex」舊架構；改成實際的 4 個 skill（code/academic/lecture/compose）+ 雙 backend（Workflow harness 預設、legacy fallback）+ fail-closed 說明。 + ## [2.14.0] - 2026-06-03 ### Added diff --git a/plugins/parallel-ai-agents/CLAUDE.md b/plugins/parallel-ai-agents/CLAUDE.md index adfdd6d..6f4f52a 100644 --- a/plugins/parallel-ai-agents/CLAUDE.md +++ b/plugins/parallel-ai-agents/CLAUDE.md @@ -6,22 +6,31 @@ ## Skills -| Skill | 用途 | 架構 | -|-------|------|------| -| `/parallel-ai-agents:ensemble-review` | 審閱文件/程式碼，交叉比對產出共識/盲點報告 | 4 Claude teammates (team) + 1 Codex | +| Skill | 用途 | +|-------|------| +| `/parallel-ai-agents:ensemble-code-review` | 審程式碼/技術文件/diff（最完整：`--diff`/`--base`/`--since`/`--commits`/`--pr` 等 diff 模式，建構走 `bin/pai-build-diff`）| +| `/parallel-ai-agents:ensemble-academic-review` | 審學術論文（methodology/writing + 文獻真偽 + 數字重算；支援多輪 mix/hybrid/auto-iterate）| +| `/parallel-ai-agents:ensemble-lecture-review` | 審教學講義（內容正確性/可讀性/逐字稿覆蓋率）| +| `/parallel-ai-agents:ensemble-compose` | 自由組合：跨 profile 挑 lens + 自訂 reviewer（`--include`/`--lens`/`--lens-file` CSV）| -## 審閱架構 +四個 skill 共用同一個 harness `workflows/ensemble-workflow.js`（`PROFILES` 內建各角色 lens；regression 測試見 `test/`）。 + +## 審閱架構（雙 backend，findings 形狀一致） + +預設 **Backend A — Workflow harness**（`workflows/ensemble-workflow.js`）：distinct-lens reviewers（× `replicas`）+ devil's-advocate（讀同儕**完稿**反駁）+ 可選 Codex 跨模型盲驗 → pure-JS mergeDedup + **fail-closed**（core lens／DA 缺席 = HIGH integrity，不可假 PASS）。reviewer 數是資料（lens 陣列 × replicas），可大量 fan-out（硬上限 `MAX_AGENTS`）。 ``` -ensemble-review -├── Claude Team（4 teammates，orchestrated） +ensemble（以 code profile 為例） +├── reviewers（× replicas，平行 barrier） │ ├── architecture — 設計、API、依賴 │ ├── correctness — 邏輯、bug、edge case -│ ├── security — 攻擊者視角 -│ └── devils-advocate — 反駁前 3 人 -└── Codex（gpt-5.5，跨模型盲驗） +│ └── security — 攻擊者視角 +├── devil's-advocate — 讀上面完稿、反駁「通過/LOW」（downstream node，非 live SendMessage） +└── Codex（gpt-5.5，跨模型盲驗，--codex 開） ``` +`Workflow` tool 不存在的舊版 Claude Code → fallback **Backend B — legacy TeamCreate fan-out**（同一 message 啟動 N 個 general-purpose Agent + Codex Bash）。兩 backend 報表一致。 + ## 依賴 - Claude Code orchestrated teams（TeamCreate、SendMessage） diff --git a/plugins/parallel-ai-agents/test/README.md b/plugins/parallel-ai-agents/test/README.md index e1c9194..8890349 100644 --- a/plugins/parallel-ai-agents/test/README.md +++ b/plugins/parallel-ai-agents/test/README.md @@ -7,8 +7,11 @@ | 檔案 | 對象 | |------|------| | `pai-build-diff.bats` | `../bin/pai-build-diff`（diff 模式建構器，3 輪硬化的單一真相源）| +| `ensemble-workflow.test.mjs` | `../workflows/ensemble-workflow.js`（共用 harness，4 個 skill 的底層）| -涵蓋：5 種模式（`--diff`/`--base`/`--since`/`--commits`/`--pr`）、退出碼契約（0 有 diff／3 無變更／1 錯誤）、ref/N 驗證（injection、dashed-ref、0/leading-zero、位數溢位）、untracked 安全（symlink no-follow、FIFO no-hang、換行檔名 C-quote）、empty-tree base、未知 mode 的多位元組 regression。 +`pai-build-diff.bats` 涵蓋：5 種模式（`--diff`/`--base`/`--since`/`--commits`/`--pr`）、退出碼契約（0 有 diff／3 無變更／1 錯誤）、ref/N 驗證（injection、dashed-ref、0/leading-zero、位數溢位）、untracked 安全（symlink no-follow、FIFO no-hang、換行檔名 C-quote）、empty-tree base、未知 mode 的多位元組 regression。 + +`ensemble-workflow.test.mjs` 涵蓋 harness 的 **fail-closed 不變式**：unknown profile、空 lens 組合、core lens 被 skip（null）/ error（throw）/ devil's-advocate 缺席 → 一律 HIGH integrity（不可假 PASS）、codex 缺席 → INFO 非阻塞、mergeDedup 對 malformed severity 穩健。把「null-skip fail-open」的修正鎖死成 regression。（純 node，無框架；把 workflow script body 包成可 import 的 async 函式、注入 mock globals 實跑。） ## 跑法 diff --git a/plugins/parallel-ai-agents/test/ensemble-workflow.test.mjs b/plugins/parallel-ai-agents/test/ensemble-workflow.test.mjs new file mode 100644 index 0000000..8a17b96 --- /dev/null +++ b/plugins/parallel-ai-agents/test/ensemble-workflow.test.mjs @@ -0,0 +1,117 @@ +#!/usr/bin/env node +// ensemble-workflow.js 的 regression 測試（純 node，無框架）。 +// +// 跑法：node test/ensemble-workflow.test.mjs +// +// harness 是 Workflow script（top-level await/return），無法被標準 runner 直接 import。 +// 解法：讀原始碼 → 把 body 包成一個可 export 的 async 函式（注入 mock 的 +// agent/parallel/phase/log/args 當參數）→ 寫成 temp module 後 import → 實跑整個 +// orchestration 並斷言。鎖死 fail-closed 不變式（含 null-skip fail-open 的修正）。 +// 載入的是本 repo 自己的 source（被測碼本身），信任等級等同 import 任何模組。 + +import { readFileSync, writeFileSync, rmSync } from 'node:fs' +import { fileURLToPath, pathToFileURL } from 'node:url' +import { dirname, join } from 'node:path' +import { tmpdir } from 'node:os' +import assert from 'node:assert/strict' + +const __dirname = dirname(fileURLToPath(import.meta.url)) +const harnessSrc = readFileSync(join(__dirname, '..', 'workflows', 'ensemble-workflow.js'), 'utf8') + .replace(/^export const meta/m, 'const meta') // strip ESM export so the body is legal inside a function + +// Wrap the workflow-script body as an exportable async fn whose destructured params ARE the workflow +// globals the harness references (agent/parallel/pipeline/phase/log/args). Write → import (no eval). +const wrapped = `export async function runEnsemble({ agent, parallel, pipeline, phase, log, args }) {\n${harnessSrc}\n}\n` +const tmpFile = join(tmpdir(), `pai-ensemble-harness-${process.pid}.mjs`) +writeFileSync(tmpFile, wrapped) +let rawRun +try { + ;({ runEnsemble: rawRun } = await import(pathToFileURL(tmpFile).href)) +} finally { + rmSync(tmpFile, { force: true }) +} + +const parallel = async (thunks) => Promise.all(thunks.map((f) => f())) +const pipeline = async () => [] +const noop = () => {} + +// run one scenario: agentImpl(prompt, opts) decides per-label behavior (return obj / null=skip / throw). +function runEnsemble(args, agentImpl) { + return rawRun({ agent: agentImpl, parallel, pipeline, phase: noop, log: noop, args }) +} + +// agent impls keyed by label +const allPass = async () => ({ findings: [] }) +const skip = (label) => async (_p, o) => (o && o.label === label ? null : { findings: [] }) +const boom = (label) => async (_p, o) => { if (o && o.label === label) throw new Error('boom'); return { findings: [] } } + +const tests = [] +const test = (name, fn) => tests.push({ name, fn }) + +test('unknown profile → HIGH harness finding, agents=0', async () => { + const out = await runEnsemble({ profile: 'nope', file: '/x' }, allPass) + assert.equal(out.verdict, 'FINDINGS') + assert.equal(out.stats.agents, 0) + assert.match(out.findings[0].title, /unknown ensemble profile/) +}) + +test('custom profile, no lenses → HIGH "no active lenses"', async () => { + const out = await runEnsemble({ profile: 'custom', file: '/x' }, allPass) + assert.equal(out.verdict, 'FINDINGS') + assert.ok(out.findings.some((f) => /no active lenses/.test(f.title) && f.severity === 'HIGH')) +}) + +test('code profile all-ok → PASS, no integrity', async () => { + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: false }, allPass) + assert.equal(out.verdict, 'PASS') + assert.equal(out.stats.integrity, 0) +}) + +test('REGRESSION: skip a core lens (null) → fail-closed HIGH (not silent PASS)', async () => { + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: false }, skip('review:security')) + assert.equal(out.verdict, 'FINDINGS') + assert.ok(out.findings.some((f) => f.lens === 'security' && f.severity === 'HIGH' && /did not complete/.test(f.title))) +}) + +test('error (throw) a core lens → fail-closed HIGH', async () => { + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: false }, boom('review:correctness')) + assert.equal(out.verdict, 'FINDINGS') + assert.ok(out.findings.some((f) => f.lens === 'correctness' && f.severity === 'HIGH' && /did not complete/.test(f.title))) +}) + +test('skip devils-advocate → fail-closed HIGH', async () => { + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: false }, skip('devils-advocate')) + assert.equal(out.verdict, 'FINDINGS') + assert.ok(out.findings.some((f) => f.lens === 'devils-advocate' && f.severity === 'HIGH')) +}) + +test('skip codex (codexEnabled) → INFO process gap, verdict stays PASS', async () => { + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: true, codexCallPath: '/bin/codex-call' }, skip('codex')) + assert.equal(out.verdict, 'PASS') // codex gap is non-blocking + assert.ok(out.findings.some((f) => f.lens === 'codex' && f.severity === 'INFO')) +}) + +test('mergeDedup robust to malformed severity (no crash, finding kept, FINDINGS)', async () => { + const weird = async (_p, o) => + o && o.label === 'review:architecture' + ? { findings: [{ severity: 'BOGUS', title: 'odd', body: 'b', file: null }] } + : { findings: [] } + const out = await runEnsemble({ profile: 'code', file: '/x', codexEnabled: false }, weird) + assert.ok(out.findings.some((f) => f.title === 'odd')) // unknown severity ranked 0, not dropped/crashed + assert.equal(out.verdict, 'FINDINGS') // 'BOGUS' !== 'INFO' +}) + +let pass = 0 +let fail = 0 +for (const t of tests) { + try { + await t.fn() + console.log(`ok - ${t.name}`) + pass++ + } catch (e) { + console.log(`not ok - ${t.name}\n ${e.message}`) + fail++ + } +} +console.log(`\n${pass}/${tests.length} passed`) +process.exit(fail === 0 ? 0 : 1) diff --git a/plugins/parallel-ai-agents/test/run.sh b/plugins/parallel-ai-agents/test/run.sh index 6ee7c1a..1d586aa 100755 --- a/plugins/parallel-ai-agents/test/run.sh +++ b/plugins/parallel-ai-agents/test/run.sh @@ -10,4 +10,7 @@ shellcheck bin/pai-build-diff echo "── bats test/ ──" bats test/ +echo "── harness regression (node) ──" +node test/ensemble-workflow.test.mjs + echo "✓ 全部通過" diff --git a/plugins/parallel-ai-agents/workflows/ensemble-workflow.js b/plugins/parallel-ai-agents/workflows/ensemble-workflow.js index 33b9fa1..99642b0 100644 --- a/plugins/parallel-ai-agents/workflows/ensemble-workflow.js +++ b/plugins/parallel-ai-agents/workflows/ensemble-workflow.js @@ -437,9 +437,12 @@ if (replicas < requested) { // Phase 1 (barrier): every base lens × replicas + optional Codex run concurrently and // independently. A barrier is correct: the devil's-advocate (phase 2) needs every reviewer's -// findings to refute them. Each thunk CATCHES its own error and tags its lens (ok:false) so a -// failed lens is observable downstream — never silently dropped. Each finding's lens is forced to -// the assigned key so an agent cannot mislabel its attribution. +// findings to refute them. Each thunk CATCHES its own error AND treats a null agent() return +// (the runtime hands back null when the user SKIPS an agent mid-run) as ok:false — both are +// "this lens did not actually review", so both must surface as a fail-closed integrity finding +// downstream, never a silent pass. (A null slipping through as ok:true with empty findings was a +// fail-OPEN false-green-light, caught by this harness's own dogfood.) Each finding's lens is forced +// to the assigned key so an agent cannot mislabel its attribution. phase('review') const reviewThunks = [] for (const l of activeLenses) { @@ -450,7 +453,9 @@ for (const l of activeLenses) { label: replicas > 1 ? `review:${l.key}#${k + 1}` : `review:${l.key}`, phase: 'review', }) - .then((r) => ({ lens: l.key, findings: ((r && r.findings) || []).map((f) => ({ ...f, lens: l.key })), ok: true })) + .then((r) => (r == null + ? { lens: l.key, findings: [], ok: false } // user-skipped → fail-closed + : { lens: l.key, findings: (r.findings || []).map((f) => ({ ...f, lens: l.key })), ok: true })) .catch(() => ({ lens: l.key, findings: [], ok: false })) ) } @@ -458,7 +463,9 @@ for (const l of activeLenses) { const codexThunk = codexOn ? () => agent(codexPrompt(profile, A), { schema: FINDINGS_SCHEMA, label: 'codex', phase: 'review' }) - .then((r) => ({ lens: 'codex', findings: ((r && r.findings) || []).map((f) => ({ ...f, lens: 'codex' })), ok: true })) + .then((r) => (r == null + ? { lens: 'codex', findings: [], ok: false } // user-skipped → surfaced as process gap + : { lens: 'codex', findings: (r.findings || []).map((f) => ({ ...f, lens: 'codex' })), ok: true })) .catch(() => ({ lens: 'codex', findings: [], ok: false })) : null @@ -468,7 +475,9 @@ const reviewerResults = round1.filter((r) => r.lens !== 'codex') // Phase 2: devil's-advocate adversarially refutes the reviewers' judgments (also fail-aware). phase('adversarial') const da = await agent(daPrompt(profile, reviewerResults, A), { schema: FINDINGS_SCHEMA, label: 'devils-advocate', phase: 'adversarial' }) - .then((r) => ({ findings: ((r && r.findings) || []).map((f) => ({ ...f, lens: 'devils-advocate' })), ok: true })) + .then((r) => (r == null + ? { findings: [], ok: false } // user-skipped DA → fail-closed + : { findings: (r.findings || []).map((f) => ({ ...f, lens: 'devils-advocate' })), ok: true })) .catch(() => ({ findings: [], ok: false })) // Phase 3: merge + dedup (pure JS; no agent, no FS). FAIL-CLOSED — a core lens (every base lens