feat(eval): model-judgment eval harness — fixture defects + grader + ensemble-eval skill

.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
       "name": "parallel-ai-agents",
       "source": "./plugins/parallel-ai-agents",
       "description": "平行派發任務給多個 AI agent（Claude + Codex），獨立執行後交叉比對結果。Codex 改走直接 HTTP wrapper（bin/codex-call，Swift script）取代 codex exec subprocess，解決 hang 問題且避開 Python 版本飄移",
-      "version": "2.16.0",
+      "version": "2.17.0",
       "author": {
         "name": "Che Cheng"
       },

plugins/parallel-ai-agents/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "parallel-ai-agents",
   "description": "平行派發任務給多個 AI agent（Claude + Codex），獨立執行後交叉比對結果。Codex 改走直接 HTTP wrapper（bin/codex-call，Swift script）取代 codex exec subprocess，解決 hang 問題且避開 Python 版本飄移",
-  "version": "2.16.0",
+  "version": "2.17.0",
   "author": {
     "name": "Che Cheng"
   }

plugins/parallel-ai-agents/CHANGELOG.md

@@ -11,6 +11,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [2.17.0] - 2026-06-10
+
+### Added
+- **Eval harness —— 測試金字塔最後一塊：模型判斷品質**（reviewer 偵測率 + `apply_fixes` 修稿品質）。確定性 surface 已全由 `test/` 覆蓋，唯一沒測的「模型抓不抓得到真缺陷」不適合單元測試 → 走 eval：
+  - **`eval/fixtures/stats-paper/`**：合成統計論文，**故意埋 4 個缺陷**（捏造文獻 Tanaka & Whitfield 2019、錯平均 5.42 vs 4.42、錯 t 值 6.34 vs 2.79、Abstract/Method 樣本數 120 vs 102 不一致）+ ground-truth `analysis/results.csv` + `manifest.json`（match patterns / fix checks）。
+  - **`bin/pai-eval-grade`**：eval 的**確定性評分器**（唯一可單元測的部分，故有單元測）。`detect` 模式 = K 次 run 容差聚合（每缺陷 hits ≥ minHits，預設過半；integrity findings 排除於命中、單獨列報）；`fix` 模式 = 修稿驗證（planted 文字消失 + corrected 值出現）。`test/pai-eval-grade.test.mjs`（11 個）。
+  - **`skills/ensemble-eval/SKILL.md`**（dev 工具）：K 次真 ensemble → 存 findings → grade。鐵律：fixture 唯讀（apply-fix 只動 temp 複本）、**reviewer context 必須中性**（manifest／「eval」字眼絕不進 prompt，否則量到 prompted recall 不是 natural recall）、容差斷言、**絕不進 CI**。
+  - End-to-end smoke 驗證：實跑一次真 ensemble（K=1、codex 關）對 fixture，`pai-eval-grade detect --min-hits 1` 驗證缺陷可被抓到。
+
 ## [2.16.0] - 2026-06-10
 
 ### Added

plugins/parallel-ai-agents/CLAUDE.md

@@ -12,8 +12,9 @@
 | `/parallel-ai-agents:ensemble-academic-review` | 審學術論文（methodology/writing + 文獻真偽 + 數字重算；支援多輪 mix/hybrid/auto-iterate）|
 | `/parallel-ai-agents:ensemble-lecture-review` | 審教學講義（內容正確性/可讀性/逐字稿覆蓋率）|
 | `/parallel-ai-agents:ensemble-compose` | 自由組合：跨 profile 挑 lens + 自訂 reviewer（`--include`/`--lens`/`--lens-file` CSV）|
+| `/parallel-ai-agents:ensemble-eval` | **dev 工具**：對 `eval/fixtures/` 埋好缺陷的論文跑 K 次真 ensemble，容差斷言偵測率（+`--apply-fix` 驗修稿）。手動跑、不進 CI |
 
-四個 skill 共用同一個 harness `workflows/ensemble-workflow.js`（`PROFILES` 內建各角色 lens；regression 測試見 `test/`）。
+審閱 skill 共用同一個 harness `workflows/ensemble-workflow.js`（`PROFILES` 內建各角色 lens；確定性 surface 的 regression 測試見 `test/`，模型判斷品質見 `eval/`）。
 
 ## 審閱架構（雙 backend，findings 形狀一致）
 

plugins/parallel-ai-agents/bin/pai-eval-grade

@@ -0,0 +1,142 @@
+#!/usr/bin/env node
+// pai-eval-grade — ensemble eval 的「評分器」（單一真相源、確定性、可單元測）。
+//
+// eval harness 的分工：跑真 ensemble（LLM、貴、手動）由 ensemble-eval skill 負責；
+// 「抓到了沒」的判定是純函式 —— findings JSON × manifest → 命中報告。本 script 只做後者。
+//
+// 兩種模式：
+//   detect — K 次 ensemble run 的 findings，逐 defect 算命中數，容差斷言（hits ≥ minHits）。
+//     pai-eval-grade detect --manifest <manifest.json> [--min-hits N] <findings.json>...
+//     （findings.json = Workflow 回傳的 {findings:[...]} 或裸 array；minHits 預設過半數 = floor(K/2)+1）
+//   fix — apply_fixes 後的修正稿，逐 defect 驗證 planted 文字已消失（可選 corrected 出現）。
+//     pai-eval-grade fix --manifest <manifest.json> <fixed-doc>
+//
+// 命中語意（detect，皆對「同一個 finding」的 title+body、不分大小寫）：
+//   match_any — 任一 pattern 以子字串出現；match_all — 全部 pattern 出現在同一 finding。
+//   兩者皆給 → 兩者皆須成立。expectLens 僅報表參考，不影響判定。
+//   integrity findings（lens did not complete）不算命中，但每 run 的 integrity 數會列出 —— 大量
+//   integrity 表示該 run 的 ensemble 本身壞掉，eval 結果不可信。
+//
+// 退出碼：0 全部 defect 通過；1 有 defect 未達標（或 fix 未修乾淨）；2 用法/IO 錯誤。
+
+const fs = require('node:fs')
+
+function die(msg, code) {
+  process.stderr.write(msg + '\n')
+  process.exit(code)
+}
+
+function readJSON(path) {
+  let raw
+  try {
+    raw = fs.readFileSync(path, 'utf8')
+  } catch (e) {
+    die(`讀檔失敗: ${path}: ${e.message}`, 2)
+  }
+  try {
+    return JSON.parse(raw)
+  } catch {
+    die(`非法 JSON: ${path}`, 2)
+  }
+}
+
+// Workflow 回傳 {findings:[...]} 或裸 array 皆可
+function extractFindings(obj, path) {
+  const arr = Array.isArray(obj) ? obj : obj && Array.isArray(obj.findings) ? obj.findings : null
+  if (!arr) die(`找不到 findings array: ${path}`, 2)
+  return arr
+}
+
+const isIntegrity = (f) => /did not complete|cross-model pass incomplete/i.test(String(f.title || ''))
+
+// defect 是否被「某一個 finding」命中
+function findingHits(finding, defect) {
+  const text = `${finding.title || ''}\n${finding.body || ''}`.toLowerCase()
+  const any = Array.isArray(defect.match_any) ? defect.match_any : null
+  const all = Array.isArray(defect.match_all) ? defect.match_all : null
+  if (any && !any.some((p) => text.includes(String(p).toLowerCase()))) return false
+  if (all && !all.every((p) => text.includes(String(p).toLowerCase()))) return false
+  return Boolean(any || all) // manifest 兩者皆缺 = 無法判定 → 不命中
+}
+
+function gradeDetect(manifest, runFiles, minHitsArg) {
+  const defects = manifest.defects || []
+  if (!defects.length) die('manifest 無 defects', 2)
+  if (!runFiles.length) die('detect 需至少一個 findings.json', 2)
+
+  const runs = runFiles.map((path) => {
+    const findings = extractFindings(readJSON(path), path).filter((f) => !isIntegrity(f))
+    const integrity = extractFindings(readJSON(path), path).filter(isIntegrity).length
+    return { path, findings, integrity }
+  })
+  const K = runs.length
+  const minHits = minHitsArg != null ? minHitsArg : Math.floor(K / 2) + 1
+  if (!(minHits >= 1 && minHits <= K)) die(`--min-hits 需在 1..${K}: ${minHits}`, 2)
+
+  const report = defects.map((d) => {
+    const runsHit = []
+    const lensSet = new Set()
+    runs.forEach((r, i) => {
+      const hits = r.findings.filter((f) => findingHits(f, d))
+      if (hits.length) {
+        runsHit.push(i + 1)
+        hits.forEach((f) => f.lens && lensSet.add(f.lens))
+      }
+    })
+    return { id: d.id, desc: d.desc, expectLens: d.expectLens || [], hits: runsHit.length, runsHit, caughtBy: [...lensSet], pass: runsHit.length >= minHits }
+  })
+  const pass = report.every((d) => d.pass)
+  return {
+    mode: 'detect',
+    runs: runs.map((r, i) => ({ run: i + 1, file: r.path, findings: r.findings.length, integrity: r.integrity })),
+    K,
+    minHits,
+    defects: report,
+    pass,
+  }
+}
+
+function gradeFix(manifest, docPath) {
+  const checks = manifest.fix || []
+  if (!checks.length) die('manifest 無 fix checks', 2)
+  let doc
+  try {
+    doc = fs.readFileSync(docPath, 'utf8')
+  } catch (e) {
+    die(`讀修正稿失敗: ${docPath}: ${e.message}`, 2)
+  }
+  const report = checks.map((c) => {
+    const plantedGone = c.plantedAbsent ? !new RegExp(c.plantedAbsent, 'i').test(doc) : true
+    const correctedIn = c.correctedPresent ? doc.toLowerCase().includes(String(c.correctedPresent).toLowerCase()) : true
+    return { id: c.id, plantedGone, correctedPresent: correctedIn, pass: plantedGone && correctedIn }
+  })
+  return { mode: 'fix', doc: docPath, checks: report, pass: report.every((c) => c.pass) }
+}
+
+function main(argv) {
+  const mode = argv[0]
+  if (mode !== 'detect' && mode !== 'fix') {
+    die('用法：pai-eval-grade detect --manifest m.json [--min-hits N] findings.json...\n' + '      pai-eval-grade fix    --manifest m.json fixed-doc', 2)
+  }
+  let manifestPath = null
+  let minHits = null
+  const positional = []
+  for (let i = 1; i < argv.length; i++) {
+    if (argv[i] === '--manifest') manifestPath = argv[++i]
+    else if (argv[i] === '--min-hits') {
+      minHits = Number(argv[++i])
+      if (!Number.isInteger(minHits)) die(`--min-hits 需整數: ${argv[i]}`, 2)
+    } else positional.push(argv[i])
+  }
+  if (!manifestPath) die('缺 --manifest', 2)
+  const manifest = readJSON(manifestPath)
+
+  const result = mode === 'detect' ? gradeDetect(manifest, positional, minHits) : gradeFix(manifest, positional[0] || die('fix 需一個修正稿路徑', 2))
+  process.stdout.write(JSON.stringify(result, null, 2) + '\n')
+  process.exit(result.pass ? 0 : 1)
+}
+
+if (require.main === module) {
+  main(process.argv.slice(2))
+}
+module.exports = { findingHits, gradeDetect, gradeFix }

plugins/parallel-ai-agents/eval/fixtures/stats-paper/analysis/results.csv

@@ -0,0 +1,11 @@
+measure,value
+n_total,102
+n_gain,52
+n_loss,50
+mean_gain,4.42
+sd_gain,1.12
+mean_loss,3.95
+sd_loss,1.21
+t_df100,2.79
+p_value,0.03
+cohens_d,0.55

plugins/parallel-ai-agents/eval/fixtures/stats-paper/manifest.json

@@ -0,0 +1,38 @@
+{
+  "fixture": "stats-paper",
+  "paper": "paper.md",
+  "groundTruth": ["analysis/results.csv"],
+  "notes": "合成論文，4 個故意埋的缺陷。match_any = 任一 pattern 出現在同一 finding（title+body，不分大小寫）即命中；match_all = 全部 pattern 須出現在同一 finding。expectLens 僅供報表參考，不影響判定（任何 lens 抓到都算）。",
+  "defects": [
+    {
+      "id": "hallucinated-ref",
+      "desc": "Tanaka & Whitfield (2019) 為捏造文獻（含捏造期刊 Journal of Behavioral Decision Science）",
+      "expectLens": ["reference-verifier"],
+      "match_any": ["tanaka", "whitfield"]
+    },
+    {
+      "id": "wrong-mean",
+      "desc": "gain 組平均：paper 寫 M=5.42，ground truth mean_gain=4.42",
+      "expectLens": ["number-verifier"],
+      "match_any": ["5.42", "4.42"]
+    },
+    {
+      "id": "wrong-t",
+      "desc": "t 統計量：paper 寫 t(100)=6.34 / p<.001，ground truth t=2.79 / p=0.03",
+      "expectLens": ["number-verifier"],
+      "match_any": ["6.34", "2.79"]
+    },
+    {
+      "id": "n-inconsistency",
+      "desc": "Abstract 寫 120 participants，Method 寫 102（ground truth n_total=102）",
+      "expectLens": ["methodology", "number-verifier", "writing"],
+      "match_all": ["120", "102"]
+    }
+  ],
+  "fix": [
+    { "id": "hallucinated-ref", "plantedAbsent": "Tanaka" },
+    { "id": "wrong-mean", "plantedAbsent": "5\\.42", "correctedPresent": "4.42" },
+    { "id": "wrong-t", "plantedAbsent": "6\\.34", "correctedPresent": "2.79" },
+    { "id": "n-inconsistency", "plantedAbsent": "120 participants", "correctedPresent": "102" }
+  ]
+}

plugins/parallel-ai-agents/eval/fixtures/stats-paper/paper.md

@@ -0,0 +1,41 @@
+# The Effect of Framing on Risk Preference: A Conceptual Replication
+
+## Abstract
+
+Framing effects are among the most robust findings in judgment and decision-making research, yet conceptual replications with modern samples remain scarce. We recruited 120 participants and randomly assigned them to a gain-framed or a loss-framed version of a fixed-stakes investment scenario. Participants in the gain-framed condition reported significantly higher risk aversion (M = 5.42, SD = 1.12) than those in the loss-framed condition (M = 3.95, SD = 1.21), t(100) = 6.34, p < .001. These results support the classical prospect-theoretic account and suggest that framing effects generalize to incentivized online samples.
+
+## Introduction
+
+Since the seminal demonstrations of preference reversals under reframing (Tversky & Kahneman, 1981), framing has served as a canonical violation of description invariance. Prospect theory (Kahneman & Tversky, 1979) explains such reversals through reference-dependent valuation and loss aversion. More recently, an attentional refocusing account has been proposed, arguing that frames redirect attention toward frame-congruent outcomes (Tanaka & Whitfield, 2019).
+
+The present study provides a conceptual replication with three goals: (1) estimate the framing effect in an incentivized online sample; (2) test whether the effect survives a fixed-stakes design that removes magnitude confounds; and (3) compare the prospect-theoretic and attentional accounts.
+
+## Method
+
+### Participants
+
+102 undergraduate students (61 women, 41 men; M_age = 20.3 years) participated in exchange for course credit. All participants provided informed consent, and the protocol was approved by the local IRB.
+
+### Design and Procedure
+
+Participants were randomly assigned to one of two framing conditions (gain vs. loss) in a between-subjects design. Both conditions described an identical investment scenario with a fixed expected value; only the outcome description differed. The primary dependent measure was a 7-point risk-aversion index (higher = more risk averse) computed from three choice items.
+
+### Analysis
+
+Condition means were compared with an independent-samples t test. All analyses were conducted in R; summary statistics are archived in `analysis/results.csv`.
+
+## Results
+
+As predicted, participants in the gain-framed condition were more risk averse (M = 5.42, SD = 1.12) than participants in the loss-framed condition (M = 3.95, SD = 1.21), t(100) = 6.34, p < .001, d = 0.55. The effect held when excluding participants who failed the comprehension check (n = 7).
+
+These findings are consistent with reference-dependent valuation, and the attention-shift pattern reported by Tanaka and Whitfield (2019) offers a complementary process-level explanation.
+
+## Discussion
+
+The framing effect replicated in a fixed-stakes, incentivized design, which speaks against magnitude-based artifacts. Limitations include the single-scenario design and a homogeneous student sample. Future work should manipulate attentional allocation directly to discriminate the prospect-theoretic and attentional refocusing accounts.
+
+## References
+
+- Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. *Econometrica, 47*(2), 263–291.
+- Tanaka, R., & Whitfield, J. (2019). Attentional refocusing under framed risk: A registered replication. *Journal of Behavioral Decision Science, 12*(4), 401–419.
+- Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. *Science, 211*(4481), 453–458.