feat: promote skill eval gates to trusted

[ZhiyaoWen999]

Summary

• add real fixture-backed behavioral eval coverage for execution skills, plus a reviewer-gated kind: "dialogue" path for conversation-shaped skills
• reject provisional or missing-fixture execution evals, enforce case minimums and valid eval kinds as errors, and support a --min-rank1 deterministic routing floor
• materialize execution fixtures into throwaway git workspaces, including patch-backed dirty-worktree scenarios, while grading dialogue evals from their transcript
• document the trusted eval requirements and run deterministic eval tests plus the 80% rank-1 floor in CI

Fixes #352

Validation

• node --test scripts/run-evals-test.js — 12/12 passing
• node scripts/run-evals.js --min-rank1 80 — 124 checks passing, 86% rank-1
• node scripts/validate-skills.js — 24 skills, 0 errors/warnings
• node scripts/run-evals.js --behavioral interview-me --dry-run
• node scripts/run-evals.js --behavioral idea-refine --dry-run

[addyosmani]

This is exactly the graduation work I scoped in #352, so it's great to see someone pick it up, especially real fixture-backed evals for every skill plus the gate that rejects provisional ones. Since it's still draft I won't go line by line yet, but the shape is right: fixtures are what make Tier 3 mean something, and enforcing the case minimums is what closes the transition window. Two things I'll want to work through when you flip it out of draft: keeping the CI cost sane if behavioral evals run in the pipeline, and making sure the gate degrades gracefully for contributors who can't run the token-spending tier locally. Really nice contribution to the framework.

[ZhiyaoWen999]

Thanks! The PR is now ready for review.

On the two points you mentioned: CI only runs the deterministic runner tests and Tier 2 with the --min-rank1 85 floor. Tier 3 behavioral evals remain opt-in and never run in CI, so contributors who can’t run the token-spending tier locally can still satisfy all required checks.

Happy to adjust that boundary based on the line-by-line review.

[addyosmani]

This is the real graduation, nice work. Fixtures for all 25 skills, the runner getting its own unit tests (run-evals-test.js) that I didn't even ask for, and keeping the token-spending tier out of CI while gating the deterministic run on a --min-rank1 floor is exactly the right split. The patch-backed dirty-worktree fixtures for the git skills are a nice touch.

Two things before I merge. First, --min-rank1 85 against an actual 86 is a single point of headroom, tight enough that an unrelated description tweak could trip CI red; I'd either give it a bit more slack or make it explicit that bumping the floor is expected maintenance. Second, and bigger: flipping provisional-rejection and the minimums to hard errors raises the bar for every in-flight skill PR the moment this lands. #360 is the immediate casualty, its collaborative-mode eval is provisional with no fixture, and interview-me is genuinely hard to fixture since it's a dialogue rather than a code task. So before this closes the door I want a real answer for how conversation-shaped skills clear the trusted gate. I also owe run-evals.js itself a proper read since this is a big change to code I wrote, so give me a bit on that.

[addyosmani]

Following up on the dialogue-skill concern, here's a concrete exemption that unblocks it, and I tested it against your branch. The idea: give each behavioral eval a kind. Default execution keeps everything you built (mandatory fixture, trace grading). A new dialogue kind is for skills whose deliverable is the conversation itself, interview-me and idea-refine, where there's no code artifact to inspect: no fixture required, and the grader judges the transcript turns rather than tool calls. It's not a loophole, it's grading the right artifact, and a human reviewer still gates which skills legitimately claim it.

The gate change is small:

const kind = ev.kind || 'execution';
const fixtureRequired = kind !== 'dialogue';
const hasFiles = Array.isArray(ev.files) && ev.files.length > 0 && ev.files.every((x) => typeof x === 'string');
if (fixtureRequired && !hasFiles) { /* error: needs files[] */ }
// provisional only rejected for execution evals; dialogue is trusted by transcript
if (fixtureRequired && ev.trust_level === 'provisional') { /* error */ }

I ran it with interview-me switched to fixtureless dialogue and the suite stays green (124 checks, 86% rank-1), and confirmed a code eval with its fixture stripped still fails, so execution skills keep the hard requirement. It also unblocks #360: its collaborative-mode eval just becomes kind: "dialogue" and drops the provisional flag. Happy to push this onto your branch or you can fold it in, whichever you prefer. I do think it's worth landing before the gate goes hard, so conversation-shaped skills aren't pushed into contrived code fixtures.

[ZhiyaoWen999]

Thanks — I folded both points into 087033d.

• kind now defaults to execution, with execution and dialogue as the only accepted values.
• execution keeps the hard fixture requirement, provisional rejection, throwaway-git workspace, and tool-call-aware grading.
• dialogue can omit files[], runs in an empty throwaway workspace, and the grader treats the conversational turns as the artifact rather than requiring tool calls. I converted both interview-me and idea-refine and removed their contrived fixtures.
• The CI rank-1 floor is now 80, leaving six points of headroom below the current 86% baseline. The docs still make raising the floor expected maintenance as routing improves.

I added runner coverage for fixtureless dialogue evals, missing execution fixtures, provisional execution evals, unknown kinds, and dialogue dry-runs. Validation is green:

• node --test scripts/run-evals-test.js — 12/12 passing
• node scripts/run-evals.js --min-rank1 80 — 124 checks passing, 86% rank-1 (65/76)
• node scripts/validate-skills.js — 24 skills, 0 errors/warnings
• fixtureless dry-runs pass for both interview-me and idea-refine

This should let #381 land first, then #360 can rebase and mark its collaborative-mode eval as kind: "dialogue" without adding a fixture.