Ask: Align the eval-harness workflow with ci.yaml — same pinned Python interpreter and SHA-pinned actions — so the review-quality gate exercises the same interpreter as production CI.
Expected files: .github/workflows/eval-harness.yaml
Problem: eval-harness.yaml (added in #488 to close #472) is the regression gate for review quality — it imports and exercises pr_reviewer.* (tool loop, parser, routing). But it runs on Python 3.12 with floating actions/checkout@v4, actions/setup-python@v5, actions/upload-artifact@v4, while ci.yaml pins SHA-pinned v7 actions and Python 3.14 — with a comment documenting that interpreter skew between local and CI is exactly how a broken regression test once shipped. A quality gate that runs on a different interpreter than the one that ships can pass while CI fails (or worse, mask an interpreter-specific regression), and the weekly scheduled run makes the skew self-reinforcing. Renovate PRs #491–493 already bump the eval-harness actions to the pinned v7 forms (in flight), so the remaining, renovate-invisible gap is the interpreter and the pinning-style inconsistency.
Evidence:
Acceptance:
eval-harness.yaml pins the same Python version as ci.yaml (single source for the version, e.g. a repo-level default or matching literal) and uses the same SHA-pinned action style.- The weekly scheduled eval run and CI both exercise the identical interpreter for the same
pr_reviewer code.
Ask: Align the eval-harness workflow with ci.yaml — same pinned Python interpreter and SHA-pinned actions — so the review-quality gate exercises the same interpreter as production CI.
Expected files: .github/workflows/eval-harness.yaml
Problem:
eval-harness.yaml(added in #488 to close #472) is the regression gate for review quality — it imports and exercisespr_reviewer.*(tool loop, parser, routing). But it runs on Python 3.12 with floatingactions/checkout@v4,actions/setup-python@v5,actions/upload-artifact@v4, whileci.yamlpins SHA-pinned v7 actions and Python 3.14 — with a comment documenting that interpreter skew between local and CI is exactly how a broken regression test once shipped. A quality gate that runs on a different interpreter than the one that ships can pass while CI fails (or worse, mask an interpreter-specific regression), and the weekly scheduled run makes the skew self-reinforcing. Renovate PRs #491–493 already bump the eval-harness actions to the pinned v7 forms (in flight), so the remaining, renovate-invisible gap is the interpreter and the pinning-style inconsistency.Evidence:
.github/workflows/eval-harness.yaml:54-59:actions/checkout@v4,actions/setup-python@v5,python-version: "3.12";:96:actions/upload-artifact@v4— all unpinned floating tags..github/workflows/ci.yaml:16-24:actions/checkout@3d3c42e5... # v7.0.1,actions/setup-python@5fda3b95... # v7.0.0,python-version: '3.14'with the comment: "a version skew between local and CI is how the duplicate-key regression test first shipped broken".scripts/eval_harness.pyimportspr_reviewermodules (e.g.tool_loop,response_parser) and drives the same code paths CI tests, so the interpreter difference is not incidental.python-version: "3.12".Acceptance:
eval-harness.yamlpins the same Python version asci.yaml(single source for the version, e.g. a repo-level default or matching literal) and uses the same SHA-pinned action style.pr_reviewercode.