Align eval-harness Python version with CI (3.14) - #498
Conversation
Fixes #496 Change python-version in eval-harness.yaml from "3.12" to '3.14' to match ci.yaml, and add a comment mirroring the version-skew rationale documented in ci.yaml. The action pins were already SHA-pinned by Renovate PRs #491–493; this closes the remaining interpreter gap. Signed-off-by: Saffron <263493777+itsmiso-ai@users.noreply.github.com>
There was a problem hiding this comment.
AI Automated Review
Full PR review.
Analysis engine: dsv4f@https://litellm.jory.dev/v1 (openai) — primary route
Recommendation
Approve. This is a minimal, well-targeted fix that aligns the eval-harness workflow's Python interpreter with CI (3.14), directly satisfying the linked issue's primary acceptance criterion ("eval-harness.yaml pins the same Python version as ci.yaml ... matching literal"). The validate CI check passes on this commit, and repo history/grep confirm ci.yaml pins python-version: '3.14' as the reference. No blockers or material issues found.
Change-by-change findings
.github/workflows/eval-harness.yaml
python-versionchanged from"3.12"to'3.14', byte-for-byte the same literal asci.yaml:24(python-version: '3.14'), so the weekly scheduled eval run and CI now exercise the identical interpreter on the samepr_reviewercode. This was the one gap Renovate couldn't close (the Renovate PRs PR 491–493 already converted the workflow's actions to SHA-pinned v7 forms, which I confirmed are present in the current file at checkout3d3c42e5...,5fda3b95..., and043fb46d...).- The new explanatory comment matches the spirit and content of the
ci.yamlcomment about the duplicate-key regression and the local 3.9 vs. CI 3.x skew; it's accurate and mirrors the accepted rationale from commit112cd5a. - The quoted scalar (
'3.14') matches the exact style used inci.yaml, which aids the stated goal of keeping the two workflows visually consistent. - No security, secret-handling, or path concerns: this file only touches workflow interpreter pinning, uses the existing
secrets.*references unchanged, and carries no risk flags.
Sources
- Full contents of
.github/workflows/eval-harness.yaml(already SHA-pinned actions v7 + new3.14pin) git grepfor3.14/python-version: matches in both.github/workflows/ci.yaml:24and.github/workflows/eval-harness.yaml:62, with identical co-located comments- Repository history:
a5f5814(this PR's head),84f0e6c(eval harness CI workflow PR 488),112cd5a(ci: catch duplicate YAML keys + pin CI Python to 3.14 PR 256), Renovate PRs PR 491–493 (action SHAs) - Issue PR 496 body (acceptance criteria and evidence)
- CI check result for
a5f5814:validatesuccess
Standards Compliance
AGENTS.md contains no workflow-interpreter or action-pinning convention that conflicts with this change. The change is consistent with the documented CI-pinning precedent (SHA-pinned v7 actions + 3.14 interpreter) and with the repository's overall stance that the review-quality gate should exercise the same interpreter as shipping CI. No prohibition on workflow comment style or quoted-scalar style exists in the standards file.
Linked Issue Fit
Issue PR 496 asks that the eval-harness workflow use the same pinned Python interpreter as ci.yaml and the same SHA-pinned action style, so the review-quality gate exercises the same interpreter as production CI. Verified against the issue's acceptance criteria:
- "pins the same Python version as ci.yaml (single source for the version, e.g. a repo-level default or matching literal)" — Met.
python-version: '3.14'is now an exact matching literal toci.yaml:24. While a repo-level default would be the ideal single source, the issue explicitly names "matching literal" as an acceptable form. - "uses the same SHA-pinned action style" — Met. The current workflow file pins
actions/checkout,actions/setup-python, andactions/upload-artifactto v7 SHAs (delivered by the in-flight Renovate PRs PR 491–493 that the issue anticipated); this PR completes the remaining interpreter gap. - "The weekly scheduled eval run and CI both exercise the identical interpreter for the same pr_reviewer code" — Met. Both workflows now run Python
3.14.
No acceptance criterion is missed or misinterpreted; the PR's scope is exactly the interpreter alignment the issue identified as remaining.
What
Pinned Python version to 3.14 in eval-harness.yaml to match CI and resolve duplicate-key regression test failure.
Why
Version skew between local (3.9) and CI (3.x) caused the duplicate-key regression test to ship broken; pinning to 3.14 ensures consistent behavior.…
Fixes #496
Opened by foreman on review GO (workload wl-misospace-pr-reviewer-action-496).