fix: gpqa grader writes results.json so the harness picks it up#34
Closed
francisco-perez-sorrosal wants to merge 1 commit into
Closed
Conversation
The orchestrator runs each task's evaluate.py with only --gen-dir and then reads gen-dir/results.json (Names.RESULTS_JSON). The gpqa grader, however, defaulted its output to gen-dir/evaluation_results.json, so under the harness it produced no results.json: run_evaluation() fell through to the "results.json not created by evaluate.py" warning and the gpqa scores never reached the feedback loop. The lawbench and longcot-chess graders already default to results.json (lawbench even notes it is "required by orchestrator"); gpqa was the lone outlier. Default the gpqa grader's output filename to results.json to match that contract, and add a regression test that runs the real grader the way the orchestrator does and asserts results.json (not the legacy name) is written. The submission-discovery path is unaffected: the reference agent writes its submission into the results/ subdirectory, which find_submission_file() checks before the gen-dir root, so the grader's own results.json is never mistaken for a submission.
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gpqa task's grader never produces results the harness can read, so gpqa runs are silently scored as a "warning" and the scores never reach the feedback loop.
The orchestrator runs each task's
evaluate.pywith only--gen-dirand then readsgen-dir/results.json:run_evaluation()invokes[python, evaluate.py, "--gen-dir", gen_directory](no--output).os.path.exists(gen-dir/results.json)(Names.RESULTS_JSON).The gpqa grader, however, defaulted its output to
gen-dir/evaluation_results.json. With no--outputpassed, it wrote the wrong filename, soresults.jsonwas never created andrun_evaluation()fell through to:The sibling graders already honor this contract —
lawbenchwritesresults.json(its code even comments "required by orchestrator") andlongcot-chessdefaults toresults.json. gpqa was the lone outlier.Fix
Default the gpqa grader's output filename to
results.json(the explicit--outputflag still overrides). This is a one-file, three-line change insia/tasks/gpqa/data/public/evaluate.py, plus a brief comment recording the contract so it does not drift again.Why it's safe
The submission-discovery path is unaffected. The gpqa reference agent writes its submission into the
results/subdirectory, andfind_submission_file()checksresults/before thegen-dirroot — so the grader's ownresults.jsonat the gen-dir root is never mistaken for a submission. This is exactly howlongcot-chessalready behaves.Tests
Added
tests/test_gpqa_grader_output.py, which runs the real grader the way the orchestrator does (--gen-dironly) against the private ground truth and a synthetic submission, then assertsresults.json(notevaluation_results.json) is written and scored correctly. Verified it fails against the previous grader and passes with this change. The existing eval/task/context/orchestrator suites continue to pass.