fix(gpqa): write evaluator output to results.json#31
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 364f9400af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| output_path = args.output | ||
| elif args.gen_dir: | ||
| output_path = args.gen_dir / "evaluation_results.json" | ||
| output_path = args.gen_dir / "results.json" |
There was a problem hiding this comment.
When --gen-dir is used, find_submission_file() runs before this write and treats any root-level results*.json as the submission candidate, returning it before checking submission*.json. After the first successful evaluation creates gen_dir/results.json, any rerun on the same generation directory will load the previous evaluator output as the model submission, so answers are absent/misread and the new results can overwrite a correct score with incorrect/missing results. Please exclude the evaluator artifact from submission discovery or otherwise disambiguate it before making it the default output name.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Hey @skyzer, Could you also please address these two together:
- Drop
results*.jsonfrom the patterns list used infind_submission_file()- since we explicitly ask agents to producesubmission.json, there's no need to match on that prefix. - Remove the "any JSON file" fallback entirely (lines 93–99), If no explicit pattern matched, guessing will cause the same bug again. Better to fail clearly with a
FileNotFoundError.
There was a problem hiding this comment.
Addressed in 06908e6: removed root-level results*.json discovery, removed the arbitrary *.json fallback, and added regression tests for both cases. Local pytest/ruff/full test suite passed.
364f940 to
cd122f9
Compare
|
Addressed both requested changes in 06908e6:
Local checks passed:
|
yogendrahexo
left a comment
yogendrahexo
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the fix!
2 participants
Summary
results.jsonby default--outputoverrides unchangedWhy
run_evaluation()treatsgen_*/results.jsonas the canonical evaluator artifact. GPQA previously wroteevaluation_results.jsonby default, so evaluation could complete but the orchestrator would still report thatresults.jsonwas missing.Test plan
python3 -m pytest tests/test_gpqa_evaluator.py -qpython3 -m pytest tests/test_gpqa_evaluator.py tests/test_run_evaluation.py tests/test_run_evaluation_outcomes.py -qpython3 -m ruff check .python3 -m pytest -q