feat(cli): surface fractional rewards in console summaries - #960
Conversation
The console binarizes rewards (pass = reward==1), so a task scoring 0.3 partial credit printed identically to a flat 0: [FAIL] task (tools=47), Job complete: 0/1 (0.0%), Score: 0/1 (0.0%). Rubric-style verifiers make 0<r<1 the common case, and the fractional signal was console-invisible. Keep the binarized counts (thresholding is intentional) and add the fractional view alongside: - per-task lines carry the scored reward: [FAIL] task (reward=0.30, tools=47); error lines (reward None) are unchanged - EvaluationResult gains mean_reward (mean over scored rollouts, None when nothing scored — errors are excluded, not zeroed) - the job-complete log line carries mean_reward=0.30 and the CLI Score line renders ', mean reward 0.30' when available (getattr-guarded: sharded aggregation doesn't carry it) - summary.json gains the same mean_reward field for machine consumers
…-up)
The construction-site extraction was a fourth ad-hoc read of
rewards.reward, written one import away from the canonical
extract_reward — and the only read in the aggregation flow that could
crash on malformed resume data ((rewards or {}).get raises on a truthy
non-dict rewards, and the resume path feeds raw json.loads payloads
with no shape validation). It also admitted NaN into the mean.
Extract mean_scored_reward() into _utils/scoring.py built on
extract_reward with the bool + isfinite guard, use it at the
EvaluationResult construction site, unit-test the helper (bool
exclusion, NaN exclusion, malformed-shape tolerance), and pin the
summary.json mean_reward field end-to-end.
GitHub Actions
Inactive
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a193d388b9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for r in results | ||
| if isinstance(rw := extract_reward(r), (int, float)) | ||
| and not isinstance(rw, bool) | ||
| and math.isfinite(rw) |
There was a problem hiding this comment.
When resuming a job whose raw result.json has an integer reward too large to convert to a C double (for example, a 1,000-digit JSON integer), this predicate accepts it as an int and math.isfinite(rw) raises OverflowError; _get_completed_tasks deliberately passes such JSON rewards through without validation, so Evaluation.run() now aborts during aggregation rather than producing the summary. Validate or bound integers, or handle the conversion failure before treating the value as scored.
Useful? React with 👍 / 👎.
| """Partial credit must survive the binarized pass/fail view: run() | ||
| computes mean_reward over scored rollouts (errors excluded, not | ||
| zeroed), the per-task lines carry reward=, error lines don't, and the | ||
| job-complete line carries mean_reward=. | ||
| """ |
There was a problem hiding this comment.
These newly added tests guard the observed regression where fractional rewards disappeared from console and summary output, but this docstring—and the other new regression cases—only restates semantics and never identifies a PR or commit. Add the guarded change identifier to each regression-test docstring so the coverage remains traceable under the repository convention.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
1 participant
The console binarizes rewards (pass = reward==1), so a task scoring 0.3 partial credit printed identically to a flat 0 — observed on a real env0 run:
[FAIL] gdoc-extract-content (tools=47),Job complete: 0/1 (0.0%),✗ Score: 0/1 (0.0%)for a rollout whose verifier returned 0.3 with rich metrics. Rubric-style verifiers make 0<r<1 the common case; the fractional signal was console-invisible.What changed
Binarized counts stay (thresholding is intentional); the fractional view renders alongside:
[FAIL] gdoc-extract-content (reward=0.30, tools=47); error lines (reward None) unchanged.EvaluationResult.mean_reward: mean over scored rollouts via a new canonicalmean_scored_reward()in_utils/scoring.py(built onextract_reward; excludes bools —classify_resulttreats a persistedtrueas a pass but a bool is not a magnitude — and non-finite values, since the resume path feeds unvalidatedjson.loadspayloads). None when nothing scored — errors are excluded, not zeroed, so an all-error run shows no fabricated 0.00.Job complete: 0/1 (0.0%), mean_reward=0.30, errors=0, …(key=value, matching its neighbors).✗ Score: 0/1 (0.0%), mean reward 0.30, errors=0(prose, matching the failure-reason lines below it). getattr-guarded: sharded aggregation omits the segment.summary.jsongains the samemean_rewardfield (additive), pinned end-to-end by test.Review
Structural review before opening (REQUEST-CHANGES → fixed): the initial construction-site extraction was a fourth ad-hoc read of
rewards.rewardone import away from the canonical helper — and the only read in the aggregation flow that could crash on a malformed resumed payload, plus a NaN hole. The shipped version extractsmean_scored_reward()besidepass_rate/count_score_outcomeswith unit tests for the bool/NaN/malformed-shape edges. Known follow-ups deliberately not in scope: sharded workers don't report per-shard means (aggregatemean_rewardstays None — needs worker payload + weighted merge), andbench eval metrics' post-hoc summary doesn't yet carry the field.Gates: ruff format/check, ty; test_scoring + test_job + test_cli_live_progress 135 passed;
-k "cli"366 passed; oracle-chokepoint/loop-strategies substring assertions verified unaffected (110 passed).