Skip to content

[codex] Track rejected rollout failure metrics#2411

Draft
rasdani wants to merge 2 commits intomainfrom
codex/swe-rebench-group-observability
Draft

[codex] Track rejected rollout failure metrics#2411
rasdani wants to merge 2 commits intomainfrom
codex/swe-rebench-group-observability

Conversation

@rasdani
Copy link
Copy Markdown
Contributor

@rasdani rasdani commented May 4, 2026

Summary

  • Keep compatibility errored_rollouts/{env} and errored_rollouts/all metrics.
  • Add scheduler-side rejected-rollout error metrics derived from rollout["failure"], with legacy fallback classification from rollout["error"].
  • Emit error/all/mean, error/{env}/mean, and reason/origin metrics under error_reason/... and error_origin/....
  • Count scheduler task exceptions as env_server_error failures instead of only logging Rollout failed: ....
  • Preserve accepted-rollout metric aggregation under the existing metrics/{env}/... path; rejected rollout failures are tracked scheduler-side before rescheduling.

Companion verifiers failure-payload PR: PrimeIntellect-ai/verifiers#1283.

Compatibility Note

  • The companion verifiers PR intentionally broadens agent_error to all agent-origin failures.
  • agent_nonzero_exit preserves the previous narrow non-zero process exit signal.

Validation

  • uv run ruff check src/prime_rl/orchestrator/scheduler.py tests/unit/orchestrator/test_scheduler.py
  • python -m py_compile src/prime_rl/orchestrator/scheduler.py
  • uv run pytest tests/unit/orchestrator/test_scheduler.py

@rasdani rasdani changed the title [codex] Log SWE rollout group mappings [codex] Track rejected rollout failure metrics May 8, 2026
@rasdani rasdani marked this pull request as draft May 8, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant