Fix ZMQ env timeouts

[xeophon]

Summary

• make the ZMQ env request timeout configurable internally, with a 24h default
• derive the env-server request timeout from existing timeout_minutes when present, or from hosted-runner PRIME_EVAL_TIMEOUT_MINUTES when running from CLI flags
• record ZMQ env request timeouts as failed samples with stop_condition="env_request_timeout" instead of aborting the whole eval, while leaving unrelated TimeoutErrors untouched

A companion platform change is prepared locally to set PRIME_EVAL_TIMEOUT_MINUTES in hosted eval runner sandboxes from the existing timeout_minutes field.

Verification

• uv run pytest tests/test_eval_cli.py tests/test_env_server.py tests/test_environment_extra.py — 104 passed
• uv run ruff check verifiers/types.py verifiers/utils/eval_utils.py verifiers/scripts/eval.py tests/test_eval_cli.py tests/test_env_server.py tests/test_environment_extra.py docs/evaluation.md docs/reference.md
• previous full uv run pytest tests/ had 1418 passed, 29 skipped, 3 unrelated failures: missing OpenAI key for toxicity_explanation/wiki_search and unsatisfiable bfcl_v3 dependency on mistralai

---

Note

Medium Risk
Changes ZMQ env request behavior by removing the 10h hardcoded default timeout, which can alter how long evals wait for env responses (potentially hanging if callers don’t supply a timeout). Scope is small and isolated to the ZMQ client.

Overview
Removes the hardcoded 10h DEFAULT_REQUEST_TIMEOUT in ZMQEnvClient and replaces it with None, making env request timeouts unlimited by default unless an explicit timeout is passed to send_request.

^{Reviewed by Cursor Bugbot for commit 64f2b92. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[xeophon]

Addressed the review comments in f37b555:

• documented timeout_minutes in docs/evaluation.md, including how hosted runner PRIME_EVAL_TIMEOUT_MINUTES maps to env-server request timeout behavior
• updated skills/evaluate-environments/SKILL.md with the hosted timeout workflow and distinction from per-rollout --timeout
• the original --env-request-timeout CLI flag comment is stale because that flag was removed in favor of the existing hosted timeout budget

Verification:

• uv run pytest tests/test_eval_cli.py tests/test_env_server.py tests/test_environment_extra.py — 104 passed
• uv run ruff check verifiers/types.py verifiers/utils/eval_utils.py verifiers/scripts/eval.py tests/test_eval_cli.py tests/test_env_server.py tests/test_environment_extra.py — passed

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 64f2b92. Configure here.}

[cursor]

Default timeout removed, requests can hang forever

High Severity

DEFAULT_REQUEST_TIMEOUT was changed from 36_000 (10 hours) to None, removing the safety-net timeout entirely. The PR description states "with a 24h default," but None means asyncio.wait_for waits indefinitely. Since all callers (EnvClient.run_rollout and run_group) pass timeout=None, and nothing in the codebase overrides DEFAULT_REQUEST_TIMEOUT on instances, every ZMQ request now blocks forever if the server fails to respond.

Additional Locations (1)

• verifiers/serve/client/zmq_env_client.py#L270-L271

 

^{Reviewed by Cursor Bugbot for commit 64f2b92. Configure here.}