feat(task)!: rename the task-config 'environment' key to 'sandbox' (no alias) - #966
Merged
Conversation
BREAKING CHANGE: task.md frontmatter no longer accepts 'environment:' (or 'verifier.environment:') for the sandbox spec — the native key is 'sandbox:' with no alias. Validation fails with an actionable message naming the rename; the one-line fix for existing task.md files is renaming the key. Legacy/Harbor task.toml keeps loading: the toml import boundary (model_validate_toml, import_task_config_toml, render_task_md) translates '[environment]', '[verifier.environment]', and 'steps[*].verifier.environment' to 'sandbox' — deliberate foreign-format translation rather than a native alias. Declaring both spellings in one file is a hard error. - Remove the TaskConfig.environment compat property; callers use TaskConfig.sandbox (rollout, sandbox setup, runtime capabilities, acceptance-live validation). - Rename VerifierConfig.environment to VerifierConfig.sandbox under the same strict/convert rules (verifier.environment_mode is unchanged). - All emitters follow the rename: model_dump_toml, bench tasks migrate, task.md/task.toml scaffolding, authoring profiles, image shorthand normalization, skill-eval/trace/mcp-atlas/toolathlon task generation, and the rubric-review wrapper now write 'sandbox'. - Migrate every in-repo task.md (clawsbench, demo_task, test fixtures, docs examples); task.toml fixtures stay legacy-spelled to pin the conversion. Runtime-capability issue paths report 'sandbox.*'. - Docs: task-authoring key table + examples, task-standard key list and provenance example; CHANGELOG marks the break and the fix. The Environment plane (--environment-manifest, environment manifests, the eval-config docker/daytona selector) is a separate subsystem and is untouched.
…sandbox_mode rename Review follow-up to the environment->sandbox rename — the import-side conversion now has its outbound mirror: - bench tasks export (target 'harbor') emits Harbor's spelling again: convert_native_keys_to_legacy_environment() — the exact inverse of the import converter — renames sandbox -> [environment], verifier.sandbox, sandbox_mode, and steps[*].verifier equivalents in _export_task_toml, so a stock Harbor consumer can read the export ([sandbox] is invisible to it). Tests assert [environment] in the exported task.toml. - Pre-rename compat envelopes no longer corrupt exports: extras spelled 'environment.*' are normalized to 'sandbox' (on a deepcopy — the converter mutates and _compat_extra returns a live sub-dict of the parsed document) before merge_compat_extra, so the merge can never place 'environment' next to 'sandbox' and produce a file that hard-errors on re-import. Reproducing test added. - BREAKING (task.md): verifier.environment_mode is renamed to verifier.sandbox_mode under the same rules — strict native with an actionable error, converted on the task.toml import path, inverse on harbor export. Enum VerifierEnvironmentMode is renamed to VerifierSandboxMode (public API; same shared/separate values). - Straggler user-facing strings now teach the new key: sandbox.skills_dir mount-path labels (skill_policy), sandbox.network_mode guidance (apple-container). - The six in-repo benchmark adapters (harvey-lab, opaquetoolsbench, hilbench, continuallearningbench, programbench, mle-bench) generate [sandbox] task.toml per the emitter policy; the continuallearningbench structural parity check follows. - Nits: harbor.py docstring credits the toml import boundary (not the TaskConfig validator) for the spelling translation; acp/runtime and task-standard doc strings updated; the rename closure bakes in the key pair; steps both-declared errors carry the step index (steps[N].verifier.).
xdotli
had a problem deploying
to
pypi-internal-preview
August 9, 2026 19:32 — with 
GitHub Actions
Failure
GitHub Actions
Failure
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c19332b71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| f"timeout_sec = {verifier_timeout_sec}", | ||
| "", | ||
| "[environment]", | ||
| "[sandbox]", |
There was a problem hiding this comment.
When bench tasks generate --task-format legacy is used, _build_task_toml() is written verbatim to the split package's task.toml, so this change emits [sandbox] even though the legacy/Harbor contract established by this commit says stock Harbor consumes [environment]. The generated package therefore loses its resource and sandbox settings when used by Harbor; keep this TOML legacy-spelled and let render_task_md() perform the existing conversion to sandbox for the native output branch.
Useful? React with 👍 / 👎.
Merged
…nt-to-sandbox # Conflicts: # CHANGELOG.md
xdotli
had a problem deploying
to
pypi-internal-preview
August 9, 2026 20:46 — with
GitHub Actions
Failure
GitHub Actions
Failure
xdotli
added a commit
that referenced
this pull request
Aug 9, 2026
…969) * ci(integration-light): grade code+fixture changes on a coherent tree rollout-smoke overlays ONLY src/benchflow onto the trusted base tree, so a PR that must change src AND the in-repo task fixtures together is graded against an incoherent tree: PR code + base fixtures. PR #966 (the task-config 'environment' -> 'sandbox' rename) hit exactly that — the overlaid new parser correctly rejected base's un-migrated task.md, so the lane reported a version skew the PR had already fixed, and main went green the moment both halves landed together. Add fixture-scenarios: a job holding NO secrets (no environment:, contents: read) that overlays src/benchflow AND docs/examples/task-md from the PR head and runs the Docker+oracle scenario. Fixtures are data under test, not verdict logic — but they are not inert (task.md carries sandbox.setup_commands and sandbox.docker_image; each task ships solve.sh and verifier/test.sh), so they execute only where there is nothing to exfiltrate. Verdict integrity is unchanged: the assertions in tests/ still come from the trusted base, so a PR cannot rewrite the test that grades it. rollout-smoke keeps the secret-bearing, trusted-fixture coverage and hands oracle_determinism to the new job. Its one remaining fixture-dependent leg (sandbox_parity) needs DAYTONA_API_KEY and so cannot move; when the PR changes fixtures it is skipped with a loud ::notice instead of failing on skew the PR already resolved — coverage is not silently dropped, since fixture-scenarios graded the coherent pair and main re-runs the leg after merge. release-gate requires the new job explicitly: adding it to needs alone would make the gate wait for it without enforcing its conclusion. * test(ci): pin the release gate to the coherent-tree job tests/test_release_workflow_wiring.py executes the release-gate script under 'set -u' and asserts its needs list, so it caught both halves of the new wiring: the added dependency and the new FIXTURE_SCENARIOS_RESULT env var. Update the pinned needs to the intended three-job chain, feed the new variable through the harness, and add a fail-closed case — a red fixture-scenarios must block the preview, which is exactly what 'needs' alone would NOT enforce.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BREAKING — merge only after the 0.6.7 release is tagged; this targets 0.7.
User decision, verbatim: "we dont want no alias. it should be a rename." The task-config sandbox-spec key had two accepted spellings (
environmentvia alias,sandboxvia populate_by_name), the docs taught one, the emitters wrote it, and SkillsBench house style used the other. One name now.The rename
sandboxonly.environment:/verifier.environment/verifier.environment_modefail with actionable rename messages that beat pydantic's extra-forbid at every nesting level (verified through both dict and task.md ingestion). The compat propertyTaskConfig.environmentis gone;VerifierEnvironmentMode→VerifierSandboxMode.convert_legacy_environment_keys()applied at every toml ingestion path (review-verified exhaustively: model_validate_toml, import_task_config_toml, render_task_md's toml arm). Foreign-format translation at the boundary, deliberately not an alias; both-spellings-in-one-file is a hard error naming the fix.bench tasks export --target harborapplies the exact inverse conversion, so exported task.toml carries[environment]for stock Harbor consumers; the pre-rename compat-envelope hole (silently exporting both spellings, hard-failing on re-ingest — reproduced by the reviewer) is closed with a converted deepcopy before the merge.sandbox: migrate, dumps, scaffolding, task_gen, skill_eval, rubric-review wrapper, and the six in-repo benchmark adapters. 15 in-repo task.md files migrated. Environment-plane (--environment-manifest, registry,ManifestEnvironment) is a different subsystem and untouched.External coordination
sandbox:.TaskConfig(environment=…)or.config.environmentbreaks loudly (actionable message / AttributeError) — the point of a hard rename. CHANGELOG carries the BREAKING entry with one-line fixes for all three renamed spellings.Review
Structural review (REQUEST-CHANGES → fixed): inbound conversion coverage verified exhaustively; both MAJORs were the same boundary principle applied outbound (Harbor export + compat envelope), both closed with reproducing tests;
sandbox_modefolded in so users eat one breaking change, not two.Gates: ruff format/check, ty; focused task/export/import suites 169+ passed; full suite 5354 passed (3 known host-specific failures, verified pre-existing).