openenv/tbench2: orphan TTL + ownership labels for per-task sandboxes#1711
Merged
Conversation
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
nblintao
force-pushed
the
tao/tb2-sandbox-recipe
branch
from
July 17, 2026 21:36
293d343 to
6bf15c1
Compare
nblintao
force-pushed
the
tao/tb2-sandbox-ttl
branch
from
July 17, 2026 21:37
0977094 to
8b5fbb2
Compare
nblintao
force-pushed
the
tao/tb2-sandbox-recipe
branch
from
July 20, 2026 02:21
6bf15c1 to
3726743
Compare
nblintao
force-pushed
the
tao/tb2-sandbox-ttl
branch
from
July 20, 2026 02:21
8b5fbb2 to
2a23cdb
Compare
nblintao
force-pushed
the
tao/tb2-sandbox-ttl
branch
from
July 21, 2026 23:26
2a23cdb to
fb54b3f
Compare
Shi-Dong
approved these changes
Jul 22, 2026
| one run's sandboxes for targeted sweeps. | ||
| """ | ||
| try: | ||
| user = getpass.getuser() |
Contributor
There was a problem hiding this comment.
This makes sense for public users, but it seems that within our devbox, getpass.getuser() always returns root. I'm worried that we'd still be unable to distinguish the sandbox owners after this PR.
Contributor
Author
There was a problem hiding this comment.
Yes. We'll use OPENENV_LAUNCHER below to distinguish. But I know it's not very convenient, since there is no default for it. Need to figure out a better way (Claude Code suggested some ways. But I don't think it's good idea to implement them on Miles. It should be changed on devbox's setup).
For now, it's not a blocker since we already have TTL, ownership label is no longer critical.
Contributor
There was a problem hiding this comment.
Kangyan just told me that we can use RADIXARK_USER_EMAIL to identify username within a devbox. Would you mind opening a new PR to update this?
Contributor
There was a problem hiding this comment.
Nvm I think this script is public facing and it'd be weird to have an internal env var hardcoded here.
nblintao
force-pushed
the
tao/tb2-sandbox-ttl
branch
from
July 22, 2026 13:44
fb54b3f to
58a443d
Compare
nblintao
force-pushed
the
tao/tb2-sandbox-recipe
branch
from
July 22, 2026 13:44
eb66c44 to
3534d28
Compare
TB2 is a per-task-image benchmark: every task pins its official runtime image in task.toml, so a cloud sandbox serving the env must be built per task — the official task image plus a tbench2_env server layer, one layer, no DinD. This module owns that recipe, miles-side, next to its only consumer (the per-task Daytona backend of the openenv TB2 adapter). Provider-agnostic core: _server_layer_commands() emits plain shell commands (uv-managed venv at /opt/envserver running the installed tbench2_env package's source, embedded into the build; task dir staged from the tasks checkout's pinned-SHA GitHub tarball with solution/ excluded), so the same layers can back a Dockerfile or another provider. Daytona materialization: create_task_sandbox() creates per-episode declaratively from the Image definition — no named snapshots, no org snapshot quota; repeat creates hit Daytona's build cache — then execs server_cmd() and waits for /health. A bake CLI can pre-register named snapshots as a warm cache. The embedded env source is located from the installed tbench2_env package and requires an editable/checkout install (pyproject.toml must be present); _env_src_dir() fails fast with instructions instead of erroring mid-build. Nothing imports this module yet; the adapter's per-task backend adopts it in a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…half Review feedback on #1710: the module mixed two concerns its own docstring already kept apart. tb2_task_recipe.py now owns everything Daytona-agnostic (server layer commands, env-source embedding, task staging, server_cmd, task.toml reading, /health polling); tb2_task_sandbox.py keeps its name, CLI, and import surface but is purely the Daytona materialization (declarative Image build, Resources, create_task_sandbox, bake). The two symbols that crossed the new module boundary go public with the move: server_layer_commands and resolve_docker_image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tbench2_env fixes are upstream now (huggingface/OpenEnv#965 + #972); embedding the installed source remains the mechanism that guarantees the sandbox runs exactly the version validated locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups on the per-task sandbox recipe: - _dir_tar_b64 is now byte-for-byte deterministic for identical source: gzip mtime=0 suppresses the header compression timestamp and _tar_filter zeroes per-entry mtimes/owners. The b64 is embedded in a build command, so the previous per-call drift changed the image definition on every episode create — defeating the provider build cache the declarative path relies on, and preventing pre-baked snapshots from ever matching. Guarded by a regression test. - rename the module pair to tb2_sandbox_recipe (provider-agnostic recipe) + tb2_sandbox_daytona (Daytona materialization): the subject of both is the sandbox, the trailing token is the role/provider, and future backends slot in as tb2_sandbox_<provider>. - drop the Daytona reference from _task_layer_command's docstring; the recipe module is provider-agnostic and the concrete ceiling is already documented at _MAX_INLINE_TAR_BYTES. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A caller that dies without reaching its delete (SIGKILL, OOM, node loss) used to leak its sandbox forever: auto_stop_interval=0 means Daytona never reclaims it, and running orphans bill CPU+memory+disk indefinitely. Arm auto-stop (30 min) + auto-delete (120 min after stop) at create as a dead-man's switch. Daytona's auto-stop clock counts only SDK interactions — preview-proxy traffic, which is ALL of an episode's I/O, does not reset it — so a bare TTL would kill any healthy episode longer than the interval. A keepalive daemon thread therefore beats refresh_activity() for as long as the creating process lives: a live episode of any length is safe, a dead caller stops beating and its orphan is reclaimed within the TTL. Also label each sandbox with who launched it (OPENENV_LAUNCHER or the unix user) and an optional run id (OPENENV_RUN_ID): the Daytona API records no creator, so in a shared org labels are the only way to attribute sandboxes and scope cleanup sweeps to one launcher or one run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nblintao
force-pushed
the
tao/tb2-sandbox-ttl
branch
from
July 22, 2026 14:01
58a443d to
b99ecf0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the three-PR stack (#1710 → this → #1675 rebased on top). Base is
tao/tb2-sandbox-recipe; only the last commit is new here.Problem
A trainer that dies without reaching its
daytona.delete()(SIGKILL, OOM, node loss) leaks its sandbox forever:create_task_sandboxarmedauto_stop_interval=0, so Daytona never reclaims it, and a running orphan bills CPU+memory+disk indefinitely. We hit exactly this: a batch launch died ~1 minute in and left 30 orphans running 25+ hours until swept by hand. In a shared Daytona org there was also no way to tell whose sandboxes they were — the API records no creator.Fix
Orphan TTL as a dead-man's switch. Creates now arm
auto_stop_interval=30+auto_delete_interval=120(both overridable kwargs). The subtlety: Daytona's auto-stop clock counts only SDK interactions — preview-proxy traffic does not reset it (docs), and an episode's I/O runs entirely over the signed preview URL, so a bare TTL would kill healthy episodes longer than the interval. A keepalive daemon thread therefore beatssandbox.refresh_activity()every 5 minutes for as long as the creating process lives:Ownership labels. Each sandbox now carries, alongside the existing
openenv-tbench2-task:openenv-launcher—$OPENENV_LAUNCHERif set (recommended on shared hosts where the unix user is a generic account), else the local unix useropenenv-run-id—$OPENENV_RUN_ID, when setso shared-org cleanup sweeps can target one launcher or one run (
daytona.list(labels={"openenv-launcher": ...})) instead of guessing.Testing
pytest examples/experimental/openenv/tests/test_tb2_task_sandbox.py -q→ 7 passed. New coverage: labels default to the unix user / honor explicit env vars / omit run-id when unset; creates arm a positive TTL by default; keepalive beats while the sandbox is alive and exits after persistent failures once it's deleted.create_task_sandbox's new params are keyword-only with defaults — the adapter callsite needs no change.Post-reorg sanity (2026-07-17): labels + TTL verified live on a real sandbox (openenv-launcher/run-id present, auto_stop=30 / auto_delete=120 armed, keepalive thread beating, clean delete); full-stack results in #1675's Validation section.
🤖 Generated with Claude Code