nemoclaw onboard --resume fails with route reservation disappeared after interrupted onboard

[mercl-lau]

Description

When nemoclaw onboard is interrupted (e.g. by timeout) during the sandbox creation step and then resumed with nemoclaw onboard --resume, the resume fails with "sandbox route reservation disappeared while onboarding was in progress". This occurs even on a dedicated gateway port with no other sandboxes sharing the route.

The interrupted onboard completes the inference provider step but times out before the sandbox name is assigned. On resume, the wizard replays from the interrupted step but the route reservation created during the first run is no longer present — the product deletes it between the interrupt and the resume.

Platform scope: Reproduced on ubuntu22, wsl-x86, ubuntu24-gpu, dgx-station, macos (5 of 8 CI runners). Passed on ubuntu24, dgxspark, ubuntu26-gpu (non-interrupted path).
Regression: Unknown — earlier versions not tested with this interrupt/resume flow.

Environment

Device:        CI runners (multiple platforms)
OS:            Ubuntu 22.04 / Ubuntu 24.04 / macOS / WSL2
Architecture:  x86_64 / arm64
Node.js:       v22.22.3
npm:           10.9.8
Docker:        29.4.0 / 29.6.1
OpenShell CLI: openshell 0.0.72
NemoClaw:      v0.0.76
OpenClaw:      N/A (onboard did not complete)

Steps to Reproduce

1. Install NemoClaw v0.0.76
2. Run nemoclaw onboard --non-interactive with a short timeout (90 seconds) so it interrupts during the sandbox creation step
3. Verify the onboard session file shows status=in_progress, lastCompletedStep=inference
4. Run nemoclaw onboard --resume to continue the interrupted onboard
5. Observe the error during the messaging/sandbox creation phase

Expected Result

The resumed onboard should continue from the interrupted step and complete successfully. The route reservation created during the first run should persist across the interrupt/resume boundary.

Actual Result

Error: sandbox route reservation '{sandbox-name}' disappeared while onboarding was in progress. Retry onboarding.

Onboarding did not finish. Resume from the step that failed with:
  nemoclaw onboard --resume
Completed steps are skipped; pass --fresh instead to start over.

Logs

[resume] Skipping preflight (cached)
[resume] Skipping gateway (running)
[resume] Skipping provider_selection (compatible-endpoint / openai/openai/gpt-5.4)
[resume] Skipping inference (compatible-endpoint / openai/openai/gpt-5.4)
[non-interactive] No complete messaging channel inputs configured. Skipping.
Error: sandbox route reservation '{sandbox-name}' disappeared while onboarding was in progress. Retry onboarding.

---

NVB#6434437

[zyang-dev]

Reopening this issue because it is still reproducible on DGX Spark with the fix from PR #6572.

[Hokonoken]

Reproduced the reopened symptom on a second platform — x86_64 (WSL2, Ubuntu 24.04) — on current
main (v0.0.79-10-g9dec1fb89, which includes #6572), traced the exact step where the reservation
is still lost, and verified by execution that PR #6626 fixes it. Details below so the repro
can be compared with the DGX Spark one.

The residual chain on main (every step executed, logs kept)

1. A first onboard run fails during sandbox image creation (I used a custom Dockerfile whose
   build fails after validation). With fix(onboard): preserve pending route reservation across resume #6572 the registry row now survives the failed run:
   PRESENT pending=true. So far so good — fix(onboard): preserve pending route reservation across resume #6572 does its job.
2. --resume initially works: it skips the completed inference steps, passes the reservation
   assert, and retries the sandbox step. If that run completes, the loop converges and there is
   no bug.
3. But if the resumed run reaches the not-ready recreate path (registry row exists AND a
   partial sandbox is live on the gateway), createSandboxWithBaseImageResolution prints
   Deleting and recreating sandbox '<name>'... and calls registry.removeSandbox(sandboxName)
   at src/lib/onboard.ts:2695 — unguarded on main. The pending route reservation is deleted.
4. If that recreate run then fails anywhere before re-registering the sandbox (in my chain: the
   OpenClaw image plugin discovery failure of a bare custom image → State was not restored and registry metadata was not updated.), the row is permanently gone.
5. Every subsequent --resume now dies with
   sandbox route reservation '<name>' disappeared while onboarding was in progress. Retry onboarding.
   and recommends --resume again — the dead end this issue describes. Only --fresh escapes.

Registry row across the chain (polling ~/.nemoclaw/sandboxes.json):

after	main v0.0.79-10	PR #6626 (0642e00)
failed fresh run	PRESENT pending=true	PRESENT pending=true + reservationSessionId stamped
first resume (late failure, sandbox live)	PRESENT pending=true	PRESENT pending=true
second resume (not-ready recreate, late failure)	ABSENT	PRESENT pending=true
third resume	reservation disappeared — dead end	retries; fails only on the real (unrelated) error — recoverable

Verification of PR #6626

Replayed the identical chain on pull-request/6626 (0642e00, rebased on current main, #6572
included), same machine, same steps:

• The recreate still runs (Deleting and recreating sandbox ...), but the row survives it:
  removeSandboxUnlessSessionReservation keeps the pending reservation because
  reservationSessionId matches the active onboard session — I confirmed the session id stamped
  at step 1 is identical across all three resumes, so the "same session" check holds across the
  interrupt/resume boundary.
• reservation disappeared never occurs again (grep count 0 on every subsequent resume). Each
  retry fails only on the genuine, unrelated error that made the recreate fail in the first
  place, and the state stays resumable.

So on this platform #6626 does break the dead-end loop. The DGX Spark trigger for the "late
failure after recreate" step may differ from mine (plugin discovery on a bare custom image), but
the loss mechanism is the same line, and the fix holds regardless of the trigger.

Two adjacent observations (for completeness, not blockers)

• pruneStaleSandboxEntry (src/lib/onboard/sandbox-lifecycle.ts:43) also removes the row with
  no pending-reservation guard, but as far as I can tell it has no runtime caller anymore (only
  re-exported for tests), so it is not a live loss path.
• The cancel rollback (cancel-rollback.ts) removes the row on Ctrl-C/SIGTERM while the sandbox
  step is armed, but it also clears the onboard session, so there is nothing left to resume —
  by design, not part of this loop.

Environment

OS:            Ubuntu 24.04 (WSL2), x86_64
Node.js:       v24.18.0
Docker:        29.2.1
OpenShell CLI: 0.0.72
NemoClaw:      main v0.0.79-10-g9dec1fb89 / PR branch 0642e00dd
Provider:      local ollama (llama3.2:1b), dedicated gateway port