[DGX Spark][CLI&UX] stop leaves the dashboard port-forward alive; status flags sandbox_dashboard_port_conflict and start/recover intermittently fail

[hulynn]

Description

nemoclaw {sandbox} stop returns success but does not tear down the host-side dashboard SSH port-forward it created. After stop, a nemoclaw-spawned ssh ... -L 127.0.0.1:18789:127.0.0.1:18789 sandbox keeps LISTENing on the dashboard port. As a result nemoclaw {sandbox} status classifies the cleanly stopped sandbox via sandbox_dashboard_port_conflict ("the dashboard port is held by a foreign listener" — actually nemoclaw's own leftover forward), and nemoclaw {sandbox} start / nemoclaw {sandbox} recover intermittently fail their gateway-readiness probe (contending for the still-held port / probing before the restarted container is healthy).

Related but distinct from #7222: that issue covers the downstream recovery hint (status recommends rebuild --yes, which aborts on the pre-rebuild backup gate; start is the real recovery). This report is the upstream cause on stop (the dashboard port-forward is left running) plus the intermittent start/recover gateway-probe failures — neither of which #7222 exercises (it crashes the container with docker kill, never runs nemoclaw stop).

Platform scope: Reproduced on DGX Spark (GB10, aarch64) only; other platforms not tested. Root cause (host-side port-forward not torn down on stop) appears platform-independent.

Regression: Unknown — earlier versions not tested.

Environment

Device:        DGX Spark (GB10, aarch64)
OS:            Ubuntu 24.04
Architecture:  aarch64
Node.js:       v22.22.1
npm:           10.9.4
Docker:        28.3.3
OpenShell CLI: 0.0.85
NemoClaw:      v0.0.88
OpenClaw:      2026.6.10

Steps to Reproduce

1. Onboard an OpenClaw sandbox (e.g. {sandbox}) on ollama-local; confirm Phase: Ready and note the dashboard port (e.g. 18789).
2. On the host, list listeners on the dashboard port; a nemoclaw-spawned forward holds it:

   ss -ltnp | grep 18789
   # -> LISTEN 127.0.0.1:18789 ... users:(("ssh",pid={pid}))
   # that ssh is: ssh ... -o ExitOnForwardFailure=yes -L 127.0.0.1:18789:127.0.0.1:18789 sandbox

3. Stop the sandbox:

   nemoclaw {sandbox} stop            # exits 0; container goes to Exited (137)

4. Re-check the dashboard port on the host:

   ss -ltnp | grep 18789
   # -> the SAME ssh -L 127.0.0.1:18789 forward is STILL listening (orphaned; stop did not tear it down)

5. Check status: nemoclaw {sandbox} status (see Actual Result).
6. Repeat a few times:

   nemoclaw {sandbox} start
   nemoclaw {sandbox} recover
   # observe intermittent gateway-probe failures (EXIT 1) that leave the container Exited, vs other runs that succeed

Expected Result

• nemoclaw {sandbox} stop tears down the host-side dashboard port-forward it created; no listener remains on the dashboard port after stop.
• nemoclaw {sandbox} status on a cleanly stopped sandbox does not report a sandbox_dashboard_port_conflict caused by nemoclaw's own leftover forward.
• nemoclaw {sandbox} start / recover deterministically restart the container and wait for it to become healthy before reporting success or failure — no intermittent false failure while the container is still coming up.

Actual Result

After nemoclaw {sandbox} stop (exit 0; container Exited 137), the dashboard forward is still alive:

$ ss -ltnp | grep 18789
LISTEN 0 128 127.0.0.1:18789 0.0.0.0:*  users:(("ssh",pid={pid},fd=3))

$ nemoclaw {sandbox} status            # EXIT 1
Failure layer: sandbox_dashboard_port_conflict — sandbox container is stopped and the dashboard port is held by a foreign listener.
  Phase: Error

start / recover are flaky across repeats:

$ nemoclaw {sandbox} start             # sometimes EXIT 1
  Probe failed: OpenClaw gateway is not running in '{sandbox}' and automatic recovery failed.
$ nemoclaw {sandbox} recover           # sometimes EXIT 1
  Probe failed: could not inspect the OpenClaw gateway inside sandbox '{sandbox}'.
# other runs: EXIT 0 "recovered OpenClaw gateway", container Up healthy in ~30-50s

Logs

# openshell reports the stopped sandbox as Provisioning (not Stopped):
$ openshell sandbox list
NAME       CREATED               PHASE
{sandbox}  2026-07-20 ...        Provisioning

# docker: the container was SIGKILLed by stop
{container} :: Exited (137)

# the orphaned host-side forward (sandbox-id and token scrubbed):
ssh -o ProxyCommand='.../openshell ssh-proxy --gateway https://127.0.0.1:8080 --sandbox-id {sandbox-id} --token {token} --gateway-name nemoclaw ...' -N -o ExitOnForwardFailure=yes -L 127.0.0.1:18789:127.0.0.1:18789 sandbox

# Recovery that reliably works: `nemoclaw {sandbox} start` until the container is Up/healthy
# (openshell PHASE=Ready), after which status is EXIT 0 Ready and agent inference works — the
# sandbox is never actually corrupted. Still reproduces on v0.0.88 / OpenShell 0.0.85.

[yanyunl1991]

Thanks for the precise report and the clear separation from #7222 — reproduced and fixed.

Root cause
stopSandbox (src/lib/actions/sandbox/stop.ts) stopped the labeled container(s) and returned, but never released the host-side dashboard port-forward it had created. That forward is a nemoclaw-managed openshell forward (ssh -L 127.0.0.1:<port>); onboarding, destroy, and forward-recovery all tear it down with openshell forward stop <port> <sandbox>, but stop did not — so the listener outlived the container. The status classifier (gateway-failure-classifier.ts, sandbox_dashboard_port_conflict) then saw a stopped container plus a live listener on its dashboard port and reported a "foreign listener" conflict — the listener was nemoclaw's own leftover forward. start/recover re-established the forward and probed gateway readiness while the just-stopped forward was still bound, which is the intermittent failure.

Fix
After the container is stopped, release the forward via a new teardownSandboxDashboardForward helper that runs openshell forward stop <port> <sandbox> — reusing the exact teardown already used by onboard/destroy/forward-recovery. It runs after a successful stop, in the already-stopped (idempotent) branch too, and never when a docker stop fails (the container is still running, so its forward must stay).

PR
#7228

Verified end-to-end: after stop no listener remains on the dashboard port, status reports sandbox_container_stopped instead of the phantom conflict, and three repeated start/stop cycles restart to Ready deterministically with the forward released each time. (Reproduced on an x86_64 host; the root cause — host-side port-forward not torn down on stop — is platform-independent.)

Will land after CI is green and maintainer review. To try it locally: git fetch origin fix/7227-stop-teardown-dashboard-forward && git checkout fix/7227-stop-teardown-dashboard-forward, then re-run the original scenario.

If you still see intermittent start/recover failures after this lands — independent of the (now-released) leftover forward — reply here and I'll take another look.

[mercl-lau]

QA verification update — PARTIAL / FAIL on v0.0.96

Environment:

• macOS arm64 (Apple M4 Pro)
• Docker Desktop 29.2.1
• NemoClaw v0.0.96
• OpenShell 0.0.85
• OpenClaw v2026.7.1
• Isolated temporary sandbox restored from a known working snapshot

This run locally corroborates the platform-independent forward teardown and start/recovery paths; the original report was reproduced on DGX Spark.

Actual results:

1. Held an active dashboard connection and confirmed a live OpenShell SSH forward on port 18790.
2. nemoclaw <sandbox> stop returned exit 0; the listener was absent afterward and the sandbox forward disappeared from openshell forward list.
3. nemoclaw <sandbox> status then reported sandbox_container_stopped, not sandbox_dashboard_port_conflict.
4. Ran three start → probe → stop cycles. Cycle 1 returned exit 1: the recreated sandbox reached OpenClaw running, but OpenShell phase Error prevented the primary dashboard/API forward and instructed running openshell forward start/recover. connect --probe-only recovered it successfully. Cycles 2 and 3 started successfully. Every stop removed the listener.

The stale dashboard-forward cleanup and status classification addressed by #7228 are fixed, but the intermittent start/recovery portion still reproduced in 1 of 3 cycles. The issue is therefore not fully verified.

Reopening for the remaining intermittent start/recovery failure.

from kookies

[yanyunl1991]

QA reproduction on latest main — the remaining start/recovery failure still reproduces

Reproduced the reopened intermittent start/recovery symptom on latest main (well past v0.0.96, so it already includes the newer readiness work — e.g. the docker-driver recovery unpause fix and the readiness-reconciliation change). It is therefore not resolved by the post-v0.0.96 changes.

Environment

• our DGX Spark aarch64 test host (GB10 GPU), Ubuntu 24.04
• NemoClaw main @ eeab81cc5 (v0.0.96-45), OpenShell 0.0.85, provider ollama-local
• Fresh onboard, then 12 × stop → start cycles

Result: 1 / 12 cycles failed (cycle 1), matching the report. The #7228 forward teardown + status classification are confirmed working — every stop removed the listener, and the other 11 cycles started cleanly with the dashboard forward running. The failing cycle:

Probe failed: OpenClaw gateway is running in 'repro-7227', but the recreated sandbox
did not become ready in OpenShell, so the primary dashboard/API host forward was not started.
Last OpenShell readiness error: Error: sandbox 'repro-7227' is not ready (phase: Error);
wait for it to reach Ready state.
Run `openshell forward start --background <port> repro-7227` manually and re-run
`nemoclaw repro-7227 recover`.

Read of the root cause: on start, the container comes up and the OpenClaw gateway runs, but OpenShell transiently reports the sandbox phase: Error during the post-start settling window. The start/recovery path treats that first Error reading as terminal and gives up on the primary dashboard/API forward instead of waiting/retrying through the transient phase, so the forward is left unstarted and the user is told to recover manually. This hit the first start after onboard in both this run and the reopen report, which suggests a cold-start timing correlation.

Working on a bounded readiness-retry so a transient post-start phase: Error is retried (with backoff) before surfacing the manual-recover hint.

[yanyunl1991]

Opened #7784 for the reopened intermittent start/recovery failure.

The earlier fix (#7228) addressed the leftover dashboard port-forward, but a second, distinct facet remained: the post-recreate OpenShell readiness wait defaulted to 30s, while connect's readiness wait (waitForSandboxReadyOrExit) allows 120s — both prove the same post-recreate sandbox readiness. On a cold start the sandbox can sit in a transient phase: Error settling window that exceeds 30s but is within 120s, so start's recovery abandons it and prints the manual-recover hint, while connect --probe-only (larger budget) recovers the same sandbox. #7784 aligns both readiness-wait default sites to 120s via a single constant; the NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS override is unchanged.

Honesty note: this is a low-probability cold-start race that reproduced only ~1 in 12 start cycles and could not be re-triggered deterministically afterward, so #7784 is a principled budget alignment rather than an empirically-timed value. It uses Refs (not Fixes) so this issue stays open to confirm over repeated cold starts before closing.