OpenShell listener-start failure leaves onboarding polling a terminated forward for 180 seconds

[cjagwani]

Investigation Summary

• Exact-head credentialed E2E for ci(e2e): add exact-SHA Brev Launchable E2E lane #7043 reached a healthy Hermes sandbox, then failed only while creating the required host dashboard forward.
• OpenShell 0.0.85 reported that 127.0.0.1:18789 did not open within 10 seconds; openshell forward list stayed empty and the host listener stayed closed.
• NemoClaw then polled the empty forward list for 180 seconds before correctly failing closed and deleting the sandbox.
• Closed [macOS][Onboard] nemoclaw onboard deletes the created sandbox after dashboard port-forward "did not appear in list within 180000ms" #6099 / merged fix(onboard): match mux-delegated ssh forwards in untracked-forward fallback (#6099) #6833 cover the different case where OpenShell cannot track a forward that is already listening. Their live-port requirement correctly rejected this run.
• Unknown: why the SSH listener failed on this runner and whether a bounded clean retry would recover. The run was not retried merely to obtain a green gate.

Description

On Ubuntu GitHub-hosted E2E, a real Hermes onboarding run built the sandbox and reported the in-sandbox dashboard healthy. The subsequent OpenShell host-forward attempt terminated after its 10-second listener check:

ssh process started but local forward listener was not reachable
local forward listener did not open on 127.0.0.1:18789 within 10000ms

No matching forward appeared and the port was not live, so NemoClaw correctly refused the existing untracked-forward fallback. However, it continued polling for 180 seconds even though OpenShell had already terminated that SSH attempt, then rolled the sandbox back.

Expected behavior needs maintainer confirmation. The smallest fail-closed option is to recognize only OpenShell's definitive listener-start termination diagnostic, verify that the port is still closed, stop/clean the exact sandbox-and-port attempt, and retry a bounded number of times. Every successful attempt must still require either an exact running (sandbox, port) row or the existing diagnostic-plus-live-listener exception. Authentication errors, another sandbox's row, arbitrary listeners, and persistent failures must never pass.

This should be handled independently from #7043; that PR does not change forwarding, Hermes, or OpenShell installation behavior.

Reproduction Steps

Recorded reproduction (not claimed to be deterministic):

1. Check out NVIDIA/NemoClaw@68d004f8ad66337fb1e1919b92a9536609a643b4.
2. Install OpenShell 0.0.85.
3. Run the credentialed E2E security-posture target for the Hermes matrix entry on Ubuntu 24.04.
4. Observe that install.sh --non-interactive --fresh builds the sandbox and reports Dashboard is live.
5. Observe the OpenShell listener-start error above, an empty forward list, five NemoClaw progress reports through 171 seconds, and fail-closed sandbox deletion.

Public evidence: child workflow run 29777141848, Hermes security-posture job 88469361681.

The same child run passed credential sanitization, cloud onboarding, full E2E, and OpenClaw security posture. Earlier Hermes security-posture runs on the same #7043 change passed, which is why this report does not claim a deterministic product regression.

Environment

• Host: GitHub-hosted Ubuntu 24.04.4 LTS, Linux 6.17.0-1020-azure, x86_64, 4 vCPU / 15.6 GiB
• Node.js: 22.23.1
• Docker Engine: 28.0.4
• OpenShell CLI: 0.0.85, Docker-driver gateway
• NemoClaw SHA: 68d004f8ad66337fb1e1919b92a9536609a643b4
• Agent: Hermes 0.18.0

Debug Output

The E2E artifact and logs are attached to the public workflow job linked above. The decisive output is:

Waiting for NemoClaw dashboard to become ready...
✓ Dashboard is live
Still waiting for forward on port 18789 to register (42s elapsed)...
Still waiting for forward on port 18789 to register (81s elapsed)...
Still waiting for forward on port 18789 to register (111s elapsed)...
Still waiting for forward on port 18789 to register (141s elapsed)...
Still waiting for forward on port 18789 to register (171s elapsed)...
Could not allocate a dashboard port for 'e2e-hermes-security-posture'.
Failed to start dashboard forward on port 18789: forward did not appear in list within 180000ms; last forward list: <empty> Error: × ssh process started but local forward listener was not reachable ╰─▶ local forward listener did not open on 127.0.0.1:18789 within 10000ms
The orphaned sandbox has been removed — you can safely retry.

Checklist

• I confirmed this bug from a real credentialed E2E run and preserved its artifacts
• I searched existing issues and this is not a duplicate