Sandbox gateway dial-back with valid shared token rejected as "pairing required" — OpenShell transparent proxy erases loopback locality, blocking channel tools on the warm daemon path

[stefanopalmieri]

nemoclaw-debug.tar.gz

Investigation Summary

• Deterministic repro: any channel-tool invocation on the warm daemon path (POST /tools/invoke or a /v1/chat/completions turn) fails tool execution failed; the in-sandbox gateway log shows pairing required: device is not approved yet on a 10.200.0.1 → 10.200.0.2:18789 connection.
• Packet capture of both proxy legs inside the sandbox network namespaces shows the dial-back's connect frame carries auth.token byte-identical (SHA-256-compared) to gateway.auth.token — the credential is not dropped by the reviewed dist patch or the OpenShell relay; the gateway still answers NOT_PAIRED and closes 1008.
• Cause confirmed in OpenClaw v2026.6.10 source: OpenShell's transparent proxy re-originates the connection from the sandbox veth peer, so isSharedSecretLoopbackLocalEquivalent (src/gateway/server/ws-connection/handshake-auth-helpers.ts:144) fails only its isLoopbackAddress(remoteAddress) conjunct → pairing locality degrades to remote → shouldSkipLocalBackendSelfPairing (:257) stops exempting the backend gateway-client → requirePairing("not-paired") (message-handler.ts:1561).
• The pairing demand can never be satisfied: in-sandbox device-identity persistence fails, so every run mints a fresh device identity, and the auto-pair watcher only approves cli/openclaw-cli/openclaw-control-ui. This appliance accumulated 10,855 paired entries; the watcher's own CLI polling mints and self-approves a new cli identity every ~13 s.
• Two independent fixes verified live on the appliance, confirming the mechanism from both sides: (a) an OPENSHELL_SANDBOX=1-gated dist patch treating private-range peers as loopback-equivalent in that one locality check (~7.7 s warm turn); (b) the fix(whatsapp): pair login over the loopback gateway and preserve npm plugin provenance #6645-pattern env-boundary fix — launching the gateway daemon with env -u OPENCLAW_GATEWAY_URL (extending nemoclaw-start.sh's existing CLI scrub idiom to its three gateway launch/respawn sites) so the embedded bridge resolves config loopback — which passes the same checks on a pristine dist (~6.5 s) and mints no pairing records at all. Remaining unknown: why in-sandbox device-identity persistence fails — the root of the device-table churn.

Description

What happened: When an agent turn runs inside the sandbox gateway daemon (dashboard chat, POST /v1/chat/completions, gateway-dispatched openclaw agent) and invokes a channel tool (e.g. the message tool → Discord), the embedded tool bridge dials back to the gateway at env OPENCLAW_GATEWAY_URL (ws://10.200.0.2:18789). The gateway rejects that connection with 1008 pairing required: device is not approved yet, so every channel tool call on the warm daemon path fails, and each attempt enqueues a new unapprovable gateway-client pairing request with a fresh device id. The same token authorizes REST POST /tools/invoke-style requests directly with HTTP 200, and the connect frame demonstrably contains that token — the rejection comes from the device-pairing locality gate, not from missing credentials.

What I expected: The daemon's own dial-back, carrying the correct shared gateway token to the sandbox's own gateway address, should be treated like the loopback backend connection it logically is (OpenClaw exempts local backend gateway-client connections with shared-secret auth from device pairing), and the channel tool should execute. Practical impact: delegated channel actions are forced onto a cold openclaw agent --local CLI workaround (~25 s per action) instead of the warm daemon path (~7 s).

Prior art — same failure class, already fixed for CLI contexts: #6645 (WhatsApp login; building on #6413/#4462) fixed this exact locality behavior for exec-context CLI flows — the in-tree comment in scripts/nemoclaw-start.sh states "a private-IP origin makes the gateway's locality check strip operator scopes regardless of token auth", and the merged remedy is to scrub the stashed private veth URL so OpenClaw resolves the loopback gateway from its own config. The gateway daemon itself, however, inherits the private-IP URL in-process from the export OPENCLAW_GATEWAY_URL="ws://${_GATEWAY_WS_HOST}:…" in scripts/nemoclaw-start.sh, so no wrapper-side scrub reaches the embedded tool bridge — it is the remaining unscrubbed consumer. Same locality root, two symptom shapes: device-less CLI connects get scopes stripped; the device-bearing embedded bridge gets an unapprovable pairing demand.

Reproduction Steps

1. NemoClaw appliance with an OpenClaw sandbox and a Discord channel configured and connected (guild + bot ready).
2. Get the operator gateway token on the host: TOK=$(~/.local/bin/nemoclaw <sandbox> gateway-token | tail -1)
3. Trigger any channel tool through the daemon path:

   curl -s -X POST http://127.0.0.1:18789/tools/invoke \
     -H "Authorization: Bearer $TOK" -H "Content-Type: application/json" \
     -d '{"tool":"message","args":{"action":"channel-list","guildId":"<guild-id>","channel":"discord"}}'

4. Observe {"ok":false,"error":{"type":"tool_error","message":"tool execution failed"}}; the in-sandbox /tmp/gateway.log shows a fresh pairing required (requestId …) from a 10.200.0.1 → 10.200.0.2:18789 connection; in-sandbox openclaw devices list --json shows a new pending entry clientId: "gateway-client", clientMode: "backend", scopes: ["operator.write"] with a different deviceId on every attempt.

Any /v1/chat/completions request whose agent turn invokes a channel tool fails the same way.

Environment

• OS: Ubuntu 24.04.4 LTS (Jetson Linux, kernel 6.8.12-1021-tegra, aarch64)
• Hardware: NVIDIA Jetson Orin Nano Developer Kit (Super), 8 GB
• Node.js: host v22.23.1 (nvm); sandbox v22.22.2
• Docker: Docker Engine 29.6.1
• NemoClaw: v0.0.81 (nemoclaw --version; local checkout — repro and evidence gathered on the stock reviewed dist)
• OpenShell: 0.0.72
• OpenClaw: 2026.6.10 (NVIDIA-reviewed dist)
• Sandbox: provider compatible-endpoint (OpenRouter, openai/gpt-oss-120b), policy presets incl. discord; GPU passthrough enabled

Debug Output

═══ System ═══
Linux jetson06 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 GNU/Linux
Mem: 7546 total / Swap: 2047

═══ Docker ═══
CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS                 NAMES
f7f20156ec88   6f44a3c36978   "/opt/openshell/bin/…"   12 hours ago   Up 9 hours (healthy)   openshell-kettlebot-ccb2ddec-…

═══ OpenShell ═══
Gateway: nemoclaw   Server: https://127.0.0.1:8080   Status: Connected   Version: 0.0.72
NAME       CREATED              PHASE
kettlebot  2026-07-19 16:40:15  Ready

═══ Onboard Session ═══
status: complete, mode: non-interactive, provider: compatible-endpoint,
model: openai/gpt-oss-120b, policyPresets: [npm, pypi, huggingface, brew,
brave, openclaw-pricing, discord, teagram], gpuPassthrough: true

═══ Sandbox Internals ═══
root  1   /opt/openshell/bin/openshell-sandbox
sandbox   bash /usr/local/bin/nemoclaw-start (supervisor + watchdog loops)
sandbox   openclaw   (gateway daemon)

### Logs

```shell
# In-sandbox /tmp/gateway.log during one repro attempt (unpatched dist, ANSI stripped):
2026-07-19T20:30:33.959+00:00 gateway connect failed: GatewayClientRequestError: device pairing required (requestId: e3b79ca8-6ac6-4f75-be7a-09a4c5fce65d)
2026-07-19T20:30:33.975+00:00 tools-invoke: tool execution failed: GatewayTransportError: gateway closed (1008): pairing required: device is not approved yet (requestId: e3b79ca8-6ac6-4f75-be7a-09a4c5fce65d)
Gateway target: ws://10.200.0.2:18789
Source: env OPENCLAW_GATEWAY_URL
Config: /sandbox/.openclaw/openclaw.json
[ws] closed before connect conn=30e31d38-543a-4783-9d4a-d16ac11d70db peer=10.200.0.1:34518->10.200.0.2:18789 remote=10.200.0.1 fwd=n/a origin=n/a host=10.200.0.2:18789 ua=n/a code=1008 reason=connect failed

# The re-originated leg's connect frame as captured at the gateway (values
# redacted; token verified byte-identical to gateway.auth.token via SHA-256):
{"method":"connect","params":{"client":{"id":"gateway-client","displayName":"agent","mode":"backend"},
 "auth":{"token":"<43 chars, matches gateway.auth.token>"},"role":"operator","scopes":["operator.write"],
 "device":{"id":"<fresh 64-hex per run>","publicKey":"…","signature":"…"}}}
# Gateway response:
{"ok":false,"error":{"code":"NOT_PAIRED","details":{"reason":"not-paired",
 "requestedRole":"operator","requestedScopes":["operator.write"]}}}

# Ongoing device-table churn (auto-pair watcher, every ~13 s; unrelated to the fix, same persistence root cause):
2026-07-19T20:55:27.260+00:00 [gateway] device pairing auto-approved device=2f4208cf… role=operator
2026-07-19T20:55:40.842+00:00 [gateway] device pairing auto-approved device=54939e4a… role=operator

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[wscurran]

✨ Thanks for the detailed investigation. The root cause in OpenShell's transparent proxy erasing loopback locality is clear. Related PR #7230 appears to address this. Maintainers will review the fix and coordinate with the OpenClaw integration.

---

Related open PRs:

• #6645 fix(whatsapp): pair login over the loopback gateway and preserve npm plugin provenance