[Linux][Inference] nemoclaw sandbox returns 503 when proxying inference requests to local vLLM custom endpoint via inference.local gateway

[lyral-commits]

Description

When a LangChain Deep Agents Code sandbox is onboarded with a local vLLM custom endpoint (e.g. http://localhost:8000/v1), the host-side vLLM server is reachable directly from the host (/v1/models and /v1/chat/completions both return 200), but requests from inside the sandbox via https://inference.local/v1/models return 503. The NemoClaw gateway proxy fails to route inference requests from the sandbox network namespace to the host's localhost endpoint.

Environment

Device:        NVIDIA vm1 (10.34.5.155), Ubuntu 24.04 x86_64 / A100 GPU
OS:            Ubuntu 24.04.4 LTS (kernel 6.8.0-117-generic)
Architecture:  x86_64
OpenShell CLI: 0.0.44
NemoClaw:      v0.0.67
OpenClaw:      deepagents-code 0.1.12
vLLM model:    Qwen/Qwen2.5-1.5B-Instruct

Steps to Reproduce

1. Start vLLM server on host:

   vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000

2. Confirm host-side access works:

   curl -fsS http://localhost:8000/v1/models        # 200 OK

3. Onboard sandbox with custom endpoint:

   export NEMOCLAW_ENDPOINT_URL=http://localhost:8000/v1
   export NEMOCLAW_MODEL=Qwen/Qwen2.5-1.5B-Instruct
   nemoclaw onboard --agent langchain-deepagents-code --non-interactive --name dcode-vllm-local --fresh

4. Run inference from inside sandbox:

   nemoclaw dcode-vllm-local exec -- dcode -n "Reply with the single word VLLM. Nothing else."

Expected Result

Step 4: The NemoClaw gateway correctly proxies the sandbox inference request to the registered custom endpoint (http://localhost:8000/v1). dcode returns "VLLM" with exit 0.

Actual Result

Step 4: Gateway returns 503 when sandbox tries to reach https://inference.local/v1/models. The agent is completely unable to call the model from inside the sandbox.

Root cause: The sandbox runs in a Docker network namespace where localhost resolves to the container's own loopback, not the host's. The NemoClaw gateway proxy does not translate localhost in the registered custom endpoint URL to the host's actual IP/hostname before forwarding from the sandbox network namespace. Host-side vLLM remains healthy throughout.

[sandbox] NET:OPEN [INFO] ALLOWED inference.local:443
[sandbox] routing proxy inference request endpoint=http://localhost:8000/v1 method=GET path=/v1/models
[gateway] ERROR: upstream connect error — connection refused
HTTP 503

---

NVB#6370881

[yimoj]

Triage update — could not reproduce; proposed localhost→host.openshell.internal rewrite is inert on OpenShell 0.0.44

I reproduced this end-to-end through the real NemoClaw CLI on the reporter's exact OpenShell version (0.0.44, Linux, native Docker Engine), onboarding langchain-deepagents-code with NEMOCLAW_ENDPOINT_URL=http://localhost:8000/v1 and a local OpenAI-compatible server on host port 8000, then driving https://inference.local/v1/models from inside the sandbox via nemoclaw <sbx> exec.

Key finding: on the 0.0.44 docker-driver gateway, the inference-forwarding context reaches the host only via the docker-bridge address, never the host loopback, and it resolves localhost, 127.0.0.1, and host.openshell.internal identically. The upstream URL host is therefore irrelevant; only the model server's bind address matters:

local server bind	upstream localhost:8000	upstream 127.0.0.1:8000	upstream host.openshell.internal:8000
0.0.0.0 (vLLM's default vllm serve bind)	200	200	200
127.0.0.1 only	503	503	503

Verified the loopback-only server's access log received none of the gateway-forwarded requests — the gateway never connects to the host's 127.0.0.1, regardless of the registered URL.

Consequences:

• The proposed fix (rewrite a loopback custom endpoint to host.openshell.internal before provider upsert, sharing vllm-local topology) is inert on 0.0.44: localhost and host.openshell.internal behave the same in the gateway's forwarding context.
• For the reporter's exact config — vllm serve ... --port 8000 (default 0.0.0.0 bind) + http://localhost:8000/v1 — https://inference.local/v1/models returns 200, not 503. I could not reproduce the reported 503.
• The only configuration that 503s is a model server bound to loopback only, which is unreachable from the sandbox/gateway path no matter how the URL is written; a URL rewrite cannot fix that.

Recommendation: this does not appear fixable via NemoClaw custom-endpoint URL normalization on the current stack. The 503 is consistent with either (a) the local server bound to 127.0.0.1 only (remedy: bind 0.0.0.0), or (b) an environment-specific Docker networking constraint (rootless/custom bridge, firewall) on the reporter's host. Could the reporter confirm the exact vllm serve --host value and Docker networking mode? Removing my assignment pending that; happy to revisit if a topology where the rewrite actually changes the outcome is identified (e.g. a cluster-driver/k3s pod gateway, or Docker-Desktop where container↔host-loopback differs).