fix(rebuild): recover managed MCP after exec loss

docs/manage-sandboxes/recover-rebuild-sandboxes.mdx

@@ -168,6 +168,18 @@ If every state directory fails, NemoClaw stops before deleting the original sand
 `rebuild --force` can continue when no state directory was preserved or a manifest-declared state file failed.
 NemoClaw restores any entries captured in the partial backup; if nothing usable was captured, it recreates the sandbox from recorded registry metadata without restoring prior sandbox state.
 Use this recovery path only when losing the state that could not be backed up is acceptable.
+When a sandbox with managed MCP servers cannot run a pre-mutation no-op, explicit `--force` uses its complete registry entries plus the exact live generated policies and provider identities to preserve MCP intent without scrubbing the unreachable in-sandbox adapter.
+Every bridge entry must record an explicit adapter that matches the sandbox's recorded agent, and the registered policy must be the canonical generated policy for that adapter, server name, URL endpoint, and current resolved-address pins.
+NemoClaw rechecks that read-only snapshot immediately before deletion and stops if the target, registry, policy, provider, or recorded gateway changed.
+NemoClaw sends the delete request and every deletion-confirmation lookup to the sandbox's exact recorded gateway.
+Across every rebuild path, NemoClaw does not attempt to stop the local NIM through the delete attempt, and cleanup is attempted on a best-effort basis only after deletion is positively confirmed.
+After a nonzero delete, an explicit missing result converges as deleted.
+A `Ready` or `Running` result triggers an attempt to restore prepared MCP state and any shields lockdown that rebuild temporarily opened.
+NemoClaw reports any MCP or shields restoration failure and does not present the operation as a successful rollback.
+Any partial or unreachable result remains ambiguous.
+NemoClaw preserves the MCP ownership and rebuild-recovery records, does not attempt to stop NIM, skips the rebuild process's immediate shields relock, and does not claim that the original sandbox is intact.
+Inspect the live sandbox and gateway state before retrying recovery.
+This recovery also stops for incomplete MCP adds or ambiguous ownership; an error after a successful no-op does not fall back to the host-side path.
 </Warning>
 
 When rebuild starts with shields up, NemoClaw opens a 30-minute shields-down window for backup and recreation.

docs/reference/commands.mdx

@@ -2616,7 +2616,7 @@ $$nemoclaw my-assistant rebuild [--yes|-y|--force] [--verbose|-v] [--tool-disclo
 | Flag | Description |
 |------|-------------|
 | `--yes`, `-y` | Skip the confirmation prompt. |
-| `--force` | Skip the confirmation prompt and continue when no state directory was preserved or a manifest-declared state file failed. NemoClaw restores any captured entries; after a total failure, it recreates from registry metadata only. |
+| `--force` | Skip the confirmation prompt and continue when no state directory was preserved or a manifest-declared state file failed. NemoClaw restores any captured entries; after a total failure, it recreates from registry metadata only. If a pre-mutation no-op cannot execute in a sandbox with managed MCP servers, it may preserve the exact registered MCP intent through host-side recovery. |
 | `--verbose`, `-v` | Log SSH commands, exit codes, and session state (also enabled by `NEMOCLAW_REBUILD_VERBOSE=1`) |
 | `--tool-disclosure <progressive\|direct>` | Change the model-visible tool catalog during this transactional rebuild. Use this path for sandboxes with managed MCP servers so their providers and adapter state are preserved. |
 | `--dcode-auto-approval <disabled\|thread-opt-in>` | Change the managed Deep Agents Code thread auto-approval capability. `thread-opt-in` is accepted only for managed Deep Agents Code sandboxes and is rejected for other agents or custom images. Enabling prints a warning, and either value requires sandbox recreation. |
@@ -2632,6 +2632,19 @@ If every state directory fails, `rebuild` exits before destroying the original s
 With `--force`, NemoClaw preserves any captured loose files in the partial manifest and restores them after recreation.
 If the backup produced nothing usable, it continues from recorded registry metadata without restoring prior sandbox state.
 Use this recovery path only when losing the state that could not be backed up is acceptable.
+For a sandbox with managed MCP servers, `--force` probes sandbox execution before MCP teardown.
+If that no-op cannot run, NemoClaw requires complete bridge entries and exact live policy and provider identities, without trying an in-sandbox adapter scrub or changing MCP ownership state.
+Each bridge must carry an explicit adapter matching the sandbox's recorded agent, and the registered policy must equal the canonical generated policy for that adapter, server name, URL endpoint, and current resolved-address pins.
+It rechecks the registry, recorded gateway, resolved targets, live generated policies, and provider identities immediately before deletion; incomplete adds, drift, or ambiguous ownership stop before deletion.
+NemoClaw sends the delete request and every deletion-confirmation lookup to the sandbox's exact recorded gateway.
+Across every rebuild path, NemoClaw does not attempt to stop local NIM until sandbox deletion is positively confirmed, then attempts NIM cleanup on a best-effort basis.
+When `openshell sandbox delete` exits nonzero, an exact recorded-gateway lookup distinguishes explicit absence from a confirmed `Ready` or `Running` sandbox.
+Any other phase or probe failure is ambiguous.
+Explicit absence continues the rebuild.
+Confirmed intact state triggers an attempt to restore prepared MCP state and any shields lockdown that rebuild temporarily opened.
+NemoClaw reports any MCP or shields restoration failure and does not present the operation as a successful rollback.
+Ambiguous state preserves MCP ownership and recovery metadata without attempting to stop NIM or claiming the original sandbox remains intact, and the rebuild process skips its immediate shields relock.
+Failures after a successful exec probe do not switch to the host-side path.
 Before backup or deletion, `rebuild` also refuses an incomplete MCP destroy transaction.
 It also refuses a pending baseline exclusion transaction before opening a shields-down window, starting backup, or deleting the sandbox, and prints the exact `policy exclude` or `policy restore` command to rerun.
 For a prepared-only transaction, the redacted diagnostic points to `$$nemoclaw <name> mcp remove <server> --force` when the sandbox is still live.

src/lib/actions/sandbox/gateway-state-owning-gateway.test.ts

@@ -55,6 +55,23 @@ describe("getReconciledSandboxGatewayState owning-gateway guard", () => {
     );
   });
 
+  it("classifies the owner-scoped Internal no-spec response as missing", () => {
+    vi.spyOn(gatewayDrift, "detectOpenShellStateRpcPreflightIssue").mockReturnValue(null);
+    vi.spyOn(gatewayDrift, "detectOpenShellStateRpcResultIssue").mockReturnValue(null);
+    const capture = vi.spyOn(openshellRuntime, "captureOpenshell").mockReturnValue({
+      status: 1,
+      output: 'status: Internal, message: "sandbox has no spec"',
+    } as never);
+
+    expect(getSandboxGatewayState("beta", "nemoclaw-8091")).toMatchObject({
+      state: "missing",
+    });
+    expect(capture).toHaveBeenCalledWith(
+      ["sandbox", "get", "-g", "nemoclaw-8091", "beta"],
+      expect.anything(),
+    );
+  });
+
   it("pins the async status RPC to the recorded owner", async () => {
     vi.spyOn(gatewayDrift, "detectOpenShellStateRpcPreflightIssue").mockReturnValue(null);
     vi.spyOn(gatewayDrift, "detectOpenShellStateRpcResultIssue").mockReturnValue(null);

src/lib/actions/sandbox/gateway-state.ts

@@ -94,6 +94,38 @@ function gatewayEndpointOverrideState(): SandboxGatewayState | null {
   }
 }
 
+/** Canonical OpenShell response classifier for an absent sandbox record. */
+export function isMissingSandboxGatewayOutput(output = ""): boolean {
+  return /\bNotFound\b|\bNot Found\b|sandbox not found|sandbox has no spec/i.test(
+    stripAnsi(String(output)),
+  );
+}
+
+/**
+ * Strict absence classifier for destructive owner-gateway reconciliation.
+ * Bare NotFound is not sufficient because OpenShell uses it for missing
+ * gateways and providers as well as sandboxes.
+ */
+export function isExplicitMissingSandboxGatewayOutput(
+  output: string,
+  sandboxName: string,
+): boolean {
+  const clean = stripAnsi(String(output)).replace(/\r/g, "").trim();
+  const exactNoSpec =
+    /^(?:error:\s*)?status:\s*Internal,\s*message:\s*["']sandbox has no spec["'](?:,\s*details:\s*\[\])?(?:,\s*metadata:\s*MetadataMap\s*\{\s*\})?$/i;
+  if (exactNoSpec.test(clean)) return true;
+
+  const escapedName = sandboxName.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+  const namedSandbox = `(?:['\"]${escapedName}['\"]|${escapedName})`;
+  return (
+    new RegExp(
+      `^(?:error:\\s*)?sandbox\\s+${namedSandbox}\\s+(?:(?:is\\s+)?not\\s+(?:found|present)|does\\s+not\\s+exist)[.!]?$`,
+      "i",
+    ).test(clean) ||
+    new RegExp(`^(?:error:\\s*)?no\\s+such\\s+sandbox\\s+${namedSandbox}[.!]?$`, "i").test(clean)
+  );
+}
+
 function formatGatewaySchemaMismatchOutput(
   issue: OpenShellStateRpcIssue,
   action: string,
@@ -183,7 +215,7 @@ export function getSandboxGatewayState(
   // sibling; an owner-scoped lookup means the sandbox is genuinely absent
   // from its recorded gateway. Both remain `missing`, and reconciliation uses
   // the presence of the explicit owner pin to distinguish those cases.
-  if (/\bNotFound\b|\bNot Found\b|sandbox not found|sandbox has no spec/i.test(output)) {
+  if (isMissingSandboxGatewayOutput(output)) {
     return { state: "missing", output };
   }
   if (
@@ -250,7 +282,7 @@ export async function getSandboxGatewayStateForStatus(
     }
     return { state: "present", output };
   }
-  if (/\bNotFound\b|\bNot Found\b|sandbox not found|sandbox has no spec/i.test(output)) {
+  if (isMissingSandboxGatewayOutput(output)) {
     return { state: "missing", output };
   }
   if (

src/lib/actions/sandbox/mcp-bridge-policy.test.ts

@@ -5,15 +5,31 @@ import { afterEach, describe, expect, it, vi } from "vitest";
 import YAML from "yaml";
 
 import * as policies from "../../policy";
+import type { McpBridgeEntry } from "../../state/registry";
 import * as registry from "../../state/registry";
 import {
   buildMcpBridgePolicyName,
   buildMcpBridgePolicyYaml,
   buildMcpBridgeProviderName,
   MCP_BRIDGE_ALLOWED_METHODS,
   MCP_BRIDGE_POLICY_MAX_BODY_BYTES,
+  MCP_BRIDGE_POLICY_SOURCE,
 } from "./mcp-bridge";
-import { applyGeneratedPolicy } from "./mcp-bridge-policy";
+import { applyGeneratedPolicy, assertGeneratedPolicyExactReadOnly } from "./mcp-bridge-policy";
+
+function githubBridgeEntry(overrides: Partial<McpBridgeEntry> = {}): McpBridgeEntry {
+  return {
+    server: "github",
+    agent: "openclaw",
+    adapter: "mcporter",
+    url: "https://api.githubcopilot.com/mcp",
+    env: ["GITHUB_MCP_TOKEN"],
+    providerName: "alpha-mcp-github-0123456789abcdef",
+    policyName: "mcp-bridge-github",
+    addedAt: "2026-06-01T00:00:00.000Z",
+    ...overrides,
+  };
+}
 
 describe("MCP OpenShell policy", () => {
   afterEach(() => {
@@ -136,6 +152,143 @@ describe("MCP OpenShell policy", () => {
     });
   });
 
+  it("accepts only the canonical generated policy for the exact bridge and DNS pins", () => {
+    const entry = githubBridgeEntry();
+    const pins = ["8.8.8.8", "2606:4700:4700::1111"];
+    const content = buildMcpBridgePolicyYaml(entry.server, entry.url, "mcporter", pins);
+    const registration = {
+      name: entry.policyName,
+      content,
+      sourcePath: MCP_BRIDGE_POLICY_SOURCE,
+    };
+    vi.spyOn(registry, "getCustomPolicies").mockReturnValue([registration]);
+    const gatewayState = vi
+      .spyOn(policies, "getPresetContentGatewayState")
+      .mockReturnValue("match");
+
+    expect(assertGeneratedPolicyExactReadOnly("alpha", entry, "mcporter", pins)).toEqual(
+      registration,
+    );
+    expect(gatewayState).toHaveBeenCalledWith("alpha", content);
+  });
+
+  it.each([
+    "owned-first",
+    "unowned-first",
+  ])("rejects duplicate same-name ownership records regardless of order (%s)", (order) => {
+    const entry = githubBridgeEntry();
+    const pins = ["8.8.8.8"];
+    const content = buildMcpBridgePolicyYaml(entry.server, entry.url, "mcporter", pins);
+    const owned = {
+      name: entry.policyName,
+      content,
+      sourcePath: MCP_BRIDGE_POLICY_SOURCE,
+    };
+    const unowned = {
+      name: entry.policyName,
+      content: `${content}\n# conflicting duplicate`,
+      sourcePath: "/tmp/operator-policy.yaml",
+    };
+    vi.spyOn(registry, "getCustomPolicies").mockReturnValue(
+      order === "owned-first" ? [owned, unowned] : [unowned, owned],
+    );
+    const gatewayState = vi.spyOn(policies, "getPresetContentGatewayState");
+
+    expect(() => assertGeneratedPolicyExactReadOnly("alpha", entry, "mcporter", pins)).toThrow(
+      /ownership is missing or ambiguous/,
+    );
+    expect(gatewayState).not.toHaveBeenCalled();
+  });
+
+  it("rejects individually valid policy records that disagree with their bridge definition", () => {
+    const entry = githubBridgeEntry();
+    const pins = ["8.8.8.8"];
+    const canonical = buildMcpBridgePolicyYaml(entry.server, entry.url, "mcporter", pins);
+    const wrongKeyDocument = YAML.parse(canonical) as {
+      network_policies: Record<string, { name: string }>;
+    };
+    wrongKeyDocument.network_policies.mcp_bridge_other = {
+      ...wrongKeyDocument.network_policies.mcp_bridge_github,
+      name: "mcp_bridge_other",
+    };
+    delete wrongKeyDocument.network_policies.mcp_bridge_github;
+
+    const mismatches: Array<{
+      label: string;
+      candidateEntry?: McpBridgeEntry;
+      candidateName?: string;
+      content: string;
+    }> = [
+      {
+        label: "host",
+        content: buildMcpBridgePolicyYaml(
+          entry.server,
+          "https://mcp.example.test/mcp",
+          "mcporter",
+          pins,
+        ),
+      },
+      {
+        label: "path",
+        content: buildMcpBridgePolicyYaml(
+          entry.server,
+          "https://api.githubcopilot.com/other",
+          "mcporter",
+          pins,
+        ),
+      },
+      {
+        label: "adapter",
+        content: buildMcpBridgePolicyYaml(entry.server, entry.url, "hermes-config", pins),
+      },
+      { label: "network policy key", content: YAML.stringify(wrongKeyDocument) },
+      {
+        label: "resolved address pins",
+        content: buildMcpBridgePolicyYaml(entry.server, entry.url, "mcporter", ["1.1.1.1"]),
+      },
+      {
+        label: "policy name",
+        candidateEntry: githubBridgeEntry({ policyName: "mcp-bridge-other" }),
+        candidateName: "mcp-bridge-other",
+        content: canonical,
+      },
+    ];
+
+    for (const mismatch of mismatches) {
+      vi.restoreAllMocks();
+      const candidateEntry = mismatch.candidateEntry ?? entry;
+      vi.spyOn(registry, "getCustomPolicies").mockReturnValue([
+        {
+          name: mismatch.candidateName ?? candidateEntry.policyName,
+          content: mismatch.content,
+          sourcePath: MCP_BRIDGE_POLICY_SOURCE,
+        },
+      ]);
+      const gatewayState = vi.spyOn(policies, "getPresetContentGatewayState");
+
+      expect(
+        () => assertGeneratedPolicyExactReadOnly("alpha", candidateEntry, "mcporter", pins),
+        mismatch.label,
+      ).toThrow(/not canonical for its recorded bridge definition/);
+      expect(gatewayState, mismatch.label).not.toHaveBeenCalled();
+    }
+  });
+
+  it("does not expose malformed persisted URLs in canonical ownership errors", () => {
+    const secret = `nvapi-${"a".repeat(32)}`;
+    const entry = githubBridgeEntry({ url: `not-a-url-${secret}` });
+    vi.spyOn(registry, "getCustomPolicies").mockReturnValue([]);
+
+    let message = "";
+    try {
+      assertGeneratedPolicyExactReadOnly("alpha", entry, "mcporter", ["8.8.8.8"]);
+    } catch (error) {
+      message = error instanceof Error ? error.message : String(error);
+    }
+    expect(message).toContain("not canonical for its recorded bridge definition");
+    expect(message).not.toContain(secret);
+  });
+
   it("pins the current OpenShell main client-to-server MCP method profile", () => {
     expect(MCP_BRIDGE_ALLOWED_METHODS).toEqual([
       "initialize",