Skip to content

feat(runtime): add native Podman support for all agents - #7872

Closed
ericksoa wants to merge 46 commits into
mainfrom
feat/native-podman-runtime
Closed

feat(runtime): add native Podman support for all agents#7872
ericksoa wants to merge 46 commits into
mainfrom
feat/native-podman-runtime

Conversation

@ericksoa

@ericksoa ericksoa commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds native Podman support for the same buildless managed-image path used by OpenClaw, Hermes, and LangChain Deep Agents Code across the existing supported host and architecture matrix. Podman must preserve sandbox GPU access and host-local Ollama, NIM, and vLLM through inference.local, including GPU-backed local inference; CPU-only or hosted-inference-only support is not an acceptable milestone. Podman lifecycle behavior stays behind the pluggable runtime/managed-bootstrap contracts established earlier in the stack; the core onboarding state machine does not depend on Docker compatibility behavior or an agent-specific path.

PR 4 of #7744. Based on #7772 and intended to merge after #7747, #7756, and #7772.\n\nCurrent draft gap: the pushed head is not the accepted architecture. It still depends on post-start privileged Podman exec, duplicates bootstrap orchestration instead of implementing the shared adapter, contains a fake socket-based MXC proof, and lacks exact-head functional/GPU/local-inference/platform evidence. This draft will not be marked ready or labeled green until those paths are replaced rather than waived.

Related Issue

Part of #7744

Changes

  • Implement native Podman preflight, socket authority, exact managed-image pull/identity, held-workload discovery, launch, readiness, lifecycle actions, rollback, rebuild, snapshot/restore, and cleanup behind the shared runtime boundary.
  • Preserve the same managed startup profile, immutable image/spec receipts, nonroot agent identity, and fail-closed recovery behavior for OpenClaw, Hermes, and DCode.
  • Handle Podman state, user namespaces, networking, mounts, runtime discovery, ownership, and the existing supported host/architecture matrix explicitly, including linux/amd64, linux/arm64/DGX, macOS Podman Machine, WSL where viable, and rootful/rootless operation. Any genuine platform blocker must be surfaced for maintainer decision rather than silently excluded.
  • Preserve sandbox GPU attachment and host-local Ollama, NIM, and vLLM routing through inference.local, including GPU-backed local inference, across OpenClaw, Hermes, and DCode.
  • Extend installer/runtime diagnostics and documentation for Podman without changing the Docker path.
  • Add exact-head all-agent Podman contract, workflow, and live E2E coverage with Docker unavailable and PR-candidate image digests: real inference turns, supported messaging smoke, nonroot identity, rollback failure injection, user-namespace, GPU, local-inference, network, mount/SELinux, lifecycle, and cleanup parity. The lane cannot depend on a later release tag.
  • Add a genuinely socket-free MXC fixture implementing all six managed-bootstrap operations for OpenClaw, Hermes, and DCode; a fake Unix socket or narrower patch hook is not sufficient.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Exact-head review is pending after the final guard-fix commit.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: pending final exact-head review
  • Evidence:
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every committed change uses signed commits
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above
  • Applicable broad gate passed
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

This PR is draft only while the final guard and full-scope runtime work is completed and pushed. It must remain draft and must not claim Podman support until exact-head CI and protected all-agent Podman E2E are green with exact candidate image digests for the supported host/architecture matrix, full lifecycle and launch-spec parity, actual inference/messaging operation, nonroot identity, rollback, sandbox GPU attachment, and host-local Ollama/NIM/vLLM through inference.local, including GPU-backed local inference.


Signed-off-by: Aaron Erickson aerickson@nvidia.com

ericksoa and others added 30 commits July 28, 2026 11:10
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
ericksoa added 14 commits July 29, 2026 15:26
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
# Conflicts:
#	agents/hermes/Dockerfile
#	ci/test-file-size-budget.json
#	test/hermes-final-image-layout.test.ts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Classify the remaining Hermes integrity pins as build-only inputs.

Export only bounded, redacted OpenShell diagnostics from managed-image validation failures.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 30, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7c772b84-fd72-4eeb-92ab-e3e9afdb813e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 405179c in the feat/native-podman-r... branch remains at 96%, unchanged from commit 4dcb89e in the feat/buildless-manag... branch.


Updated July 30, 2026 06:55 UTC

OPENSHELL_SERVER_PORT: "8443",
OPENSHELL_SUPERVISOR_IMAGE: "ghcr.io/nvidia/openshell/supervisor@sha256:def",
});
expect(fs.readFileSync(configPath)).toEqual(initialConfig);
if (!stat.isFile() || stat.isSymbolicLink()) {
throw new Error("not a regular file");
}
config = fs.readFileSync(configPath);
it("provides the synchronous sleep contract required by runtime lifecycle polling", () => {
const wait = vi.spyOn(Atomics, "wait").mockReturnValue("timed-out");

expect(sleepSeconds(0.25)).toBeUndefined();
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 11 warnings · 1 suggestion

Nemotron output stays in workflow artifacts and does not change the assessment above.

Since last review: 0 prior items resolved · 0 still apply · 0 new items found

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, full-e2e, hermes-e2e, security-posture, channels-add-remove, channels-stop-start, inference-routing, network-policy, onboard-repair, onboard-resume, state-backup-restore, upgrade-stale-sandbox, ubuntu-repo-cloud-langchain-deepagents-code, onboard-managed-image-buildless-e2e

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@jyaunches

jyaunches commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Architecture review — PR 4

Review scope: PR SHA 405179c7b0931a1c0818c31e4489e98c4f74103f, base SHA 3f29187d10304d70f6b506d58b92ac4c93e5ac28.

This is a point-in-time review. The preceding stack PR, #7772, is now at 64dc55a5c83914beda9d10ae4493ef703fa91313, so this PR should be restacked and the architecture review rerun against the resulting diff.

Overall assessment

This PR is a substantial implementation with a strong core direction. It adds native Podman support without introducing agent-specific branches, reuses several established lifecycle concepts, and handles the difficult container-recreation details with care. The separation between compute-driver identity and gateway ownership is especially useful groundwork for future drivers.

The remaining architectural work is primarily consolidation rather than a rewrite: preserve the low-level Podman implementation, then place its transaction flow behind the shared bootstrap contract accepted in #7744. That would make the new capability feel native to the codebase while keeping the Podman-specific mechanics appropriately isolated.

Strong foundations already in place

  • OpenShellComputePlan provides a clean separation between compute-driver identity and gateway ownership.
  • The production Podman modules remain agent-independent, which supports OpenClaw, Hermes, and future agents without parallel implementations.
  • Start, stop, doctor, snapshot, cleanup, and workload operations are routed through existing adapter concepts rather than implemented as a separate product stack.
  • The Podman recreation code validates full container IDs, socket authority, deterministic create arguments, unsupported container shapes, and rollback state. These are thoughtful safeguards around the riskiest part of the implementation.
  • The managed-image and startup-profile representations are shared instead of creating Podman-only equivalents.
  • The implementation aims to reuse the existing NIM/vLLM workflows, which is the right architectural direction even though the engine-selection seam can be made more explicit.
  • The code already exposes many of the components needed for a coherent driver boundary; the next step is to register and coordinate them as one unit.

These choices provide a solid base to build on.

Opportunities to strengthen the architecture

  1. Converge bootstrap orchestration on the accepted shared contract.

    Issue [Epic] Support native Podman with buildless managed onboarding #7744 specifies a six-operation ManagedBootstrapAdapter: createHeldWorkload, discoverHeldWorkload, inspectHeldWorkload, replaceForBootstrap, awaitBootstrap, and finalizeBootstrap. The current shared seam is an eight-hook SandboxCreateRuntimePatch, while the Podman implementation coordinates recreation, root-level application, shared state, supervisor reconnection, and rollback.

    The existing Podman primitives can become the first complete implementation behind the shared adapter. Moving transaction ordering into one core bootstrap coordinator would let onboarding, rebuild, and snapshot restore reuse the same flow and give future drivers a clear contract to implement.

  2. Move the final bootstrap transition into the managed image.

    podman-root-apply.ts currently applies the startup plan with podman exec --interactive --user 0:0. This works as an implementation bridge, but it leaves the host coordinator responsible for an engine-specific privileged mutation after container start.

    Letting the managed startup manifest/trampoline own this transition, followed by an exact supervisor handoff, would simplify the host-side adapter and keep startup semantics consistent across drivers.

  3. Persist mutation authority across process interruptions.

    createPodmanSandboxCreatePatch retains recreation, startup, apply, finalize, and error state in closure-local variables. Discovery uses a generated container name and the openshell.managed / openshell.sandbox-name labels in sandbox-recreate.ts.

    The existing full-ID validation and rollback work provide a good basis for durable recovery. Persisting the bootstrap identity, plan fingerprint, full old and replacement IDs, phase, and rollback target before the first destructive mutation would extend those protections across process interruption and resume.

  4. Register each driver through one coherent bundle.

    Podman currently participates in separate registration points for planning, capabilities, gateway profiles, lifecycle, recovery, doctor, workload, authority, create patch, managed-startup requirements, snapshot, cleanup, and host-local inference. The individual seams are useful, but a future driver could easily omit one.

    A single driver bundle keyed by driver identity could compose these existing pieces and validate its required operations at registration. The MXC test in driver-pluggability-contract.test.ts is a useful starting point; extending it to exercise a socket-free fixture through all six bootstrap operations, including commit and rollback, would demonstrate the boundary end to end.

  5. Make container-engine selection explicit.

    HostContainerEngineCommand enables reuse by changing a process-global active command and allowing helpers named dockerRun, dockerInspect, and similar to execute Podman with argument translation.

    Reusing the established NIM/vLLM workflows avoids valuable duplication. Injecting a typed ContainerEngine dependency into those workflows would retain that reuse while making engine choice visible, improving test isolation, and keeping Docker and Podman argument construction in their respective implementations.

  6. Use one capability record for the supported surface.

    The runtime registry rejects sandbox GPU for this path in registry.ts. The PR's E2E configuration uses hosted inference with sandbox GPU disabled in e2e.yaml, and the platform matrix records Linux amd64 CPU/hosted coverage in platform-matrix.json.

    The current Linux amd64 hosted-inference path is a concrete, testable first slice. One capability record could drive preflight, runtime behavior, documentation, and E2E selection while the broader platform, direct sandbox GPU, and local Ollama/NIM/vLLM matrix from [Epic] Support native Podman with buildless managed onboarding #7744 is implemented or refined through a maintainer decision.

  7. Keep the top-level onboarding coordinator driver-neutral.

    src/lib/onboard.ts grows substantially and directly constructs Podman gateway/watcher authority. The architecture-size check is giving useful feedback here: driver-specific coordination is accumulating at the top-level entrypoint.

    Extracting driver resolution and bootstrap coordination behind the shared interfaces would leave onboard.ts with a simpler responsibility: select a plan and invoke the coordinator. The existing Podman modules can retain the detailed engine behavior.

Recommended target shape

onboarding / rebuild / snapshot restore
                  |
       shared bootstrap coordinator
                  |
 ManagedBootstrapAdapter<Handle, Snapshot>
      /             |              \
 OpenShell        Podman            MXC fixture
                  |
     one registered driver bundle
(plan, capabilities, lifecycle, workload, authority,
 bootstrap, snapshot, cleanup, container engine)

The Podman-specific code should remain responsible for Podman inspection, deterministic replacement arguments, exact identity checks, and engine calls. Core code should remain responsible for transaction ordering, persistent phases, rollback policy, and agent-independent startup semantics.

Suggested completion criteria

  • Restack this PR onto the current feat(onboard): add managed image selection contracts #7772 head and rerun the review against the new PR SHA.
  • Converge the create-stream patch orchestration on the accepted six-operation bootstrap adapter, or record a superseding accepted design decision.
  • Add exact-head behavior tests for commit, rollback, process interruption/resume, identity mismatch, and a real socket-free driver fixture.
  • Align the supported platform/inference matrix with preflight, runtime behavior, and E2E evidence.
  • Restore the architecture-size check by moving driver-specific orchestration out of src/lib/onboard.ts.

This PR already provides a strong Podman foundation. With the shared coordinator, durable transaction record, explicit engine dependency, and unified driver bundle, it can also provide a clean template for future compute drivers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants