Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ metrics = "0.24"
metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] }

# Unix/Process
nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term"] }
nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term", "net"] }
rustix = { version = "1.1", features = ["process"] }

# Serialization
Expand Down
2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ when a sandbox create request asks for GPU resources.
| Runtime | Best fit | Sandbox boundary | Notes |
|---|---|---|---|
| Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, CDI GPU devices when available, and bindability-checked callback listeners for direct-host loopback gateway binds. Bridge-mode gateways can listen on the Podman bridge IP when bindable; rootless pasta can add a protected non-loopback local-interface listener when TLS client certificate authentication is enabled. Containerized gateways skip host listener discovery and require explicit port publishing. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
Expand Down
28 changes: 27 additions & 1 deletion crates/openshell-driver-podman/NETWORKING.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,29 @@ if config.grpc_endpoint.is_empty() {
The bridge gateway IP is not a stable substitute in rootless mode because it
can live inside the user namespace rather than on the host.

### Rootless pasta callback listener

When the gateway binds only to loopback, host-local clients can reach
`127.0.0.1:<port>`, but sandbox containers do not share the host loopback
namespace. In rootless Podman with pasta, `host.containers.internal` reaches the
host through pasta's forwarding path and may not connect to a listener bound
only to `127.0.0.1`.

For direct host gateways, the Podman driver detects rootless pasta from Podman
system info (`rootless=true` and `rootlessNetworkCmd=pasta`). If the gateway
bind address is loopback-only, TLS client certificate authentication is active,
and auto-discovery is enabled, the driver adds a second callback listener on a
bindable non-loopback local interface address. Candidate addresses come from
local interface enumeration via `getifaddrs`; the driver does not probe an
external IP to find this address. The supervisor endpoint remains
`host.containers.internal:<port>`.

Set `enable_auto_callback_listener = false` in `[openshell.drivers.podman]` or
`OPENSHELL_PODMAN_ENABLE_AUTO_CALLBACK_LISTENER=false` to disable the automatic
listener. Containerized gateways skip host listener discovery because their
network namespace is not the host namespace; publish the gateway container port
on an address sandbox containers can reach.

### Layer 3 Inner Sandbox Network Namespace

Inside the container, the supervisor creates another network namespace for the
Expand Down Expand Up @@ -391,7 +414,10 @@ Gateway

The gateway binds to `127.0.0.1:17670` by default in the RPM packaging. Client
certificates are auto-generated by `openshell-gateway generate-certs` on first
start and bind-mounted into sandbox containers by the Podman driver.
start and bind-mounted into sandbox containers by the Podman driver. In
rootless pasta mode, the protected callback listener described above allows the
supervisor to keep using `host.containers.internal:<port>` while the primary
listener remains loopback-only.

## Differences from the Kubernetes Driver

Expand Down
6 changes: 6 additions & 0 deletions crates/openshell-driver-podman/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,11 @@ Key points:
host. Linux defaults to Podman's `host-gateway` resolver. macOS Podman
machine defaults to gvproxy's host-loopback IP, `192.168.127.254`, because
stale Podman machines may fail to resolve `host-gateway`.
- Callback listener: direct host gateways that bind to loopback can add a
protected non-loopback listener for rootless pasta callbacks when TLS client
certificate authentication is enabled. Disable this with
`enable_auto_callback_listener = false`. Gateway containers skip host
listener discovery and require explicit port publishing.
- nsenter: the supervisor uses `nsenter --net=` instead of `ip netns exec` for
namespace operations, avoiding the sysfs remount path that fails in rootless
containers.
Expand Down Expand Up @@ -338,6 +343,7 @@ Podman resources after out-of-band container removal or label drift.
| `OPENSHELL_GATEWAY_PORT` | `--gateway-port` | `17670` | Gateway port used for endpoint auto-detection by the standalone binary. |
| `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name. |
| `OPENSHELL_PODMAN_HOST_GATEWAY_IP` | `--host-gateway-ip` | empty on Linux, `192.168.127.254` on macOS | Host gateway IP used for sandbox host aliases. Empty uses Podman's `host-gateway` resolver. |
| `OPENSHELL_PODMAN_ENABLE_AUTO_CALLBACK_LISTENER` | none | `true` | Enable the protected non-loopback callback listener for rootless pasta when the gateway is loopback-bound and TLS client certificate authentication is enabled. |
| `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Supervisor Unix socket path in `PodmanComputeConfig`. |
| `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `10` | Container stop timeout in seconds. |
| `OPENSHELL_SANDBOX_PIDS_LIMIT` | `--sandbox-pids-limit` | `2048` | Podman cgroup PID limit for sandbox containers. Set `0` to inherit Podman's runtime/default PID limit. |
Expand Down
2 changes: 2 additions & 0 deletions crates/openshell-driver-podman/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,8 @@ pub struct HostInfo {
#[serde(default)]
pub network_backend: String,
#[serde(default)]
pub rootless_network_cmd: String,
#[serde(default)]
pub security: SecurityInfo,
}

Expand Down
35 changes: 34 additions & 1 deletion crates/openshell-driver-podman/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// SPDX-License-Identifier: Apache-2.0

use openshell_core::config::{DEFAULT_STOP_TIMEOUT_SECS, DEFAULT_SUPERVISOR_IMAGE};
use std::net::IpAddr;
use std::net::{IpAddr, SocketAddr};
use std::path::PathBuf;
use std::str::FromStr;

Expand Down Expand Up @@ -68,6 +68,7 @@ impl FromStr for ImagePullPolicy {

#[derive(Clone, serde::Serialize, serde::Deserialize)]
#[serde(default, deny_unknown_fields)]
#[allow(clippy::struct_excessive_bools)]
pub struct PodmanComputeConfig {
/// Path to the Podman API Unix socket.
/// Default: `$XDG_RUNTIME_DIR/podman/podman.sock` (Linux),
Expand All @@ -89,6 +90,15 @@ pub struct PodmanComputeConfig {
/// so the correct port is used even when `--port` differs from the
/// default. Defaults to [`openshell_core::config::DEFAULT_SERVER_PORT`].
pub gateway_port: u16,
/// Full gateway bind address supplied by the gateway process at runtime.
#[serde(skip)]
pub gateway_bind_address: Option<SocketAddr>,
/// Whether the gateway has TLS enabled. Supplied by the gateway process.
#[serde(skip)]
pub gateway_tls_enabled: bool,
/// Whether gateway callbacks are protected by listener-level client certificate auth.
#[serde(skip)]
pub gateway_callback_client_cert_auth_enabled: bool,
/// Unix socket path the in-container supervisor bridges relay traffic to.
pub sandbox_ssh_socket_path: String,
/// Name of the Podman bridge network.
Expand Down Expand Up @@ -134,10 +144,19 @@ pub struct PodmanComputeConfig {
/// Set to `0` to disable health checks entirely.
/// Defaults to [`DEFAULT_HEALTH_CHECK_INTERVAL_SECS`] (10 seconds).
pub health_check_interval_secs: u64,
/// Automatically add a non-loopback callback listener for rootless Podman
/// pasta when the primary gateway bind is loopback-only and TLS client
/// certificate authentication is active.
#[serde(default = "default_true")]
pub enable_auto_callback_listener: bool,
}

pub const DEFAULT_HEALTH_CHECK_INTERVAL_SECS: u64 = 10;

fn default_true() -> bool {
true
}

impl PodmanComputeConfig {
/// Returns `true` when all three TLS paths are configured.
#[must_use]
Expand Down Expand Up @@ -251,6 +270,9 @@ impl Default for PodmanComputeConfig {
image_pull_policy: ImagePullPolicy::default(),
grpc_endpoint: String::new(),
gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
gateway_bind_address: None,
gateway_tls_enabled: false,
gateway_callback_client_cert_auth_enabled: false,
sandbox_ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
network_name: DEFAULT_NETWORK_NAME.to_string(),
host_gateway_ip: Self::default_host_gateway_ip(),
Expand All @@ -262,6 +284,7 @@ impl Default for PodmanComputeConfig {
sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT,
enable_bind_mounts: false,
health_check_interval_secs: DEFAULT_HEALTH_CHECK_INTERVAL_SECS,
enable_auto_callback_listener: true,
}
}
}
Expand All @@ -274,6 +297,12 @@ impl std::fmt::Debug for PodmanComputeConfig {
.field("image_pull_policy", &self.image_pull_policy.as_str())
.field("grpc_endpoint", &self.grpc_endpoint)
.field("gateway_port", &self.gateway_port)
.field("gateway_bind_address", &self.gateway_bind_address)
.field("gateway_tls_enabled", &self.gateway_tls_enabled)
.field(
"gateway_callback_client_cert_auth_enabled",
&self.gateway_callback_client_cert_auth_enabled,
)
.field("sandbox_ssh_socket_path", &self.sandbox_ssh_socket_path)
.field("network_name", &self.network_name)
.field("host_gateway_ip", &self.host_gateway_ip)
Expand All @@ -288,6 +317,10 @@ impl std::fmt::Debug for PodmanComputeConfig {
"health_check_interval_secs",
&self.health_check_interval_secs,
)
.field(
"enable_auto_callback_listener",
&self.enable_auto_callback_listener,
)
.finish()
}
}
Expand Down
Loading
Loading