fix(podman): harden sandbox callbacks

Deliver per-sandbox JWTs through Podman secrets so containerized gateways do not leak gateway-local paths to the host Podman API.

Add protected, bindability-checked callback listeners for host Podman gateways, including rootless pasta loopback binds discovered from local interfaces. Default RPM gateways to loopback, expose enable_auto_callback_listener for opt-out, and improve containerized socket diagnostics.

Signed-off-by: Adam Miller <admiller@redhat.com>

Author: maxamillion

Cargo.lock

@@ -64,7 +64,7 @@ metrics = "0.24"
 metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] }
 
 # Unix/Process
-nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term"] }
+nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term", "net"] }
 rustix = { version = "1.1", features = ["process"] }
 
 # Serialization

Cargo.toml

@@ -31,7 +31,7 @@ when a sandbox create request asks for GPU resources.
 | Runtime | Best fit | Sandbox boundary | Notes |
 |---|---|---|---|
 | Docker | Local development with Docker available. | Container plus nested sandbox namespace. | Uses host networking so loopback gateway endpoints work from the supervisor. |
-| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
+| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, CDI GPU devices when available, Podman secrets for sandbox JWTs, and bindability-checked callback listeners for direct-host loopback gateway binds. Bridge-mode gateways can listen on the Podman bridge IP when bindable; rootless pasta can add a protected non-loopback local-interface listener when TLS/auth are enabled. Containerized gateways skip host listener discovery and require explicit port publishing. |
 | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
 | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
 | Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

architecture/compute-runtimes.md

@@ -255,6 +255,29 @@ if config.grpc_endpoint.is_empty() {
 The bridge gateway IP is not a stable substitute in rootless mode because it
 can live inside the user namespace rather than on the host.
 
+### Rootless pasta callback listener
+
+When the gateway binds only to loopback, host-local clients can reach
+`127.0.0.1:<port>`, but sandbox containers do not share the host loopback
+namespace. In rootless Podman with pasta, `host.containers.internal` reaches the
+host through pasta's forwarding path and may not connect to a listener bound
+only to `127.0.0.1`.
+
+For direct host gateways, the Podman driver detects rootless pasta from Podman
+system info (`rootless=true` and `rootlessNetworkCmd=pasta`). If the gateway
+bind address is loopback-only, TLS/auth are active, and
+auto-discovery is enabled, the driver adds a second callback listener on a
+bindable non-loopback local interface address. Candidate addresses come from
+local interface enumeration via `getifaddrs`; the driver does not probe an
+external IP to find this address. The supervisor endpoint remains
+`host.containers.internal:<port>`.
+
+Set `enable_auto_callback_listener = false` in `[openshell.drivers.podman]` or
+`OPENSHELL_PODMAN_ENABLE_AUTO_CALLBACK_LISTENER=false` to disable the automatic
+listener. Containerized gateways skip host listener discovery because their
+network namespace is not the host namespace; publish the gateway container port
+on an address sandbox containers can reach.
+
 ### Layer 3 Inner Sandbox Network Namespace
 
 Inside the container, the supervisor creates another network namespace for the
@@ -391,7 +414,10 @@ Gateway
 
 The gateway binds to `127.0.0.1:17670` by default in the RPM packaging. Client
 certificates are auto-generated by `openshell-gateway generate-certs` on first
-start and bind-mounted into sandbox containers by the Podman driver.
+start and bind-mounted into sandbox containers by the Podman driver. In
+rootless pasta mode, the protected callback listener described above allows the
+supervisor to keep using `host.containers.internal:<port>` while the primary
+listener remains loopback-only.
 
 ## Differences from the Kubernetes Driver
 
@@ -406,7 +432,7 @@ start and bind-mounted into sandbox containers by the Podman driver.
 | DNS | Kubernetes CoreDNS | Podman bridge DNS through aardvark-dns when DNS is enabled. |
 | Network policy | Kubernetes network policy for pod ingress plus supervisor policy | nftables inside inner sandbox netns plus supervisor policy. |
 | Supervisor delivery | Kubernetes driver managed pod image or template | OCI image volume mount. |
-| Secrets | Kubernetes Secret volume and env vars | Mounted TLS client materials from a Podman secret. |
+| Secrets | Kubernetes Secret volume and env vars | Per-sandbox JWT via Podman secret; TLS client materials from configured host files. |
 
 Both drivers use the same reverse gRPC relay for SSH transport. The most
 important Podman-specific difference is network reachability: in rootless

crates/openshell-driver-podman/NETWORKING.md

@@ -68,8 +68,8 @@ mount types:
   pulls the image during provisioning using the sandbox image pull policy.
 
 Host bind mounts are disabled by default because they expose gateway host paths
-to sandbox requests. The driver still uses internal bind mounts for
-OpenShell-owned token and TLS material.
+to sandbox requests. The driver still uses internal bind mounts for configured
+TLS material; per-sandbox gateway JWTs are delivered through Podman secrets.
 
 Podman `bind` mounts accept `source`, `target`, and optional `read_only`.
 User-supplied bind and volume mounts are read-only by default; set
@@ -214,6 +214,10 @@ Key points:
   host. Linux defaults to Podman's `host-gateway` resolver. macOS Podman
   machine defaults to gvproxy's host-loopback IP, `192.168.127.254`, because
   stale Podman machines may fail to resolve `host-gateway`.
+- Callback listener: direct host gateways that bind to loopback can add a
+  protected non-loopback listener for rootless pasta callbacks when TLS/auth are
+  enabled. Disable this with `enable_auto_callback_listener = false`. Gateway
+  containers skip host listener discovery and require explicit port publishing.
 - nsenter: the supervisor uses `nsenter --net=` instead of `ip netns exec` for
   namespace operations, avoiding the sysfs remount path that fails in rootless
   containers.
@@ -336,6 +340,7 @@ Podman resources after out-of-band container removal or label drift.
 | `OPENSHELL_GATEWAY_PORT` | `--gateway-port` | `17670` | Gateway port used for endpoint auto-detection by the standalone binary. |
 | `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name. |
 | `OPENSHELL_PODMAN_HOST_GATEWAY_IP` | `--host-gateway-ip` | empty on Linux, `192.168.127.254` on macOS | Host gateway IP used for sandbox host aliases. Empty uses Podman's `host-gateway` resolver. |
+| `OPENSHELL_PODMAN_ENABLE_AUTO_CALLBACK_LISTENER` | none | `true` | Enable the protected non-loopback callback listener for rootless pasta when the gateway is loopback-bound and TLS/auth are enabled. |
 | `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Supervisor Unix socket path in `PodmanComputeConfig`. |
 | `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `10` | Container stop timeout in seconds. |
 | `OPENSHELL_SANDBOX_PIDS_LIMIT` | `--sandbox-pids-limit` | `2048` | Podman cgroup PID limit for sandbox containers. Set `0` to inherit Podman's runtime/default PID limit. |

crates/openshell-driver-podman/README.md

@@ -245,6 +245,8 @@ pub struct HostInfo {
     #[serde(default)]
     pub network_backend: String,
     #[serde(default)]
+    pub rootless_network_cmd: String,
+    #[serde(default)]
     pub security: SecurityInfo,
 }
 
@@ -391,6 +393,23 @@ impl PodmanClient {
         }
     }
 
+    /// Perform a versioned HTTP request with a raw byte body (not JSON).
+    async fn request_raw(
+        &self,
+        method: hyper::Method,
+        path: &str,
+        content_type: &str,
+        body: Bytes,
+    ) -> Result<(hyper::StatusCode, Bytes), PodmanApiError> {
+        let req = Self::build_request(
+            method,
+            &format!("/{API_VERSION}{path}"),
+            Full::new(body),
+            Some(content_type),
+        );
+        self.send_request(req, API_TIMEOUT).await
+    }
+
     /// POST a JSON body and ignore 409 Conflict (resource already exists).
     async fn create_ignore_conflict(&self, path: &str, body: &Value) -> Result<(), PodmanApiError> {
         match self
@@ -518,6 +537,63 @@ impl PodmanClient {
         .await
     }
 
+    // ── Secret operations ────────────────────────────────────────────────
+
+    /// Create a Podman secret with the given name and raw value.
+    ///
+    /// Idempotent: if a secret with the same name already exists it is
+    /// replaced (delete + recreate) so the value is always up-to-date.
+    pub async fn create_secret(&self, name: &str, value: &[u8]) -> Result<(), PodmanApiError> {
+        validate_name(name)?;
+        let encoded_name = url_encode(name);
+        let path = format!("/libpod/secrets/create?name={encoded_name}");
+        let (status, bytes) = self
+            .request_raw(
+                hyper::Method::POST,
+                &path,
+                "application/octet-stream",
+                Bytes::copy_from_slice(value),
+            )
+            .await?;
+
+        match status.as_u16() {
+            200 | 201 => Ok(()),
+            409 => {
+                self.remove_secret(name).await?;
+                let (status2, bytes2) = self
+                    .request_raw(
+                        hyper::Method::POST,
+                        &path,
+                        "application/octet-stream",
+                        Bytes::copy_from_slice(value),
+                    )
+                    .await?;
+                if status2.is_success() {
+                    Ok(())
+                } else {
+                    Err(error_from_response(status2.as_u16(), &bytes2))
+                }
+            }
+            _ => Err(error_from_response(status.as_u16(), &bytes)),
+        }
+    }
+
+    /// Remove a Podman secret by name. Idempotent (not-found is ignored).
+    pub async fn remove_secret(&self, name: &str) -> Result<(), PodmanApiError> {
+        validate_name(name)?;
+        match self
+            .request_ok(
+                hyper::Method::DELETE,
+                &format!("/libpod/secrets/{name}"),
+                None,
+            )
+            .await
+        {
+            Ok(()) | Err(PodmanApiError::NotFound(_)) => Ok(()),
+            Err(e) => Err(e),
+        }
+    }
+
     // ── Network operations ───────────────────────────────────────────────
 
     /// Create a bridge network with DNS enabled. Idempotent.

crates/openshell-driver-podman/src/client.rs

@@ -2,7 +2,7 @@
 // SPDX-License-Identifier: Apache-2.0
 
 use openshell_core::config::{DEFAULT_STOP_TIMEOUT_SECS, DEFAULT_SUPERVISOR_IMAGE};
-use std::net::IpAddr;
+use std::net::{IpAddr, SocketAddr};
 use std::path::PathBuf;
 use std::str::FromStr;
 
@@ -68,6 +68,7 @@ impl FromStr for ImagePullPolicy {
 
 #[derive(Clone, serde::Serialize, serde::Deserialize)]
 #[serde(default, deny_unknown_fields)]
+#[allow(clippy::struct_excessive_bools)]
 pub struct PodmanComputeConfig {
     /// Path to the Podman API Unix socket.
     /// Default: `$XDG_RUNTIME_DIR/podman/podman.sock` (Linux),
@@ -89,6 +90,15 @@ pub struct PodmanComputeConfig {
     /// so the correct port is used even when `--port` differs from the
     /// default.  Defaults to [`openshell_core::config::DEFAULT_SERVER_PORT`].
     pub gateway_port: u16,
+    /// Full gateway bind address supplied by the gateway process at runtime.
+    #[serde(skip)]
+    pub gateway_bind_address: Option<SocketAddr>,
+    /// Whether the gateway has TLS enabled. Supplied by the gateway process.
+    #[serde(skip)]
+    pub gateway_tls_enabled: bool,
+    /// Whether gateway callbacks are protected by an auth mechanism.
+    #[serde(skip)]
+    pub gateway_callback_auth_enabled: bool,
     /// Unix socket path the in-container supervisor bridges relay traffic to.
     pub sandbox_ssh_socket_path: String,
     /// Name of the Podman bridge network.
@@ -134,10 +144,18 @@ pub struct PodmanComputeConfig {
     /// Set to `0` to disable health checks entirely.
     /// Defaults to [`DEFAULT_HEALTH_CHECK_INTERVAL_SECS`] (10 seconds).
     pub health_check_interval_secs: u64,
+    /// Automatically add a non-loopback callback listener for rootless Podman
+    /// pasta when the primary gateway bind is loopback-only and TLS/auth are active.
+    #[serde(default = "default_true")]
+    pub enable_auto_callback_listener: bool,
 }
 
 pub const DEFAULT_HEALTH_CHECK_INTERVAL_SECS: u64 = 10;
 
+fn default_true() -> bool {
+    true
+}
+
 impl PodmanComputeConfig {
     /// Returns `true` when all three TLS paths are configured.
     #[must_use]
@@ -251,6 +269,9 @@ impl Default for PodmanComputeConfig {
             image_pull_policy: ImagePullPolicy::default(),
             grpc_endpoint: String::new(),
             gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
+            gateway_bind_address: None,
+            gateway_tls_enabled: false,
+            gateway_callback_auth_enabled: false,
             sandbox_ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
             network_name: DEFAULT_NETWORK_NAME.to_string(),
             host_gateway_ip: Self::default_host_gateway_ip(),
@@ -262,6 +283,7 @@ impl Default for PodmanComputeConfig {
             sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT,
             enable_bind_mounts: false,
             health_check_interval_secs: DEFAULT_HEALTH_CHECK_INTERVAL_SECS,
+            enable_auto_callback_listener: true,
         }
     }
 }
@@ -274,6 +296,12 @@ impl std::fmt::Debug for PodmanComputeConfig {
             .field("image_pull_policy", &self.image_pull_policy.as_str())
             .field("grpc_endpoint", &self.grpc_endpoint)
             .field("gateway_port", &self.gateway_port)
+            .field("gateway_bind_address", &self.gateway_bind_address)
+            .field("gateway_tls_enabled", &self.gateway_tls_enabled)
+            .field(
+                "gateway_callback_auth_enabled",
+                &self.gateway_callback_auth_enabled,
+            )
             .field("sandbox_ssh_socket_path", &self.sandbox_ssh_socket_path)
             .field("network_name", &self.network_name)
             .field("host_gateway_ip", &self.host_gateway_ip)
@@ -288,6 +316,10 @@ impl std::fmt::Debug for PodmanComputeConfig {
                 "health_check_interval_secs",
                 &self.health_check_interval_secs,
             )
+            .field(
+                "enable_auto_callback_listener",
+                &self.enable_auto_callback_listener,
+            )
             .finish()
     }
 }