Skip to content

Commit 915bf6e

Browse files
committed
feat(kubernetes): add proxy-pod supervisor topology
Add the Kubernetes proxy-pod topology with one supervisor Deployment and Service per sandbox, NetworkPolicy confinement, proxy-pod Helm/Skaffold configuration, topology documentation, and focused supervisor identity tests. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
1 parent 254cdbe commit 915bf6e

35 files changed

Lines changed: 2697 additions & 321 deletions

File tree

.agents/skills/debug-openshell-cluster/SKILL.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -299,7 +299,28 @@ all denied, inspect that file and the network sidecar logs.
299299
The shared state directory should preserve `sandbox_gid` inheritance
300300
(`02775`), and the SSH socket should be group-connectable (`0660`) so the
301301
network sidecar can bridge gateway relay requests to the process supervisor.
302-
Inspect all three when sandbox registration or egress enforcement fails:
302+
303+
If `supervisor_topology = "proxy-pod"` is rendered, each sandbox should have a
304+
separate supervisor Deployment with one supervisor pod, a headless supervisor
305+
Service, a proxy CA Secret, and two per-sandbox NetworkPolicies. The agent pod
306+
should have `openshell.ai/sandbox-role=agent`; the supervisor pod should have
307+
`openshell.ai/sandbox-role=supervisor`; both should share the same
308+
`openshell.ai/sandbox-id`. The supervisor Deployment must have a controlling
309+
`Sandbox` ownerReference. The Deployment pod template must carry the
310+
`openshell.io/sandbox-id` annotation so the TokenReview bootstrap path can mint
311+
a sandbox JWT. For supervisor pods, the gateway validates the
312+
`Pod -> ReplicaSet -> Deployment -> Sandbox` owner chain, so missing
313+
`apps/replicasets get` RBAC can also break bootstrap. Helm renders the
314+
Deployment, ReplicaSet, Service, Secret, and NetworkPolicy RBAC only when
315+
`supervisor.topology=proxy-pod`; if those resources fail with forbidden errors,
316+
confirm both the rendered `gateway.toml` and Helm values use proxy-pod topology.
317+
If the agent cannot reach the gateway, check DNS to the headless Service, the
318+
agent egress NetworkPolicy DNS exception for kube-dns/CoreDNS, and the
319+
supervisor ingress NetworkPolicy allowing only that agent pod on ports `3128`
320+
and `18080`.
321+
322+
Inspect the relevant containers when sandbox registration or egress enforcement
323+
fails:
303324

304325
```bash
305326
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep supervisor_topology

.agents/skills/helm-dev-environment/SKILL.md

Lines changed: 46 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,10 +70,22 @@ mise run helm:skaffold:run:sidecar
7070
mise run helm:skaffold:run:sidecar-mtls
7171
```
7272

73-
Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
73+
**Supervisor proxy-pod topology** (build once and leave running):
74+
```bash
75+
mise run helm:skaffold:run:proxy-pod
76+
```
77+
78+
All Skaffold commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
7479
chart. The sidecar profile renders an `openshell-network-init` init container for
7580
nftables setup and a non-root `openshell-supervisor-network` runtime sidecar for
76-
proxying. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway
81+
proxying. The sidecar-mtls profile re-enables built-in TLS/mTLS after the
82+
Skaffold dev values. The proxy-pod profile renders network supervision in a separate
83+
supervisor Deployment with one pod and relies on Kubernetes NetworkPolicy
84+
enforcement so the agent pod can reach only its paired supervisor plus DNS. The
85+
default local k3s/k3d cluster keeps k3s's embedded NetworkPolicy controller
86+
enabled; if you replace the CNI, install a policy-enforcing CNI before using
87+
proxy-pod. The
88+
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway
7789
generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in;
7890
see the Optional Add-ons section below.
7991

@@ -84,6 +96,31 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
8496
create the Secret named `openshell-ha-pg` with a `uri` key, then run
8597
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.
8698

99+
### Kubernetes e2e profiles
100+
101+
Run the default Kubernetes e2e environment:
102+
103+
```bash
104+
mise run e2e:kubernetes
105+
```
106+
107+
Run the sidecar topology e2e environment:
108+
109+
```bash
110+
mise run e2e:kubernetes:sidecar
111+
```
112+
113+
Run the proxy-pod topology e2e environment:
114+
115+
```bash
116+
mise run e2e:kubernetes:proxy-pod
117+
```
118+
119+
The proxy-pod e2e task applies `ci/values-proxy-pod.yaml` through
120+
`OPENSHELL_E2E_KUBE_EXTRA_VALUES`. Use an existing cluster with NetworkPolicy
121+
enforcement, or let the wrapper create the default local k3d/k3s cluster with
122+
k3s's embedded NetworkPolicy controller enabled.
123+
87124
### TLS behaviour
88125

89126
`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
@@ -145,6 +182,12 @@ For a sidecar-profile deployment:
145182
mise run helm:skaffold:delete:sidecar
146183
```
147184

185+
For a proxy-pod-profile deployment:
186+
187+
```bash
188+
mise run helm:skaffold:delete:proxy-pod
189+
```
190+
148191
### Delete the cluster entirely
149192

150193
```bash
@@ -271,6 +314,7 @@ for dependencies still declared in `Chart.yaml`.
271314
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
272315
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
273316
| `deploy/helm/openshell/ci/values-sidecar-mtls.yaml` | Supervisor sidecar topology overlay with built-in TLS/mTLS re-enabled after Skaffold dev values |
317+
| `deploy/helm/openshell/ci/values-proxy-pod.yaml` | Supervisor proxy-pod topology overlay for Kubernetes e2e/dev; requires NetworkPolicy enforcement |
274318
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
275319
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
276320
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |

Cargo.lock

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ http-body-util = "0.1"
3737
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
3838
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
3939
rustls-pemfile = "2"
40-
rcgen = { version = "0.13", features = ["crypto", "pem"] }
40+
rcgen = { version = "0.13", features = ["crypto", "pem", "x509-parser"] }
4141
webpki-roots = "1"
4242

4343
# CLI

architecture/gateway.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,11 @@ Podman, and VM drivers deliver the initial token through supervisor-only
6464
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
6565
token through `IssueSandboxToken`. The gateway validates that projected token
6666
with Kubernetes `TokenReview`, requires the configured sandbox service account,
67-
checks the returned pod binding against the live pod UID, and verifies the pod's
68-
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
69-
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
67+
checks the returned pod binding against the live pod UID, and verifies the
68+
pod's ownership against the live Sandbox CR UID and sandbox-id label before
69+
minting the gateway JWT. Agent pods must be directly controlled by the
70+
`Sandbox` CR. Proxy-pod supervisor pods may be controlled through the Kubernetes
71+
`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts
7072
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
7173
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
7274
deployments. Supervisors renew gateway JWTs in memory before expiry only while

crates/openshell-core/src/sandbox_env.rs

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,17 @@ pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";
5555
/// File written by the network supervisor when sidecar networking is ready.
5656
pub const SUPERVISOR_READY_FILE: &str = "OPENSHELL_SUPERVISOR_READY_FILE";
5757

58+
/// TCP address the process supervisor waits for before starting when the
59+
/// network supervisor runs outside the agent process.
60+
pub const SUPERVISOR_READY_ADDR: &str = "OPENSHELL_SUPERVISOR_READY_ADDR";
61+
5862
/// File written by the process supervisor with the workload entrypoint PID and
5963
/// read by the network sidecar for process/binary-bound network policy checks.
6064
pub const ENTRYPOINT_PID_FILE: &str = "OPENSHELL_ENTRYPOINT_PID_FILE";
6165

66+
/// Address where an external network supervisor forwards gateway gRPC traffic.
67+
pub const GATEWAY_FORWARD_ADDR: &str = "OPENSHELL_GATEWAY_FORWARD_ADDR";
68+
6269
/// Local protobuf policy snapshot written by the network sidecar and read by
6370
/// the process-only supervisor in Kubernetes sidecar topology.
6471
pub const SIDECAR_POLICY_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE";
@@ -70,10 +77,26 @@ pub const SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_PROVIDER
7077
/// Optional TLS server name override used when connecting to the gateway.
7178
pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";
7279

80+
/// Explicit URL injected into sandbox child processes for proxy-mode egress.
81+
///
82+
/// Kubernetes proxy-pod topology uses a headless Service DNS name, which
83+
/// cannot be represented by the policy's `SocketAddr` proxy field.
84+
pub const PROXY_URL: &str = "OPENSHELL_PROXY_URL";
85+
86+
/// Explicit listener address for the network supervisor's HTTP CONNECT proxy.
87+
pub const PROXY_BIND_ADDR: &str = "OPENSHELL_PROXY_BIND_ADDR";
88+
7389
/// Directory where the network supervisor writes the proxy CA files consumed
7490
/// by workload child processes.
7591
pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";
7692

93+
/// Optional CA certificate PEM path used by the network supervisor instead of
94+
/// generating an ephemeral CA.
95+
pub const PROXY_CA_CERT_PATH: &str = "OPENSHELL_PROXY_CA_CERT_PATH";
96+
97+
/// Optional CA private key PEM path paired with [`PROXY_CA_CERT_PATH`].
98+
pub const PROXY_CA_KEY_PATH: &str = "OPENSHELL_PROXY_CA_KEY_PATH";
99+
77100
/// Path to the CA certificate for mTLS communication with the gateway.
78101
pub const TLS_CA: &str = "OPENSHELL_TLS_CA";
79102

crates/openshell-driver-kubernetes/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ tracing = { workspace = true }
3434
tracing-subscriber = { workspace = true }
3535
thiserror = { workspace = true }
3636
miette = { workspace = true }
37+
rcgen = { workspace = true }
3738

3839
[dev-dependencies]
3940
temp-env = "0.3"

crates/openshell-driver-kubernetes/README.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,18 @@ In this mode OpenShell preserves gateway session and SSH behavior, but the
6565
process supervisor defaults to network-only mode and does not apply Landlock
6666
filesystem policy, process privilege dropping, or process/binary identity
6767
checks. Network endpoint and L7 policy remain enforced by the network sidecar.
68-
Set `process_enforcement = "full"` only when you want combined-mode
69-
process/filesystem guards and accept the added agent-container permissions.
68+
69+
The `proxy-pod` supervisor topology runs network enforcement and gateway
70+
forwarding in a separate supervisor Deployment with one pod. The agent pod runs
71+
only the process-mode supervisor and reaches the supervisor through a
72+
per-sandbox headless Service. The driver creates an owner-referenced supervisor
73+
Deployment with one replica plus Service, proxy CA Secret, and NetworkPolicy
74+
resources so agent egress is limited to its paired supervisor pod plus DNS. If
75+
the supervisor pod is deleted, the Deployment recreates it.
76+
77+
Set `process_enforcement = "full"` in sidecar or proxy-pod topology only when
78+
you want combined-mode process/filesystem guards and accept the added
79+
agent-container permissions.
7080

7181
Sidecar mode uses the pod `fsGroup` to make the projected service-account token
7282
and sandbox client TLS secret group-readable so the non-root process supervisor

crates/openshell-driver-kubernetes/src/config.rs

Lines changed: 26 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default";
1515
/// Default storage size for the workspace PVC.
1616
pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";
1717

18-
/// Default UID for the long-running Kubernetes network supervisor sidecar.
18+
/// Default UID for the long-running Kubernetes network proxy.
1919
pub const DEFAULT_PROXY_UID: u32 = 1337;
2020

2121
/// How the supervisor binary is delivered into sandbox pods.
@@ -65,13 +65,17 @@ pub enum SupervisorTopology {
6565
/// Run network supervision in a privileged sidecar and process supervision
6666
/// as a low-capability wrapper in the agent container.
6767
Sidecar,
68+
/// Run network supervision in a separate supervisor pod and process
69+
/// supervision as a low-capability wrapper in the agent pod.
70+
ProxyPod,
6871
}
6972

7073
impl std::fmt::Display for SupervisorTopology {
7174
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
7275
match self {
7376
Self::Combined => f.write_str("combined"),
7477
Self::Sidecar => f.write_str("sidecar"),
78+
Self::ProxyPod => f.write_str("proxy-pod"),
7579
}
7680
}
7781
}
@@ -83,22 +87,23 @@ impl FromStr for SupervisorTopology {
8387
match s {
8488
"combined" => Ok(Self::Combined),
8589
"sidecar" => Ok(Self::Sidecar),
90+
"proxy-pod" => Ok(Self::ProxyPod),
8691
other => Err(format!("unknown supervisor topology '{other}'")),
8792
}
8893
}
8994
}
9095

91-
/// Process/filesystem controls applied by the process supervisor in split
92-
/// Kubernetes topologies.
96+
/// Process/filesystem controls applied by the process supervisor in
97+
/// non-combined Kubernetes topologies.
9398
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
9499
#[serde(rename_all = "kebab-case")]
95100
pub enum ProcessEnforcementMode {
96-
/// Preserve process launch and session relay behavior, but leave
97-
/// filesystem/process guards to the network supervisor topology.
101+
/// Preserve process launch and session relay behavior while network
102+
/// enforcement is handled by the sidecar or proxy pod.
98103
#[default]
99104
NetworkOnly,
100-
/// Run the process supervisor with the same process/filesystem controls as
101-
/// combined topology.
105+
/// Run the process supervisor with combined-mode process/filesystem
106+
/// controls.
102107
Full,
103108
}
104109

@@ -255,9 +260,10 @@ pub struct KubernetesComputeConfig {
255260
/// non-combined topologies. `network-only` keeps the low-permission agent
256261
/// shape; `full` grants the agent supervisor combined-mode controls.
257262
pub process_enforcement: ProcessEnforcementMode,
258-
/// UID used by the long-running network sidecar in `sidecar` topology.
259-
/// The network init container installs nftables rules that exempt this
260-
/// UID, so it must not match the sandbox workload UID.
263+
/// UID used by the long-running network proxy in sidecar and proxy-pod
264+
/// topologies. In sidecar topology, the network init container installs
265+
/// nftables rules that exempt this UID, so it must not match the sandbox
266+
/// workload UID.
261267
pub proxy_uid: u32,
262268
pub grpc_endpoint: String,
263269
pub ssh_socket_path: String,
@@ -565,6 +571,16 @@ mod tests {
565571
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
566572
}
567573

574+
#[test]
575+
fn serde_override_supervisor_topology_proxy_pod() {
576+
let json = serde_json::json!({
577+
"supervisor_topology": "proxy-pod"
578+
});
579+
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
580+
assert_eq!(cfg.supervisor_topology, SupervisorTopology::ProxyPod);
581+
assert_eq!(cfg.supervisor_topology.to_string(), "proxy-pod");
582+
}
583+
568584
#[test]
569585
fn serde_override_process_enforcement_full() {
570586
let json = serde_json::json!({

0 commit comments

Comments
 (0)