Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,28 @@ The shared state directory should preserve `sandbox_gid` inheritance
`@openshell-sidecar-ssh`; the network sidecar verifies its peer PID before
bridging gateway relay requests. No `ssh.sock` file should appear in the shared
state directory.
Inspect all three when sandbox registration or egress enforcement fails:

If `topology = "proxy-pod"` is rendered, each sandbox should have a
separate supervisor Deployment with one supervisor pod, a headless supervisor
Service, a proxy CA Secret, and two per-sandbox NetworkPolicies. The agent pod
should have `openshell.ai/sandbox-role=agent`; the supervisor pod should have
`openshell.ai/sandbox-role=supervisor`; both should share the same
`openshell.ai/sandbox-id`. The supervisor Deployment must have a controlling
`Sandbox` ownerReference. The Deployment pod template must carry the
`openshell.io/sandbox-id` annotation so the TokenReview bootstrap path can mint
a sandbox JWT. For supervisor pods, the gateway validates the
`Pod -> ReplicaSet -> Deployment -> Sandbox` owner chain, so missing
`apps/replicasets get` RBAC can also break bootstrap. Helm renders the
Deployment, ReplicaSet, Service, Secret, and NetworkPolicy RBAC only when
`supervisor.topology=proxy-pod`; if those resources fail with forbidden errors,
confirm both the rendered `gateway.toml` and Helm values use proxy-pod topology.
If the agent cannot reach the gateway, check DNS to the headless Service, the
agent egress NetworkPolicy DNS exception for kube-dns/CoreDNS, and the
supervisor ingress NetworkPolicy allowing only that agent pod on ports `3128`
and `18080`.

Inspect the relevant containers when sandbox registration or egress enforcement
fails:

```bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep -E '^\[openshell\.drivers\.kubernetes\]|^topology\s*='
Expand Down
52 changes: 48 additions & 4 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,15 +70,27 @@ mise run helm:skaffold:run:sidecar
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
**Supervisor proxy-pod topology** (build once and leave running):
```bash
mise run helm:skaffold:run:proxy-pod
```

All Skaffold commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and an `openshell-supervisor-network` runtime sidecar for proxying.
Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
`DAC_READ_SEARCH`; relaxed mode can run it as the configured proxy UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. Envoy Gateway opt-in; see the Optional Add-ons section below.
`server.disableTls=false` inline for Skaffold. The proxy-pod profile renders
network supervision in a separate supervisor Deployment with one pod and relies
on Kubernetes NetworkPolicy enforcement so the agent pod can reach only its
paired supervisor plus DNS. The
default local k3s/k3d cluster keeps k3s's embedded NetworkPolicy controller
enabled; if you replace the CNI, install a policy-enforcing CNI before using
proxy-pod. The
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway
generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in;
see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.

Expand All @@ -87,6 +99,31 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
create the Secret named `openshell-ha-pg` with a `uri` key, then run
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.

### Kubernetes e2e profiles

Run the default Kubernetes e2e environment:

```bash
mise run e2e:kubernetes
```

Run the sidecar topology e2e environment:

```bash
mise run e2e:kubernetes:sidecar
```

Run the proxy-pod topology e2e environment:

```bash
mise run e2e:kubernetes:proxy-pod
```

The proxy-pod e2e task applies `ci/values-proxy-pod.yaml` through
`OPENSHELL_E2E_KUBE_EXTRA_VALUES`. Use an existing cluster with NetworkPolicy
enforcement, or let the wrapper create the default local k3d/k3s cluster with
k3s's embedded NetworkPolicy controller enabled.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
Expand Down Expand Up @@ -149,6 +186,12 @@ For a sidecar-profile deployment:
mise run helm:skaffold:delete:sidecar
```

For a proxy-pod-profile deployment:

```bash
mise run helm:skaffold:delete:proxy-pod
```

### Delete the cluster entirely

```bash
Expand Down Expand Up @@ -274,6 +317,7 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-proxy-pod.yaml` | Supervisor proxy-pod topology overlay for Kubernetes e2e/dev; requires NetworkPolicy enforcement |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ http-body-util = "0.1"
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
rustls-pemfile = "2"
rcgen = { version = "0.13", features = ["crypto", "pem"] }
rcgen = { version = "0.13", features = ["crypto", "pem", "x509-parser"] }
webpki-roots = "1"

# CLI
Expand Down
8 changes: 5 additions & 3 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,11 @@ Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway validates that projected token
with Kubernetes `TokenReview`, requires the configured sandbox service account,
checks the returned pod binding against the live pod UID, and verifies the pod's
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
checks the returned pod binding against the live pod UID, and verifies the
pod's ownership against the live Sandbox CR UID and sandbox-id label before
minting the gateway JWT. Agent pods must be directly controlled by the
`Sandbox` CR. Proxy-pod supervisor pods may be controlled through the Kubernetes
`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
Expand Down
19 changes: 19 additions & 0 deletions crates/openshell-core/src/sandbox_env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,32 @@ pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";
/// container.
pub const SIDECAR_CONTROL_SOCKET: &str = "OPENSHELL_SIDECAR_CONTROL_SOCKET";

/// Address where an external network supervisor forwards gateway gRPC traffic.
pub const GATEWAY_FORWARD_ADDR: &str = "OPENSHELL_GATEWAY_FORWARD_ADDR";

/// Optional TLS server name override used when connecting to the gateway.
pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";

/// Explicit URL injected into sandbox child processes for proxy-mode egress.
///
/// Kubernetes proxy-pod topology uses a headless Service DNS name, which
/// cannot be represented by the policy's `SocketAddr` proxy field.
pub const PROXY_URL: &str = "OPENSHELL_PROXY_URL";

/// Explicit listener address for the network supervisor's HTTP CONNECT proxy.
pub const PROXY_BIND_ADDR: &str = "OPENSHELL_PROXY_BIND_ADDR";

/// Directory where the network supervisor writes the proxy CA files consumed
/// by workload child processes.
pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";

/// Optional CA certificate PEM path used by the network supervisor instead of
/// generating an ephemeral CA.
pub const PROXY_CA_CERT_PATH: &str = "OPENSHELL_PROXY_CA_CERT_PATH";

/// Optional CA private key PEM path paired with [`PROXY_CA_CERT_PATH`].
pub const PROXY_CA_KEY_PATH: &str = "OPENSHELL_PROXY_CA_KEY_PATH";

/// Path to the CA certificate for mTLS communication with the gateway.
pub const TLS_CA: &str = "OPENSHELL_TLS_CA";

Expand Down
1 change: 1 addition & 0 deletions crates/openshell-driver-kubernetes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ tracing = { workspace = true }
tracing-subscriber = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
rcgen = { workspace = true }

[dev-dependencies]
temp-env = "0.3"
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,17 @@ abstract socket whose peer PID must match that authenticated supervisor. Both
supervisors exit if the control connection closes, coupling their container
restart lifecycle before a new authoritative client can be established.

The `proxy-pod` supervisor topology runs network enforcement and gateway
forwarding in a separate supervisor Deployment with one pod. The agent pod runs
the sandbox image directly and reaches the supervisor through a per-sandbox
headless Service. The driver creates an owner-referenced supervisor
Deployment with one replica plus Service, proxy CA Secret, and NetworkPolicy
resources so agent egress is limited to its paired supervisor pod plus DNS. If
the supervisor pod is deleted, the Deployment recreates it. The workload pod
does not mount gateway credentials or the supervisor binary. This topology
intentionally omits filesystem/process/binary enforcement, SSH/exec,
upload/download, sync, and provider environment injection.

The driver can request a Kubernetes AppArmor profile through
`app_armor_profile`.

Expand Down
Loading
Loading