Skip to content

feat(kubernetes): add sidecar supervisor topology#2076

Open
TaylorMutch wants to merge 8 commits into
mainfrom
feat/kubernetes-sidecar-topology-v2
Open

feat(kubernetes): add sidecar supervisor topology#2076
TaylorMutch wants to merge 8 commits into
mainfrom
feat/kubernetes-sidecar-topology-v2

Conversation

@TaylorMutch

@TaylorMutch TaylorMutch commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the Kubernetes sidecar supervisor topology after the combined-topology base from #2074.
The default combined topology remains unchanged, and sidecar is opt-in.

sidecar moves pod-level network enforcement and gateway forwarding into a dedicated supervisor sidecar. The agent container can run as the resolved sandbox UID/GID with runAsNonRoot, no privilege escalation, and all Linux capabilities dropped.

This draft intentionally includes the prerequisite UID/GID commits from #1973 for now. We expect to rebase those out after that work lands separately.

Runtime validation status:

Sidecar mode preserves gateway session and SSH behavior, but intentionally runs the process supervisor in network-only mode. Filesystem policy, process privilege dropping, and process/binary identity checks are not applied in this mode.

Related Issue

References #1827, #981, #899, #1305.

Related PRs: #1973, #2074, #2016.

Changes

  • Include prerequisite sandbox UID/GID support: numeric sandbox identities, configurable Kubernetes sandbox UID/GID values, and OpenShift SCC annotation fallback.
  • Add the sidecar supervisor topology and the related processEnforcement and proxyUid configuration.
  • Render sidecar-mode sandbox pods with a network init container, non-root supervisor sidecar, and unprivileged agent container.
  • Add process-supervisor network-only behavior for sidecar mode while keeping SSH/session relay behavior intact.
  • Add sidecar e2e Helm values and Skaffold profile support.
  • Add a sidecar topology row to the branch Kubernetes E2E matrix.
  • Document topology choice, architecture diagrams, permission model, RuntimeClass validation status, and network-only tradeoffs.

Testing

  • git diff --check origin/main..feat/kubernetes-sidecar-topology-v2
  • cargo check -p openshell-driver-kubernetes -p openshell-sandbox -p openshell-supervisor-process -p openshell-supervisor-network
  • cargo test -p openshell-driver-kubernetes --lib
  • cargo test -p openshell-supervisor-process --lib
  • mise run helm:test
  • markdownlint-cli2 docs/kubernetes/topology.mdx docs/kubernetes/setup.mdx
  • ruby -e 'require "yaml"; YAML.load_file(".github/workflows/branch-e2e.yml"); puts "ok"'
  • mise run rust:lint
  • Prior validation from feat(kubernetes): add sidecar and proxy-pod topology configurations #2016: HELM_K3S_LB_HOST_PORT=18080 mise run e2e:kubernetes:sidecar
  • Prior validation from feat(kubernetes): add sidecar and proxy-pod topology configurations #2016: sidecar topology smoke-tested as experimental on Kata Containers; known to fail on gVisor.
  • Sidecar topology validated on GKE Autopilot using --workload-policies=allow-net-admin.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Jun 30, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@TaylorMutch TaylorMutch added the test:e2e Requires end-to-end coverage label Jun 30, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2076 is at {"messa while the PR head is 6276de3. A maintainer needs to comment /ok to test 6276de38ac92b65b0b24a68b8d09a5eed06f50d7 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

@TaylorMutch TaylorMutch marked this pull request as ready for review June 30, 2026 22:46
@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch 2 times, most recently from 4f981c2 to 8c9ae53 Compare July 2, 2026 21:12
@mrunalp

mrunalp commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Some comments with help of agent

1. Central trust boundary — the workload can use the sandbox's gateway identity .
Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

2. nftables fence only covers TCP/UDP.
The output chain is policy accept and only rejects TCP/UDP from non-proxy_uid UIDs. ICMP (e.g. unprivileged ping sockets) and other L4 protocols fall through. CAP_NET_RAW is dropped so raw sockets aren't available, but this is a narrower egress boundary than combined mode's per-netns setup. Low severity — suggest a brief comment in generate_sidecar_bypass_ruleset noting the intentional scope.

3. processEnforcement=full does not restore network binary-identity.
supervisor_sidecar_env hard-codes NETWORK_BINARY_IDENTITY=relaxed regardless of process_enforcement, so full restores agent-container Landlock/privilege-drop but network policy stays endpoint/L7-scoped. The docs are technically accurate ("combined-mode process/filesystem controls"), but a reader could reasonably infer full ≈ combined. A one-line note that network policy remains endpoint-scoped in sidecar mode regardless of processEnforcement would prevent that.

4. Doc drift in the debug skill.
debug-openshell-cluster/SKILL.md says the sidecar runs "with primary GID 0" to read the token, but the implementation uses runAsGroup: sandbox_gid + fsGroup: sandbox_gid (which re-owns the projected volume to sandbox_gid). Please update to match.

@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch from f48ebb5 to e00a00b Compare July 6, 2026 16:26
Add the Kubernetes sidecar supervisor topology, its Helm/Skaffold configuration, topology documentation, and sidecar e2e matrix coverage. Skip root-only sandbox identity rewriting when process enforcement is network-only so the low-permission sidecar process container can start successfully.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch from e00a00b to ed858eb Compare July 6, 2026 18:52
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
@TaylorMutch

Copy link
Copy Markdown
Collaborator Author
  1. Central trust boundary — the workload can use the sandbox's gateway identity .
    Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

Addressed this by moving the supervisor and sandbox to run with different GIDs.

Updated the other areas with clarifying comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants