feat(webhook): Wire envoy-sidecar mTLS through pod mutator + fix svid-output mount

[huang195]

Summary

• Wires Spec.MTLSMode on AgentRuntime CRs through to a per-agent rendered Envoy data-plane TLS configuration when Spec.AuthBridgeMode: envoy-sidecar. Closes the operator-side gap that the kagenti-extensions PR Bug: SharedTrust and MLflow controllers block all reconciliation when RBAC is missing #440 demo had to work around with hand-crafted manifests.
• Fixes a bundled bug: the /opt mount on envoy-sidecar's authbridge container was set readOnly: true (asymmetric with proxy-sidecar's RW), blocking the in-process spiffe Provider mirror from writing /opt/svid*.pem and Envoy from booting.
• Relaxes the admission webhook's envoy-sidecar + non-disabled mtls rejection. CRD enum validation already pins mtlsMode; no remaining incompatible combinations today.

Design

Mode semantics match what the kagenti-extensions PR #440 demo proved at the data-plane level:

mtlsMode	Inbound (peer-facing :15124)	Outbound (app-egress :15123)
disabled (default)	plaintext	plaintext
permissive	tls_inspector + two filter chains: TLS chain terminates mTLS, raw_buffer chain accepts plaintext	plaintext (no TLS-wrap attempt)
strict	tls_inspector + single TLS chain; plaintext rejected at chain match	UpstreamTlsContext on original_destination_tls cluster: TLS-or-fail

Inbound is byte-identical to proxy-sidecar's tlssniff.Listener semantics, expressed as Envoy filter chains — matches Istio's PERMISSIVE/STRICT inbound exactly.

Outbound is Istio-shaped: no per-connection try-then-fallback (Envoy has no native primitive for it, and Istio itself doesn't do it). Blanket TLS-or-fail in strict mode is practical for the same reason proxy-sidecar's strict mode is: outbound calls that need plaintext (Keycloak / JWKS / external HTTPS) bypass the listener (plugin-side via Go net/http, external HTTPS via iptables exclusion).

Wiring

• envoy.yaml.tmpl — adds conditional TLS blocks gated on MTLSEnabled (permissive: tls_inspector + TLS chain + raw_buffer chain; strict: tls_inspector + TLS chain only) plus a strict-only original_destination_tls cluster with UpstreamTlsContext. SVID files at /opt/svid*.pem.
• envoy_template.go — envoyTemplateData carries MTLSEnabled bool + MTLSMode string; RenderEnvoyConfig populates from ResolvedConfig.MTLSMode.
• resolved_config.go — adds MTLSMode + AuthBridgeMode fields, populated via the same CR > namespace ConfigMap > "" chain pod_mutator already implements (ExtractMode / ExtractMTLSMode helpers).
• pod_mutator.go — envoy-sidecar branch now passes the resolved mtlsMode to ensurePerAgentConfigMap (was hardcoded "" under the old webhook gate). When mtls is non-disabled, calls a new ensurePerAgentEnvoyConfigMap that SSAs an envoy-config-<crName> ConfigMap with the rendered envoy.yaml. SSA + OwnerReference machinery mirrors the existing ensurePerAgentConfigMap.
• volume_builder.go — adds overrideEnvoyConfigMapInVolumes (mirroring the existing overrideAuthBridgeConfigMapInVolumes) so the envoy-config volume reference points at the per-agent CM.
• agentruntime_webhook.go — checkMTLSCompatibleWithMode no longer rejects envoy-sidecar + non-disabled mtls. The function stays as a hook for future incompatible combinations.

Bug fix

• container_builder.go:117 — removed ReadOnly: true on the svid-output volumeMount in BuildEnvoyProxyContainerWithSpireOption. Asymmetric with BuildProxySidecarContainerWithPorts (correctly RW). The in-process spiffe Provider mirror inside the authbridge-envoy combined image needs to write SVID files there; under the RO mount it logged atomicWrite: ... read-only file system and Envoy refused to boot with Invalid path: /opt/svid_bundle.pem. Historical context: the RO was a copy-paste from the days of an external spiffe-helper writer container that no longer exists.
• container_builder_test.go — flipped the wrong test assertion. The test currently checks if !vm.ReadOnly { t.Error("svid-output mount should be read-only") } — i.e., it was asserting the bug as the desired behavior. Replaced with the opposite check.

Tests

• envoy_template_test.go — three new cases lock in the rendered shape per mode (disabled = no TLS bits; permissive = both chains, no UpstreamTls; strict = TLS-only chain + UpstreamTlsContext + TLS cluster).
• agentruntime_webhook_test.go — envoy-sidecar + permissive and envoy-sidecar + strict cases flipped from wantErr: true to wantErr: false. Drops the strings.Contains("envoy-sidecar mTLS is tracked as a follow-up") check that's no longer applicable.
• container_builder_test.go — assertion flipped (svid-output mount must be RW).

Test plan

• go test -race -count=1 ./internal/webhook/... passes (config + injector + v1alpha1).
• go test -race -count=1 ./... passes everything except test/e2e which wants a kind cluster named "kind" and is infra-dependent.
• go build ./... clean.
• End-to-end on a SPIRE-enabled local kind cluster: built the operator image with this PR, loaded into kind, restarted the controller, applied an AgentRuntime CR with Spec.AuthBridgeMode: envoy-sidecar + Spec.MTLSMode: strict plus its target Deployment. Verified:
  • Operator created envoy-config-<name> with tls_inspector + DownstreamTlsContext + UpstreamTlsContext + original_destination_tls cluster (strict-mode shape).
  • Operator created authbridge-config-<name> with mtls.mode: strict propagated through the resolution chain.
  • Pod reached 2/2 Running. The in-process spiffe Provider mirror successfully wrote /opt/svid.pem, /opt/svid_key.pem, /opt/svid_bundle.pem — confirming the bug 1 fix is what unblocks Envoy.
  • Envoy logs: starting workers cleanly, no Invalid path: /opt/svid_bundle.pem failures.

Follow-up (separate, small PRs, not blocking):

• kagenti-extensions: update authbridge/demos/mtls/scripts/patch-mtls-config.sh to patch the namespace authbridge-runtime-config CM instead of the per-agent CM (operator-owned). Fixes the existing make demo-mtls (proxy-sidecar) verification step.
• kagenti: relax the envoy-sidecar + non-disabled mtls rejection in backend/app/routers/agents.py and remove the matching UI lock in ImportAgentPage.tsx.

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[pdettori]

Clean PR — correct Envoy TLS wiring (matches Istio PERMISSIVE/STRICT semantics), well-justified ReadOnly bug fix, and good template-level test coverage. All 16 CI checks pass including E2E.

Two minor suggestions inline, nothing blocking.

[pdettori]

nit: The double-check cfg.MTLSMode != "" && cfg.MTLSMode != MTLSModeDisabled is defensive — worth a one-line comment noting that "" is treated as disabled (since ResolvedConfig defaults it to "" when unset). Prevents a future reader from wondering why both conditions are needed.

[pdettori]

suggestion: overrideEnvoyConfigMapInVolumes has no unit test. A small table-driven test (volume found → CM name swapped, not found → list unchanged) would lock in correctness cheaply.

[pdettori]

nit: The 6-line comment block (lines 595-600) is helpful context but dense — consider condensing to ~2 lines, e.g. "ResolveConfig is cheap/idempotent; we clear EnvoyYAML so RenderEnvoyConfig uses the template path instead of short-circuiting."