fix(sandbox): invalidate stale l7 tunnels after reload

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

Author: johntmyers

architecture/sandbox.md

@@ -292,18 +292,22 @@ The proxy calls `evaluate_network_action()` (not `evaluate_network()`) as its ma
 
 ### L7 endpoint config query
 
-After L4 allows a connection, `query_endpoint_config(input)` evaluates `data.openshell.sandbox.matched_endpoint_config` to get the full endpoint object. If the endpoint has a `protocol` field, `l7::parse_l7_config()` extracts the L7 config for protocol-aware inspection.
+After L4 allows a connection, `query_endpoint_config_with_generation(input)` evaluates `data.openshell.sandbox.matched_endpoint_config` to get the full endpoint object and the policy generation used for the query. If the endpoint has a `protocol` field, `l7::parse_l7_config()` extracts the L7 config for protocol-aware inspection.
 
 ### Engine cloning for L7
 
-`clone_engine_for_tunnel()` clones the inner `regorus::Engine`. With the `arc` feature, this shares compiled policy via `Arc` and only duplicates interpreter state (microseconds). The cloned engine is wrapped in its own `std::sync::Mutex` and used by the L7 relay without contention on the main engine.
+`clone_engine_for_tunnel(expected_generation)` clones the inner `regorus::Engine` only if the current policy generation still matches the endpoint config generation captured above. With the `arc` feature, this shares compiled policy via `Arc` and only duplicates interpreter state (microseconds). The cloned engine is wrapped in a generation-bound `TunnelPolicyEngine` and used by the L7 relay without contention on the main engine.
+
+The L7 relay checks the captured generation before parsing, evaluating, and forwarding each request. If a policy reload has advanced the shared generation, the relay closes the tunnel before forwarding more bytes. This applies live policy changes to the next L7 request on a keep-alive tunnel and avoids pairing stale endpoint config with a newer policy engine. HTTP passthrough tunnels without endpoint `protocol` are also generation-bound so credential-injection-only keep-alive tunnels close after a reload before forwarding another request.
+
+Raw streams are connection-scoped and outside the L7 live-reload guarantee. This includes `tls: skip`, binary/non-HTTP CONNECT tunnels, SQL audit fallback passthrough, HTTP upgrades such as WebSocket after `101 Switching Protocols`, and already-forwarded long-lived response bodies such as SSE. Policy reloads affect the next connection or the next parsed HTTP request; they do not interrupt raw byte streams that have already moved outside the request parser.
 
 ### Hot reload
 
 Two reload methods exist:
 
 - **`reload(policy, data_yaml)`**: Builds a new engine from raw Rego + YAML strings and atomically replaces the inner engine. Used in tests and by the file-mode path.
-- **`reload_from_proto(proto)`**: Builds a new engine through the same validated pipeline as `from_proto()` -- proto-to-JSON conversion, L7 validation, access preset expansion -- then atomically swaps the inner `regorus::Engine`. On success, all subsequent `evaluate_network_action()` and `query_endpoint_config()` calls use the new policy. On failure (e.g., L7 validation errors), the previous engine is untouched (last-known-good behavior). This is the method used by the policy poll loop for live reloads in gRPC mode.
+- **`reload_from_proto(proto)`**: Builds a new engine through the same validated pipeline as `from_proto()` -- proto-to-JSON conversion, L7 validation, access preset expansion -- then atomically swaps the inner `regorus::Engine`. On success, all subsequent `evaluate_network_action()` and `query_endpoint_config()` calls use the new policy, and the engine generation increments so active L7 tunnels close before forwarding another request under stale state. On failure (e.g., L7 validation errors), the previous engine and generation are untouched (last-known-good behavior). This is the method used by the policy poll loop for live reloads in gRPC mode.
 
 Both methods hold the `Mutex` only for the final swap (`*engine = new_engine`), so evaluation is blocked for only the duration of a pointer-sized assignment.
 
@@ -962,23 +966,23 @@ flowchart TD
 
 After a CONNECT is allowed, the SSRF check passes, and the upstream TCP connection is established, the proxy determines how to handle the tunnel traffic. TLS detection is automatic — the proxy peeks the first bytes of the client stream to decide.
 
-1. **Query L7 config**: `query_l7_config()` asks the OPA engine for `matched_endpoint_config`. If the endpoint has a `protocol` field, parse it into `L7EndpointConfig`.
+1. **Query L7 route**: `query_l7_route_snapshot()` asks the OPA engine for `matched_endpoint_config` and the current policy generation. If the endpoint has a `protocol` field, parse it into a generation-bound `L7ConfigSnapshot`. If the endpoint has no `protocol`, retain the generation for HTTP passthrough keep-alive tunnels.
 
 2. **Check for `tls: skip`**: If the endpoint has `tls: skip`, bypass all auto-detection and relay raw bytes via `copy_bidirectional()`. This is the escape hatch for client-cert mTLS or non-standard protocols.
 
 3. **Peek and auto-detect**: Read up to 8 bytes from the client stream via `TcpStream::peek()`. Classify the traffic using `looks_like_tls()` (checks for TLS ClientHello record: byte 0 = `0x16`, bytes 1-2 = TLS version `0x03xx`) and `looks_like_http()` (checks for HTTP method prefix).
 
 4. **TLS detected** (`is_tls = true`):
    - Terminate TLS unconditionally via `tls_terminate_client()` + `tls_connect_upstream()`. This happens for all HTTPS endpoints, not just those with L7 config.
-   - If L7 config is present: clone the OPA engine (`clone_engine_for_tunnel()`), run `relay_with_inspection()` for per-request policy evaluation.
-   - If no L7 config: run `relay_passthrough_with_credentials()` — parses HTTP minimally to inject credentials (via `SecretResolver`) and log requests, but does not evaluate L7 OPA rules. This enables credential injection on all HTTPS endpoints without requiring `protocol` in the policy.
+   - If L7 config is present: clone the OPA engine for the captured generation (`clone_engine_for_tunnel(generation)`), run `relay_with_inspection()` for per-request policy evaluation. If the generation changed between config lookup and clone, close the tunnel before inspection.
+   - If no L7 config: run `relay_passthrough_with_credentials()` — parses HTTP minimally to inject credentials (via `SecretResolver`) and log requests, but does not evaluate L7 OPA rules. The passthrough relay is bound to the policy generation captured at connection setup and closes before forwarding another request after a reload. This enables credential injection on all HTTPS endpoints without requiring `protocol` in the policy.
    - If TLS state is not configured: fall back to raw `copy_bidirectional()` with a warning.
 
 5. **Plaintext HTTP detected** (`is_http = true`, `is_tls = false`):
-   - If L7 config present: clone OPA engine, run `relay_with_inspection()` directly on the plaintext streams.
-   - If no L7 config: run `relay_passthrough_with_credentials()` for credential injection and observability.
+   - If L7 config present: clone the OPA engine for the captured generation, run `relay_with_inspection()` directly on the plaintext streams.
+   - If no L7 config: run `relay_passthrough_with_credentials()` for credential injection and observability, with the same per-request generation guard.
 
-6. **Neither TLS nor HTTP**: Raw `copy_bidirectional()` tunnel (binary protocols, SSH-over-CONNECT, etc.).
+6. **Neither TLS nor HTTP**: Raw `copy_bidirectional()` tunnel (binary protocols, SSH-over-CONNECT, etc.). These raw streams are connection-scoped and continue until either side closes; live policy reload does not interrupt them.
 
 ```mermaid
 flowchart TD
@@ -1002,7 +1006,7 @@ flowchart TD
 
 **Files:** `crates/openshell-sandbox/src/l7/`
 
-The L7 subsystem inspects application-layer traffic within CONNECT tunnels. Instead of raw `copy_bidirectional`, each request is parsed, evaluated against OPA rules, and either forwarded or blocked.
+The L7 subsystem inspects application-layer traffic within CONNECT tunnels. Instead of raw `copy_bidirectional`, each request is parsed, evaluated against OPA rules, and either forwarded or blocked. The relay uses a generation-bound policy snapshot; after a successful policy reload, an existing L7 keep-alive tunnel closes before forwarding another request. Once an HTTP request has upgraded into a raw stream, or when a response body is a long-lived stream, that stream is connection-scoped and is not interrupted by L7 live reload.
 
 ### Architecture
 
@@ -1245,6 +1249,8 @@ Implements `L7Provider` for HTTP/1.1:
 6. If allowed (or audit mode): relay request to upstream via `relay_http_request_with_resolver()` (which rewrites all remaining credential placeholders in headers, query parameters, path segments, and Basic auth tokens) and relay the response back to client, then loop
 7. If denied in enforce mode: send 403 (using redacted target in the response body) and close the connection
 
+Before parsing, before evaluation, and before forwarding each request, the relay checks whether its captured policy generation still matches the shared engine generation. If not, it emits a denied OCSF network event and closes the tunnel without forwarding the request.
+
 ## Process Identity
 
 ### SHA256 TOFU (Trust-On-First-Use)

architecture/security-policy.md

@@ -203,7 +203,7 @@ The poll loop:
 2. Fetches the current policy via `GetSandboxSettings`, which returns the latest version, its policy payload, and a SHA-256 hash.
 3. Compares the returned version against the locally tracked `current_version`. If the server version is not greater, the loop sleeps and retries.
 4. On a new version, calls `OpaEngine::reload_from_proto()` which builds a complete new `regorus::Engine` through the same validated pipeline as the initial load (proto-to-JSON conversion, L7 validation, access preset expansion).
-5. If the new engine builds successfully, it atomically replaces the inner `Mutex<regorus::Engine>`. If it fails, the previous engine is untouched.
+5. If the new engine builds successfully, it atomically replaces the inner `Mutex<regorus::Engine>` and increments the policy generation. Active L7 keep-alive tunnels close before forwarding another request after they observe the new generation. If reload fails, the previous engine and generation are untouched.
 6. Reports success or failure back to the server via `ReportPolicyStatus`.
 
 See `crates/openshell-sandbox/src/grpc_client.rs` -- `CachedOpenShellClient`.
@@ -723,7 +723,11 @@ flowchart LR
 
 This is the single most important behavioral trigger in the policy language. An endpoint with no `protocol` field passes traffic opaquely after the L4 (CONNECT) check. Adding `protocol: rest` activates per-request HTTP parsing and policy evaluation inside the proxy.
 
-**Implementation path**: After L4 CONNECT is allowed, the proxy calls `query_l7_config()` which evaluates the Rego rule `data.openshell.sandbox.matched_endpoint_config`. This rule only matches endpoints that have a `protocol` field set (see `sandbox-policy.rego` line `ep.protocol`). If a config is returned, the proxy enters `relay_with_inspection()` instead of `copy_bidirectional()`. See `crates/openshell-sandbox/src/proxy.rs` -- `handle_tcp_connection()`.
+**Implementation path**: After L4 CONNECT is allowed, the proxy calls `query_l7_route_snapshot()` which evaluates the Rego rule `data.openshell.sandbox.matched_endpoint_config` and records the policy generation. If an endpoint `protocol` config is returned, the proxy enters `relay_with_inspection()` instead of `copy_bidirectional()`. See `crates/openshell-sandbox/src/proxy.rs` -- `handle_tcp_connection()`.
+
+For L7-inspected CONNECT tunnels, the proxy binds endpoint config and the per-tunnel policy engine clone to the policy generation observed at tunnel setup. If a live policy reload advances the generation, the relay closes the existing keep-alive tunnel before forwarding another request. HTTP passthrough tunnels without endpoint `protocol` use the same generation guard for parsed requests even though they do not evaluate L7 OPA rules. Clients should reconnect so the next request is evaluated under the current policy.
+
+Raw streams are connection-scoped and outside L7 live-reload guarantees. This includes endpoints with `tls: skip`, non-HTTP CONNECT payloads, SQL audit fallback passthrough, HTTP upgrades after `101 Switching Protocols`, and already-forwarded streaming response bodies such as SSE. A policy reload applies to the next connection or next parsed HTTP request; it does not terminate raw bytes already relayed outside the HTTP request parser.
 
 **Validation requirement**: When `protocol` is set, either `rules` or `access` must also be present. An endpoint with `protocol` but no rules/access is rejected at validation time because it would deny all traffic (no allow rules means nothing matches). See `crates/openshell-sandbox/src/l7/mod.rs` -- `validate_l7_policies()`.
 
@@ -758,9 +762,9 @@ If any condition fails, the proxy returns `403 Forbidden`.
 5. Resolves DNS and validates all IPs are private and within `allowed_ips`
 6. Connects to upstream
 7. Rewrites the request: absolute-form → origin-form (`GET /path HTTP/1.1`), strips hop-by-hop headers, adds `Via: 1.1 openshell-sandbox` and `Connection: close`
-8. Forwards the rewritten request, then relays bidirectionally using `tokio::io::copy_bidirectional` (supports chunked transfer, SSE streams, and other long-lived responses with no idle timeout)
+8. Relays the rewritten request and response through the shared guarded HTTP relay. This reuses the same request body framing, CL/TE rejection, credential rewrite fail-closed behavior, unsolicited `101` blocking, and policy-generation checks as CONNECT L7 HTTP.
 
-**V1 simplifications**: Forward proxy v1 injects `Connection: close` (no keep-alive). Every forward proxy connection handles exactly one request-response exchange. When an endpoint has L7 rules configured, the forward proxy evaluates the single request's method and path against L7 policy before forwarding.
+**V1 simplifications**: Forward proxy v1 injects `Connection: close` (no keep-alive). Every forward proxy connection handles exactly one request-response exchange. The request is bound to the policy generation used for the L4 allow decision and is checked again before upstream connect and request forwarding. When an endpoint has L7 rules configured, the forward proxy also evaluates the single request's method and path against L7 policy before forwarding.
 
 **Implementation**: See `crates/openshell-sandbox/src/proxy.rs` -- `handle_forward_proxy()`, `parse_proxy_uri()`, `rewrite_forward_request()`.
 
@@ -785,7 +789,7 @@ flowchart TD
     K -- No --> L["403 Forbidden"]
     K -- Yes --> M["TCP connect to upstream"]
     M --> N["Rewrite request to origin-form<br/>Add Via + Connection: close"]
-    N --> O["Forward request + copy_bidirectional"]
+    N --> O["Guarded HTTP relay"]
 ```
 
 #### Example: Forward Proxy Policy
@@ -836,7 +840,7 @@ TLS termination is automatic. The proxy peeks the first bytes of every CONNECT t
 
 **Certificate caching**: Per-hostname leaf certificates are cached (up to 256 entries, then the entire cache is cleared). See `crates/openshell-sandbox/src/l7/tls.rs` -- `CertCache`.
 
-**Credential injection**: When TLS is auto-terminated but no L7 policy is configured (no `protocol` field), the proxy enters a passthrough relay that rewrites credential placeholders in HTTP headers (via `SecretResolver`) and logs requests for observability, but does not evaluate L7 OPA rules. This means credential injection works on all HTTPS endpoints automatically.
+**Credential injection**: When TLS is auto-terminated but no L7 policy is configured (no `protocol` field), the proxy enters a passthrough relay that rewrites credential placeholders in HTTP headers (via `SecretResolver`) and logs requests for observability, but does not evaluate L7 OPA rules. The relay still closes parsed keep-alive HTTP tunnels after policy generation changes before forwarding another request. This means credential injection works on all HTTPS endpoints automatically.
 
 **Validation warnings**:
 
@@ -972,9 +976,11 @@ sequenceDiagram
     OPA-->>Proxy: allowed=true, matched_policy="api_policy"
     Proxy-->>Client: 200 Connection Established
 
-    Note over Proxy: Query L7 config for matched endpoint
-    Proxy->>OPA: query_endpoint_config(host, port, binary)
-    OPA-->>Proxy: {protocol: rest, enforcement: enforce}
+    Note over Proxy: Query L7 config and generation for matched endpoint
+    Proxy->>OPA: query_endpoint_config_with_generation(host, port, binary)
+    OPA-->>Proxy: {protocol: rest, enforcement: enforce}, generation=N
+    Proxy->>OPA: clone_engine_for_tunnel(generation=N)
+    OPA-->>Proxy: generation-bound tunnel evaluator
 
     Note over Proxy: Auto-detect TLS (peek first bytes)
     Note over Proxy: TLS ClientHello detected → terminate
@@ -988,9 +994,12 @@ sequenceDiagram
 
     loop Per HTTP request in tunnel
         Client->>Proxy: GET /repos/myorg/foo HTTP/1.1
+        Note over Proxy: Close if policy generation changed
         Note over Proxy: Parse HTTP request line + headers
+        Note over Proxy: Close if policy generation changed
         Proxy->>OPA: allow_request(method=GET, path=/repos/myorg/foo)
         OPA-->>Proxy: allowed=true
+        Note over Proxy: Close if policy generation changed
         Proxy->>Upstream: GET /repos/myorg/foo HTTP/1.1
         Upstream-->>Proxy: 200 OK (response body)
         Proxy-->>Client: 200 OK (response body)