Warm Pool Feasibility Study [Spec + Experiments]

.gitignore

@@ -229,3 +229,13 @@ scripts/lint-mermaid/node_modules/
 # Nix
 /result
 /result-*
+
+# Warm pool experiment shared libraries (overrides Python lib/ pattern above)
+!experiments/lib/
+
+# spex: generated/local files (only constitution is committed)
+# Use worktree-specific pattern to avoid shadowing root .claude/ tracked files
+.claude/worktrees/*/.claude/
+**/.specify/**
+!**/.specify/memory/
+!**/.specify/memory/constitution.md

.specify/memory/constitution.md

@@ -0,0 +1,50 @@
+# [PROJECT_NAME] Constitution
+<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
+
+## Core Principles
+
+### [PRINCIPLE_1_NAME]
+<!-- Example: I. Library-First -->
+[PRINCIPLE_1_DESCRIPTION]
+<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
+
+### [PRINCIPLE_2_NAME]
+<!-- Example: II. CLI Interface -->
+[PRINCIPLE_2_DESCRIPTION]
+<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
+
+### [PRINCIPLE_3_NAME]
+<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
+[PRINCIPLE_3_DESCRIPTION]
+<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
+
+### [PRINCIPLE_4_NAME]
+<!-- Example: IV. Integration Testing -->
+[PRINCIPLE_4_DESCRIPTION]
+<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
+
+### [PRINCIPLE_5_NAME]
+<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
+[PRINCIPLE_5_DESCRIPTION]
+<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
+
+## [SECTION_2_NAME]
+<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
+
+[SECTION_2_CONTENT]
+<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
+
+## [SECTION_3_NAME]
+<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
+
+[SECTION_3_CONTENT]
+<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
+
+## Governance
+<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
+
+[GOVERNANCE_RULES]
+<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
+
+**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
+<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->

brainstorm/00-overview.md

@@ -0,0 +1,32 @@
+# Brainstorm Overview
+
+Last updated: 2026-07-09
+
+## Sessions
+
+| # | Date | Topic | Status | Spec | Issue |
+|---|------|-------|--------|------|-------|
+| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - |
+| 02 | 2026-07-09 | cluster-setup | active | - | - |
+| 03 | 2026-07-09 | warm-pool-measurements | active | - | - |
+| 04 | 2026-07-09 | results-and-recommendations | active | - | - |
+
+## Structure
+
+01 is the parent document defining the overall feasibility study. 02-04 are execution phases:
+
+- **02** depends on: nothing (first step)
+- **03** depends on: 02 (cluster must be running)
+- **04** depends on: 03 (measurements must be complete)
+
+## Open Threads
+
+- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02)
+- Does env var injection at claim time trigger cold start? (from #01, #03)
+- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02)
+- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03)
+- Should findings be posted to upstream agent-sandbox repo? (from #04)
+
+## Parked Ideas
+
+None.

brainstorm/01-warm-pool-feasibility.md

@@ -0,0 +1,67 @@
+# Brainstorm: Warm Pool Feasibility Study for OpenShell
+
+**Date:** 2026-07-09
+**Status:** active
+
+## Problem Framing
+
+OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable.
+
+The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`.
+
+This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture.
+
+## Approaches Considered
+
+### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell)
+
+Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead.
+
+- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required
+- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection
+
+### B: Measure OpenShell End-to-End with Code Changes
+
+Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency.
+
+- Pros: Realistic numbers that include all OpenShell overhead
+- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow
+
+### C: Layered Approach (A then B)
+
+Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally.
+
+- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data
+- Cons: More phases to execute
+
+## Decision
+
+**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency.
+
+## Key Requirements
+
+1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements
+2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs
+3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup
+4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control
+5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns)
+6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates
+7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration
+8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope.
+
+## Experiment Phases
+
+This study decomposes into three execution phases, each with its own brainstorm document:
+
+| Phase | Brainstorm | What it covers |
+|-------|------------|----------------|
+| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment |
+| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization |
+| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations |
+
+## Open Questions
+
+- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD?
+- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection?
+- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)?
+- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers?

brainstorm/02-cluster-setup.md

@@ -0,0 +1,78 @@
+# Brainstorm: Cluster Setup & OpenShell Deployment
+
+**Date:** 2026-07-09
+**Status:** active
+**Parent:** 01-warm-pool-feasibility
+
+## Problem Framing
+
+Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation.
+
+Three components must work together:
+1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+)
+2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim)
+3. OpenShell deployed and functional (can create cold-start sandboxes)
+
+## Approaches Considered
+
+### A: ROSA HCP with Upstream Agent Sandbox Manifests
+
+Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart.
+
+- Pros: Uses upstream directly, most current version, full control over version
+- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration
+
+### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview)
+
+Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart.
+
+- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path
+- Cons: Tech preview may not include extension CRDs yet, version may lag upstream
+
+### C: Both Paths Available
+
+Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them.
+
+- Pros: Best of both worlds, flexibility
+- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts
+
+## Decision
+
+**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available.
+
+## Key Requirements
+
+1. **ROSA HCP cluster** via `rosa:create` with the AAET profile
+   - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA)
+   - Region: us-east-2 (matches AAET profile subnets)
+   - Worker nodes: at least 3 for realistic scheduling behavior
+
+2. **Agent Sandbox installation**
+   - Red Hat Agent Sandbox operator from OperatorHub (if available)
+   - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim
+   - Verify all four CRDs are served: `kubectl api-resources | grep agents`
+
+3. **OpenShell deployment**
+   - Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility
+   - Run `deploy.sh` with default configuration
+   - Verify: `openshell status` shows Connected
+   - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works)
+
+4. **Image pre-pulling**
+   - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes
+   - This removes image pull latency from warm pool measurements
+   - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull
+
+5. **Validation checklist**
+   - Gateway pod is Running
+   - Agent Sandbox controller is Running
+   - Extension CRDs are registered
+   - Cold-start sandbox creation works end-to-end
+   - Images are pre-pulled on all nodes
+
+## Open Questions
+
+- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`.
+- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource?
+- Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes?
+- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously.