feat: add warm pool feasibility study spec and planning artifacts

Brainstorm, specification, implementation plan, task breakdown, and
review guide for evaluating Agent Sandbox warm pooling on OpenShift.
Measures whether sandbox startup latency can be reduced from 8-12s
to under 2s using SandboxWarmPool/SandboxClaim extension CRDs.

Assisted-By: 🤖 Claude Code

Author: rhuss

.gitignore

@@ -229,3 +229,10 @@ scripts/lint-mermaid/node_modules/
 # Nix
 /result
 /result-*
+
+# spex: generated/local files (only constitution is committed)
+# Use worktree-specific pattern to avoid shadowing root .claude/ tracked files
+.claude/worktrees/*/.claude/
+**/.specify/**
+!**/.specify/memory/
+!**/.specify/memory/constitution.md

.specify/memory/constitution.md

@@ -0,0 +1,50 @@
+# [PROJECT_NAME] Constitution
+<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
+
+## Core Principles
+
+### [PRINCIPLE_1_NAME]
+<!-- Example: I. Library-First -->
+[PRINCIPLE_1_DESCRIPTION]
+<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
+
+### [PRINCIPLE_2_NAME]
+<!-- Example: II. CLI Interface -->
+[PRINCIPLE_2_DESCRIPTION]
+<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
+
+### [PRINCIPLE_3_NAME]
+<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
+[PRINCIPLE_3_DESCRIPTION]
+<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
+
+### [PRINCIPLE_4_NAME]
+<!-- Example: IV. Integration Testing -->
+[PRINCIPLE_4_DESCRIPTION]
+<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
+
+### [PRINCIPLE_5_NAME]
+<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
+[PRINCIPLE_5_DESCRIPTION]
+<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
+
+## [SECTION_2_NAME]
+<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
+
+[SECTION_2_CONTENT]
+<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
+
+## [SECTION_3_NAME]
+<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
+
+[SECTION_3_CONTENT]
+<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
+
+## Governance
+<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
+
+[GOVERNANCE_RULES]
+<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
+
+**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
+<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->

brainstorm/00-overview.md

@@ -0,0 +1,32 @@
+# Brainstorm Overview
+
+Last updated: 2026-07-09
+
+## Sessions
+
+| # | Date | Topic | Status | Spec | Issue |
+|---|------|-------|--------|------|-------|
+| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - |
+| 02 | 2026-07-09 | cluster-setup | active | - | - |
+| 03 | 2026-07-09 | warm-pool-measurements | active | - | - |
+| 04 | 2026-07-09 | results-and-recommendations | active | - | - |
+
+## Structure
+
+01 is the parent document defining the overall feasibility study. 02-04 are execution phases:
+
+- **02** depends on: nothing (first step)
+- **03** depends on: 02 (cluster must be running)
+- **04** depends on: 03 (measurements must be complete)
+
+## Open Threads
+
+- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02)
+- Does env var injection at claim time trigger cold start? (from #01, #03)
+- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02)
+- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03)
+- Should findings be posted to upstream agent-sandbox repo? (from #04)
+
+## Parked Ideas
+
+None.

brainstorm/01-warm-pool-feasibility.md

@@ -0,0 +1,67 @@
+# Brainstorm: Warm Pool Feasibility Study for OpenShell
+
+**Date:** 2026-07-09
+**Status:** active
+
+## Problem Framing
+
+OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable.
+
+The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`.
+
+This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture.
+
+## Approaches Considered
+
+### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell)
+
+Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead.
+
+- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required
+- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection
+
+### B: Measure OpenShell End-to-End with Code Changes
+
+Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency.
+
+- Pros: Realistic numbers that include all OpenShell overhead
+- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow
+
+### C: Layered Approach (A then B)
+
+Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally.
+
+- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data
+- Cons: More phases to execute
+
+## Decision
+
+**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency.
+
+## Key Requirements
+
+1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements
+2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs
+3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup
+4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control
+5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns)
+6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates
+7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration
+8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope.
+
+## Experiment Phases
+
+This study decomposes into three execution phases, each with its own brainstorm document:
+
+| Phase | Brainstorm | What it covers |
+|-------|------------|----------------|
+| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment |
+| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization |
+| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations |
+
+## Open Questions
+
+- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD?
+- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection?
+- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)?
+- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers?

brainstorm/02-cluster-setup.md

@@ -0,0 +1,78 @@
+# Brainstorm: Cluster Setup & OpenShell Deployment
+
+**Date:** 2026-07-09
+**Status:** active
+**Parent:** 01-warm-pool-feasibility
+
+## Problem Framing
+
+Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation.
+
+Three components must work together:
+1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+)
+2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim)
+3. OpenShell deployed and functional (can create cold-start sandboxes)
+
+## Approaches Considered
+
+### A: ROSA HCP with Upstream Agent Sandbox Manifests
+
+Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart.
+
+- Pros: Uses upstream directly, most current version, full control over version
+- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration
+
+### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview)
+
+Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart.
+
+- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path
+- Cons: Tech preview may not include extension CRDs yet, version may lag upstream
+
+### C: Both Paths Available
+
+Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them.
+
+- Pros: Best of both worlds, flexibility
+- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts
+
+## Decision
+
+**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available.
+
+## Key Requirements
+
+1. **ROSA HCP cluster** via `rosa:create` with the AAET profile
+   - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA)
+   - Region: us-east-2 (matches AAET profile subnets)
+   - Worker nodes: at least 3 for realistic scheduling behavior
+
+2. **Agent Sandbox installation**
+   - Red Hat Agent Sandbox operator from OperatorHub (if available)
+   - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim
+   - Verify all four CRDs are served: `kubectl api-resources | grep agents`
+
+3. **OpenShell deployment**
+   - Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility
+   - Run `deploy.sh` with default configuration
+   - Verify: `openshell status` shows Connected
+   - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works)
+
+4. **Image pre-pulling**
+   - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes
+   - This removes image pull latency from warm pool measurements
+   - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull
+
+5. **Validation checklist**
+   - Gateway pod is Running
+   - Agent Sandbox controller is Running
+   - Extension CRDs are registered
+   - Cold-start sandbox creation works end-to-end
+   - Images are pre-pulled on all nodes
+
+## Open Questions
+
+- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`.
+- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource?
+- Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes?
+- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously.