diff --git a/.gitignore b/.gitignore index 34b24bf768..93cfdebe37 100644 --- a/.gitignore +++ b/.gitignore @@ -229,3 +229,13 @@ scripts/lint-mermaid/node_modules/ # Nix /result /result-* + +# Warm pool experiment shared libraries (overrides Python lib/ pattern above) +!experiments/lib/ + +# spex: generated/local files (only constitution is committed) +# Use worktree-specific pattern to avoid shadowing root .claude/ tracked files +.claude/worktrees/*/.claude/ +**/.specify/** +!**/.specify/memory/ +!**/.specify/memory/constitution.md diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md new file mode 100644 index 0000000000..a4670ff469 --- /dev/null +++ b/.specify/memory/constitution.md @@ -0,0 +1,50 @@ +# [PROJECT_NAME] Constitution + + +## Core Principles + +### [PRINCIPLE_1_NAME] + +[PRINCIPLE_1_DESCRIPTION] + + +### [PRINCIPLE_2_NAME] + +[PRINCIPLE_2_DESCRIPTION] + + +### [PRINCIPLE_3_NAME] + +[PRINCIPLE_3_DESCRIPTION] + + +### [PRINCIPLE_4_NAME] + +[PRINCIPLE_4_DESCRIPTION] + + +### [PRINCIPLE_5_NAME] + +[PRINCIPLE_5_DESCRIPTION] + + +## [SECTION_2_NAME] + + +[SECTION_2_CONTENT] + + +## [SECTION_3_NAME] + + +[SECTION_3_CONTENT] + + +## Governance + + +[GOVERNANCE_RULES] + + +**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE] + diff --git a/brainstorm/00-overview.md b/brainstorm/00-overview.md new file mode 100644 index 0000000000..1c5497fd5e --- /dev/null +++ b/brainstorm/00-overview.md @@ -0,0 +1,32 @@ +# Brainstorm Overview + +Last updated: 2026-07-09 + +## Sessions + +| # | Date | Topic | Status | Spec | Issue | +|---|------|-------|--------|------|-------| +| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - | +| 02 | 2026-07-09 | cluster-setup | active | - | - | +| 03 | 2026-07-09 | warm-pool-measurements | active | - | - | +| 04 | 2026-07-09 | results-and-recommendations | active | - | - | + +## Structure + +01 is the parent document defining the overall feasibility study. 02-04 are execution phases: + +- **02** depends on: nothing (first step) +- **03** depends on: 02 (cluster must be running) +- **04** depends on: 03 (measurements must be complete) + +## Open Threads + +- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02) +- Does env var injection at claim time trigger cold start? (from #01, #03) +- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02) +- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03) +- Should findings be posted to upstream agent-sandbox repo? (from #04) + +## Parked Ideas + +None. diff --git a/brainstorm/01-warm-pool-feasibility.md b/brainstorm/01-warm-pool-feasibility.md new file mode 100644 index 0000000000..11932c80f2 --- /dev/null +++ b/brainstorm/01-warm-pool-feasibility.md @@ -0,0 +1,67 @@ +# Brainstorm: Warm Pool Feasibility Study for OpenShell + +**Date:** 2026-07-09 +**Status:** active + +## Problem Framing + +OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable. + +The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`. + +This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture. + +## Approaches Considered + +### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell) + +Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead. + +- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required +- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection + +### B: Measure OpenShell End-to-End with Code Changes + +Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency. + +- Pros: Realistic numbers that include all OpenShell overhead +- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow + +### C: Layered Approach (A then B) + +Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally. + +- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data +- Cons: More phases to execute + +## Decision + +**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency. + +## Key Requirements + +1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements +2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs +3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup +4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control +5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns) +6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates +7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration +8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope. + +## Experiment Phases + +This study decomposes into three execution phases, each with its own brainstorm document: + +| Phase | Brainstorm | What it covers | +|-------|------------|----------------| +| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment | +| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization | +| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations | + +## Open Questions + +- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD? +- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection? +- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)? +- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers? diff --git a/brainstorm/02-cluster-setup.md b/brainstorm/02-cluster-setup.md new file mode 100644 index 0000000000..af0580dc38 --- /dev/null +++ b/brainstorm/02-cluster-setup.md @@ -0,0 +1,78 @@ +# Brainstorm: Cluster Setup & OpenShell Deployment + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation. + +Three components must work together: +1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+) +2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) +3. OpenShell deployed and functional (can create cold-start sandboxes) + +## Approaches Considered + +### A: ROSA HCP with Upstream Agent Sandbox Manifests + +Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart. + +- Pros: Uses upstream directly, most current version, full control over version +- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration + +### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview) + +Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart. + +- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path +- Cons: Tech preview may not include extension CRDs yet, version may lag upstream + +### C: Both Paths Available + +Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them. + +- Pros: Best of both worlds, flexibility +- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts + +## Decision + +**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available. + +## Key Requirements + +1. **ROSA HCP cluster** via `rosa:create` with the AAET profile + - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA) + - Region: us-east-2 (matches AAET profile subnets) + - Worker nodes: at least 3 for realistic scheduling behavior + +2. **Agent Sandbox installation** + - Red Hat Agent Sandbox operator from OperatorHub (if available) + - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim + - Verify all four CRDs are served: `kubectl api-resources | grep agents` + +3. **OpenShell deployment** + - Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility + - Run `deploy.sh` with default configuration + - Verify: `openshell status` shows Connected + - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works) + +4. **Image pre-pulling** + - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes + - This removes image pull latency from warm pool measurements + - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull + +5. **Validation checklist** + - Gateway pod is Running + - Agent Sandbox controller is Running + - Extension CRDs are registered + - Cold-start sandbox creation works end-to-end + - Images are pre-pulled on all nodes + +## Open Questions + +- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`. +- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource? +- Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes? +- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously. diff --git a/brainstorm/03-warm-pool-measurements.md b/brainstorm/03-warm-pool-measurements.md new file mode 100644 index 0000000000..42b24cfef3 --- /dev/null +++ b/brainstorm/03-warm-pool-measurements.md @@ -0,0 +1,175 @@ +# Brainstorm: Baseline & Warm Pool Measurements + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +With the cluster running, we need a structured measurement plan that answers: how fast can we get a sandbox with warm pooling, and what are the bottlenecks? The measurements must cover cold-start baseline (control), raw Agent Sandbox warm pooling (no OpenShell), and progressively more realistic configurations that approximate what OpenShell would need. + +The key unknowns are: +- What is the actual cold-start latency breakdown on OpenShift? +- Does warm pooling deliver sub-2s claim-to-ready? +- Is the readiness probe interval the dominant bottleneck, and can Pod Readiness Gates (KEP-580) or Knative-style readiness wrapping eliminate it? +- Can env vars be injected at claim time without triggering cold start? + +## Approaches Considered + +### A: Simple Before/After + +Measure cold start, then warm pool, compare. Two data points. + +- Pros: Fastest to execute +- Cons: No insight into what drives the latency, can't identify optimization levers + +### B: Phase Breakdown with Configuration Matrix + +Measure cold start with per-phase timestamps. Then measure warm pooling across a matrix of configurations (probe intervals, readiness gates, sidecar patterns, env var injection). + +- Pros: Identifies specific bottlenecks, tests optimization hypotheses, actionable data +- Cons: More experiments to run, needs a measurement script + +### C: Full Benchmark Suite with Statistical Rigor + +N=50+ runs per configuration, p50/p90/p99, automated benchmark harness, CSV output for analysis. + +- Pros: Publication-quality data, statistically meaningful +- Cons: Overkill for a feasibility study, takes days + +## Decision + +**Approach B: Phase breakdown with configuration matrix.** N=10-20 runs per configuration is enough to see the pattern. We need per-phase timestamps to identify bottlenecks, and we need the configuration matrix to test our optimization hypotheses (readiness gates, sidecar patterns). A simple shell script with `kubectl` timestamps is sufficient. + +## Experiment Design + +### Experiment 1: Cold-Start Baseline (Control) + +Measure OpenShell sandbox creation latency without pooling. Captures the current state. + +**What to measure:** +- Total time: `openshell sandbox create` to sandbox Ready +- Phase breakdown using `kubectl get events` and pod timestamps: + - API call to pod Scheduled + - Scheduled to image pulled (should be 0 with pre-pulled images) + - Image pulled to init containers complete + - Init containers complete to supervisor Ready + - Supervisor Ready to SSH available + +**Runs:** N=10 with pre-pulled images, N=5 without (to quantify image pull impact) + +### Experiment 2: Vanilla Agent Sandbox Warm Pool (No OpenShell) + +Measure raw Agent Sandbox warm pooling without OpenShell. This isolates Kubernetes overhead from OpenShell overhead. + +**Setup:** +```yaml +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: base-template +spec: + sandbox: + spec: + containers: + - name: agent + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: base-pool +spec: + templateRef: + name: base-template + replicas: 5 +``` + +**What to measure:** +- Pool fill time: SandboxWarmPool created to all 5 replicas Ready +- Claim-to-ready time: SandboxClaim created to adopted Sandbox transitioning to Ready +- Claim-to-ready with default readiness probe (10s periodSeconds) +- Claim-to-ready with aggressive readiness probe (1s periodSeconds) + +**Runs:** N=10 per configuration + +### Experiment 3: Pod Readiness Gates (KEP-580) + +Test whether Pod Readiness Gates can replace polling-based readiness for faster claim-to-ready. + +**How it works:** +- Add a custom ReadinessGate condition (e.g., `sandbox.openshell.io/claimed`) to the pod template +- Warm-pooled pods start with the condition missing (defaults to False, pod is NotReady) +- On claim, an external controller sets the condition to True via a pod status patch +- Pod transitions to Ready immediately (no probe interval wait) + +**What to measure:** +- Time from condition patch to pod Ready status +- Compare with probe-based readiness at 1s and 10s intervals + +**Runs:** N=10 + +**KEP-580 status:** GA since Kubernetes 1.14. Available on all OpenShift versions. No feature gate needed. + +### Experiment 4: Sidecar Readiness Pattern + +Test the Knative-style pattern where a sidecar controls pod readiness. + +**How it works:** +- Define a supervisor-like sidecar (init container with `restartPolicy: Always`) +- Sidecar starts, runs a readiness HTTP endpoint that returns 503 (not ready) +- On claim, inject a signal (touch a file in a shared emptyDir, or set an env var) +- Sidecar detects the signal, flips readiness to 200 +- Pod transitions to Ready + +**What to measure:** +- Time from signal injection to sidecar readiness flip +- Time from sidecar readiness flip to pod Ready +- End-to-end claim-to-ready with sidecar pattern + +**Runs:** N=10 + +**KEP-753 status (native sidecars):** GA since Kubernetes 1.33 (April 2025). Requires OpenShift 4.20+. Sidecar readiness probes contribute to pod readiness. + +### Experiment 5: Env Var Injection at Claim Time + +Test whether SandboxClaim env var injection works without forcing cold start. + +**Setup:** +- SandboxTemplate with `envVarsInjectionPolicy: Allowed` +- SandboxClaim with env vars simulating OpenShell identity (OPENSHELL_SANDBOX_ID, OPENSHELL_ENDPOINT) + +**What to measure:** +- Does the claim adopt a warm sandbox, or does it trigger cold start? +- If warm adoption works: claim-to-ready latency +- If cold start is triggered: document this as a constraint + +**Runs:** N=5 + +### Experiment 6: Combined (Sidecar + Readiness Gates + Env Var Injection) + +Combine the best-performing readiness pattern with env var injection. This approximates what a real OpenShell integration would look like. + +**What to measure:** +- End-to-end claim-to-ready with the full stack +- Compare with cold-start baseline + +**Runs:** N=10 + +## Measurement Script + +A shell script that: +1. Creates a SandboxClaim with `kubectl apply` +2. Records the creation timestamp +3. Watches the pod status until Ready (or timeout) +4. Records the Ready timestamp +5. Calculates the delta +6. Collects pod events for phase breakdown +7. Outputs CSV: run_number, config, create_ts, ready_ts, delta_ms, phases + +## Open Questions + +- Can we use `kubectl wait --for=condition=Ready` with millisecond precision, or do we need a custom watcher? +- For the sidecar experiment, what's the simplest possible sidecar binary? A Go binary that listens on :8080 and watches for a file in /tmp/signal/? +- How does pool replenishment affect back-to-back claim latency? Should we measure burst patterns (claim 5 sandboxes simultaneously)? +- What happens when the pool is exhausted? Does SandboxClaim fall back to cold start or stay Pending? diff --git a/brainstorm/04-results-and-recommendations.md b/brainstorm/04-results-and-recommendations.md new file mode 100644 index 0000000000..7df557618c --- /dev/null +++ b/brainstorm/04-results-and-recommendations.md @@ -0,0 +1,113 @@ +# Brainstorm: Results Document & OpenShell Recommendations + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +After the measurements are complete, we need to synthesize findings into a document that serves two audiences: + +1. **The OpenShell core team**: What should OpenShell change to support warm pooling? Which approach from Issue #2157 is backed by data? What are the architectural constraints? +2. **The Red Hat integration team**: Is the upstream Agent Sandbox warm pooling viable for enterprise OpenShift deployments? What gaps exist in the Red Hat tech preview? + +The document should be concrete enough to inform Issue #2157's design decisions. + +## Approaches Considered + +### A: Internal Technical Report + +A detailed technical document with raw data, analysis, and recommendations. Shared internally and referenced in GitHub issue comments. + +- Pros: Complete record, referenceable, can share selectively +- Cons: May be too dense for quick consumption + +### B: GitHub Issue Comment + Summary Doc + +Post key findings as a comment on Issue #2157 with a link to a detailed report. The comment is the "executive summary," the report is the appendix. + +- Pros: Directly visible to the upstream community, invites discussion, keeps the issue alive +- Cons: GitHub comments are hard to update as understanding evolves + +### C: Both (Report + Issue Comment) + +Write the full report as a document, then distill key findings into a GitHub comment on #2157. + +- Pros: Best of both, full technical depth plus community visibility +- Cons: Two artifacts to maintain + +## Decision + +**Approach C: Full report plus GitHub comment.** The report is published as a standalone RFC in `rfc/` (see clarification in spec.md). A distilled comment on #2157 can be posted as a follow-up step to share findings with the upstream community. + +## Report Structure + +### 1. Executive Summary +- One paragraph: can warm pooling hit sub-2s on OpenShift? +- Key finding: what is the dominant bottleneck? +- Recommendation: which integration path for OpenShell? + +### 2. Experiment Setup +- Cluster configuration (OpenShift version, K8s version, node count, instance type) +- Agent Sandbox version and CRDs installed +- OpenShell version and deployment method +- Image pre-pull status + +### 3. Results + +#### Cold-Start Baseline +- Table: phase breakdown with p50/p90 latencies +- Chart: latency distribution + +#### Warm Pool Results +- Table: configuration matrix with claim-to-ready latencies +- Comparison: default probes vs. aggressive probes vs. readiness gates vs. sidecar pattern +- Finding: which configuration achieves the target? + +#### Health Check Analysis +- Readiness probe interval impact (10s vs. 1s) +- Pod Readiness Gates (KEP-580) performance +- Sidecar readiness pattern performance +- Knative-style comparison + +#### Env Var Injection +- Does claim-time injection work without cold start? +- Which injection policy is needed? +- What can be injected (env vars) vs. what needs another mechanism (TLS certs, files)? + +### 4. OpenShell Architecture Recommendations + +Map findings to OpenShell's architecture: + +- **Kubernetes driver changes:** What needs to change in `driver.rs` to support SandboxClaim creation? +- **Supervisor changes:** Should the supervisor become a native sidecar with late-bind identity? +- **Gateway store changes:** How should the gateway handle sandbox records for warm-pooled sandboxes? +- **Identity binding:** Which mechanism works (env var injection, volume projection, API call)? +- **Configuration surface:** Where should warm pool configuration live (driver config, workspace admin, operator)? +- **Issue #2157 recommendation:** Which of the four alternatives from the issue is best supported by data? +- **Issue #1447 comparison:** Native pool vs. upstream CRDs, with data backing + +### 5. Gaps and Risks +- Missing features in the Agent Sandbox extension API (e.g., volumeClaimTemplates, Issue #453) +- Red Hat tech preview coverage gaps +- KEP-753 availability on the target OpenShift version +- Pool replenishment under burst load +- Identity isolation between warm slot reuse + +### 6. Next Steps +- Concrete list of upstream contributions (issues, PRs, RFCs) +- Internal work items for the next sprint +- Follow-up actions for stakeholder discussions + +## Key Requirements + +1. **Report saved to the configured notes vault** with a date prefix +2. **RFC in `rfc/`** as the canonical, version-controlled results document +3. **No Google Drive links** in any public-facing artifacts +4. **Data tables with raw numbers**, not just qualitative assessments +5. **Clear recommendation** for the OpenShell core team, not just "it depends" + +## Open Questions + +- Should we also post findings to the upstream Agent Sandbox repo (e.g., as a discussion or issue comment on Issue #390)? +- Should the report include a proposed RFC outline for OpenShell warm pooling, or is that a separate step? diff --git a/experiments/lib/common.sh b/experiments/lib/common.sh new file mode 100644 index 0000000000..ccd7eeafc2 --- /dev/null +++ b/experiments/lib/common.sh @@ -0,0 +1,212 @@ +#!/usr/bin/env bash +# Shared measurement functions for warm pool experiments. +# Source this file from experiment scripts: source "$(dirname "$0")/../lib/common.sh" + +set -euo pipefail + +EXPERIMENT_ID="${EXPERIMENT_ID:-unknown}" +RESULTS_DIR="${RESULTS_DIR:-experiments/results}" + +CLEANUP_RESOURCES=() +register_cleanup() { CLEANUP_RESOURCES+=("$1"); } +deregister_cleanup() { CLEANUP_RESOURCES=("${CLEANUP_RESOURCES[@]/$1}"); } +_cleanup_on_exit() { + for res in "${CLEANUP_RESOURCES[@]}"; do + [[ -z "$res" ]] && continue + log "Cleaning up: $res" + kubectl delete $res --ignore-not-found --wait=false 2>/dev/null || true + done +} +trap _cleanup_on_exit EXIT + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" +} + +ensure_results_dir() { + mkdir -p "$RESULTS_DIR" +} + +capture_timestamp() { + if command -v gdate &>/dev/null; then + gdate +%s%N + elif date +%s%N | grep -qv N; then + date +%s%N + else + # Fallback: seconds with 000000000 appended (no nanosecond support) + echo "$(date +%s)000000000" + fi +} + +write_csv_header() { + local file="$1" + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,init_ms,supervisor_ms,ssh_ms" > "$file" +} + +write_csv_row() { + local file="$1" + local run="$2" + local config="$3" + local create_ts="$4" + local ready_ts="$5" + local delta_ms="$6" + local pod="$7" + local row_status="$8" + local scheduled_ms="${9:-}" + local pulled_ms="${10:-}" + local init_ms="${11:-}" + local supervisor_ms="${12:-}" + local ssh_ms="${13:-}" + echo "${run},${EXPERIMENT_ID},${config},${create_ts},${ready_ts},${delta_ms},${pod},${row_status},${scheduled_ms},${pulled_ms},${init_ms},${supervisor_ms},${ssh_ms}" >> "$file" +} + +_iso_to_epoch() { + local ts="$1" + if [[ -z "$ts" || "$ts" == "null" ]]; then + echo "" + return + fi + if command -v gdate &>/dev/null; then + gdate -d "$ts" +%s 2>/dev/null || echo "" + else + date -d "$ts" +%s 2>/dev/null || echo "" + fi +} + +_event_ts() { + local events_json="$1" + local reason="$2" + local pick="${3:-first}" + local selector=".reason == \"$reason\"" + if [[ "$reason" == *"|"* ]]; then + local r1="${reason%%|*}" + local r2="${reason#*|}" + selector=".reason == \"$r1\" or .reason == \"$r2\"" + fi + if [[ "$pick" == "last" ]]; then + echo "$events_json" | jq -r "[.items[] | select($selector)] | last | (.lastTimestamp // .eventTime // null)" 2>/dev/null + else + echo "$events_json" | jq -r "[.items[] | select($selector)] | first | (.lastTimestamp // .eventTime // null)" 2>/dev/null + fi +} + +extract_phase_deltas() { + local events_json="$1" + local create_ts="$2" + local create_s=$((create_ts / 1000000000)) + + local sched_ts pull_ts init_ts start_ts + sched_ts=$(_event_ts "$events_json" "Scheduled" "first") + pull_ts=$(_event_ts "$events_json" "Pulled|Pulling" "last") + init_ts=$(_event_ts "$events_json" "Created" "first") + start_ts=$(_event_ts "$events_json" "Started" "last") + + local sched_ms="" pull_ms="" init_ms="" start_ms="" + + local epoch + epoch=$(_iso_to_epoch "$sched_ts") + [[ -n "$epoch" ]] && sched_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$pull_ts") + [[ -n "$epoch" ]] && pull_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$init_ts") + [[ -n "$epoch" ]] && init_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$start_ts") + [[ -n "$epoch" ]] && start_ms=$(( (epoch - create_s) * 1000 )) + + echo "${sched_ms},${pull_ms},${init_ms},${start_ms}," +} + +collect_pod_events() { + local pod="$1" + local ns="${2:-default}" + kubectl get events \ + --namespace="$ns" \ + --field-selector="involvedObject.name=$pod,involvedObject.kind=Pod" \ + --sort-by='.lastTimestamp' \ + -o json 2>/dev/null || echo '{"items":[]}' +} + +detect_adoption() { + local pod_name="$1" + local ns="${2:-default}" + if [[ -z "$pod_name" ]]; then + echo "unknown" + return + fi + + local events_json + events_json=$(collect_pod_events "$pod_name" "$ns") + + local scheduled_count + scheduled_count=$(echo "$events_json" | \ + jq '[.items[] | select(.reason == "Scheduled")] | length' 2>/dev/null || echo "0") + + local pulled_count + pulled_count=$(echo "$events_json" | \ + jq '[.items[] | select(.reason == "Pulling" or .reason == "Pulled")] | length' 2>/dev/null || echo "0") + + if (( scheduled_count == 0 && pulled_count == 0 )); then + echo "warm-adopted" + else + echo "cold-fallback" + fi +} + +compute_stats() { + local csv_file="$1" + if [[ ! -f "$csv_file" ]]; then + log "ERROR: CSV file not found: $csv_file" + return 1 + fi + + awk -F',' ' + NR == 1 { next } + $8 != "ok" { skipped++; next } + $6 == "" { skipped++; next } + { + v = $6 + 0 + n++ + vals[n] = v + sum += v + if (n == 1 || v < min) min = v + if (n == 1 || v > max) max = v + } + END { + if (n == 0) { + print "No data rows found." + exit 1 + } + + # Sort values (insertion sort) + for (i = 2; i <= n; i++) { + key = vals[i] + j = i - 1 + while (j >= 1 && vals[j] > key) { + vals[j+1] = vals[j] + j-- + } + vals[j+1] = key + } + + p50_idx = int(n * 0.5 + 0.5) + p90_idx = int(n * 0.9 + 0.5) + if (p50_idx < 1) p50_idx = 1 + if (p90_idx < 1) p90_idx = 1 + if (p50_idx > n) p50_idx = n + if (p90_idx > n) p90_idx = n + + mean = sum / n + printf "\n--- Statistics (%d samples) ---\n", n + printf " min: %10.1f ms\n", min + printf " max: %10.1f ms\n", max + printf " mean: %10.1f ms\n", mean + printf " p50: %10.1f ms\n", vals[p50_idx] + printf " p90: %10.1f ms\n", vals[p90_idx] + printf "-------------------------------\n" + if (skipped > 0) printf " (%d non-ok rows excluded)\n", skipped + } + ' "$csv_file" +} diff --git a/experiments/lib/wait-ready.sh b/experiments/lib/wait-ready.sh new file mode 100644 index 0000000000..8433e536af --- /dev/null +++ b/experiments/lib/wait-ready.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Pod and resource readiness wait functions. +# Source this file from experiment scripts: source "$(dirname "$0")/../lib/wait-ready.sh" + +set -euo pipefail + +wait_for_ready() { + local resource="$1" + local timeout_s="${2:-120}" + kubectl wait --for=condition=Ready "$resource" --timeout="${timeout_s}s" 2>/dev/null +} + +wait_for_pod_ready() { + local pod="$1" + local ns="${2:-default}" + local timeout_s="${3:-120}" + local elapsed=0 + + while [[ $elapsed -lt $timeout_s ]]; do + local status + status=$(kubectl get pod "$pod" \ + --namespace="$ns" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local phase + phase=$(kubectl get pod "$pod" \ + --namespace="$ns" \ + -o jsonpath='{.status.phase}' \ + 2>/dev/null) || true + + if [[ "$phase" == "Failed" || "$phase" == "Succeeded" ]]; then + echo "Pod $pod terminated with phase: $phase" >&2 + return 1 + fi + + if (( elapsed % 10 == 0 && elapsed > 0 )); then + echo "Waiting for pod $pod to be Ready... (${elapsed}s/${timeout_s}s, phase: ${phase:-unknown})" >&2 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + + echo "Timeout: pod $pod not Ready after ${timeout_s}s" >&2 + return 1 +} + +wait_for_sandbox_ready() { + local name="$1" + local ns="${2:-default}" + local timeout_s="${3:-120}" + local elapsed=0 + + while [[ $elapsed -lt $timeout_s ]]; do + local status + status=$(kubectl get sandbox "$name" \ + --namespace="$ns" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + if (( elapsed % 10 == 0 && elapsed > 0 )); then + echo "Waiting for sandbox $name to be Ready... (${elapsed}s/${timeout_s}s)" >&2 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + + echo "Timeout: sandbox $name not Ready after ${timeout_s}s" >&2 + return 1 +} diff --git a/experiments/manifests/image-prepull-daemonset.yaml b/experiments/manifests/image-prepull-daemonset.yaml new file mode 100644 index 0000000000..3a192dbfee --- /dev/null +++ b/experiments/manifests/image-prepull-daemonset.yaml @@ -0,0 +1,34 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: image-prepull + namespace: default + labels: + app.kubernetes.io/name: image-prepull + app.kubernetes.io/part-of: openshell-warm-pool +spec: + selector: + matchLabels: + app.kubernetes.io/name: image-prepull + template: + metadata: + labels: + app.kubernetes.io/name: image-prepull + spec: + initContainers: + - name: pull-sandbox + image: ghcr.io/nvidia/openshell/sandbox:latest + imagePullPolicy: IfNotPresent + command: ["/bin/true"] + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: + requests: + cpu: "1m" + memory: "4Mi" + limits: + cpu: "1m" + memory: "4Mi" + tolerations: + - operator: Exists diff --git a/experiments/manifests/readiness-gate-pod.yaml b/experiments/manifests/readiness-gate-pod.yaml new file mode 100644 index 0000000000..fcf8d77ca7 --- /dev/null +++ b/experiments/manifests/readiness-gate-pod.yaml @@ -0,0 +1,16 @@ +apiVersion: v1 +kind: Pod +metadata: + name: readiness-gate-test-PLACEHOLDER +spec: + readinessGates: + - conditionType: sandbox.openshell.io/claimed + containers: + - name: sandbox + image: ghcr.io/nvidia/openshell/sandbox:latest + ports: + - containerPort: 2222 + resources: + requests: + cpu: "500m" + memory: "512Mi" diff --git a/experiments/manifests/sandbox-claim.yaml b/experiments/manifests/sandbox-claim.yaml new file mode 100644 index 0000000000..8f612a63a4 --- /dev/null +++ b/experiments/manifests/sandbox-claim.yaml @@ -0,0 +1,17 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxClaim +metadata: + name: claim-PLACEHOLDER +spec: + warmPoolRef: + name: openshell-warm-pool + +# Variant with env var injection (requires envVarsInjectionPolicy: Allowed on template): +# spec: +# warmPoolRef: +# name: openshell-warm-pool +# env: +# - name: AGENT_ID +# value: "agent-001" +# - name: SESSION_TOKEN +# value: "tok-xxx" diff --git a/experiments/manifests/sandbox-template.yaml b/experiments/manifests/sandbox-template.yaml new file mode 100644 index 0000000000..69d9d5e378 --- /dev/null +++ b/experiments/manifests/sandbox-template.yaml @@ -0,0 +1,25 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: openshell-warm +spec: + podTemplateSpec: + spec: + containers: + - name: sandbox + image: ghcr.io/nvidia/openshell/sandbox:latest + ports: + - containerPort: 2222 + name: ssh + readinessProbe: + tcpSocket: + port: 2222 + initialDelaySeconds: 2 + periodSeconds: 10 + resources: + requests: + cpu: "500m" + memory: "512Mi" + limits: + cpu: "2" + memory: "2Gi" diff --git a/experiments/manifests/sidecar-readiness.yaml b/experiments/manifests/sidecar-readiness.yaml new file mode 100644 index 0000000000..e6419411ea --- /dev/null +++ b/experiments/manifests/sidecar-readiness.yaml @@ -0,0 +1,34 @@ +apiVersion: v1 +kind: Pod +metadata: + name: sidecar-readiness-test-PLACEHOLDER +spec: + initContainers: + - name: readiness-sidecar + image: ghcr.io/nvidia/openshell/sidecar-readiness:latest + restartPolicy: Always + ports: + - containerPort: 8080 + readinessProbe: + httpGet: + path: /healthz + port: 8080 + periodSeconds: 1 + volumeMounts: + - name: signal + mountPath: /tmp/signal + containers: + - name: sandbox + image: ghcr.io/nvidia/openshell/sandbox:latest + ports: + - containerPort: 2222 + volumeMounts: + - name: signal + mountPath: /tmp/signal + resources: + requests: + cpu: "500m" + memory: "512Mi" + volumes: + - name: signal + emptyDir: {} diff --git a/experiments/manifests/warm-pool.yaml b/experiments/manifests/warm-pool.yaml new file mode 100644 index 0000000000..a605b10f84 --- /dev/null +++ b/experiments/manifests/warm-pool.yaml @@ -0,0 +1,8 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: openshell-warm-pool +spec: + templateRef: + name: openshell-warm + replicas: 5 diff --git a/experiments/measure-cold-start.sh b/experiments/measure-cold-start.sh new file mode 100755 index 0000000000..c6128f955e --- /dev/null +++ b/experiments/measure-cold-start.sh @@ -0,0 +1,247 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="cold-start" +CONFIG="prepulled" +RUNS="" +NAMESPACE="${NAMESPACE:-default}" +SANDBOX_IMAGE="${SANDBOX_IMAGE:-ghcr.io/nvidia/openshell/sandbox:latest}" + +usage() { + cat </dev/null || true + kubectl delete sandbox "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + ;; + vanilla) + kubectl delete sandbox "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + ;; + esac + # Wait for pod cleanup + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandbox "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + (( elapsed++ )) + done +} + +get_sandbox_pod() { + local name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/name=$name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +run_openshell_config() { + local config="$1" + local num_runs="$2" + local csv_file="${RESULTS_DIR}/cold-start-${config}.csv" + + log "Starting cold-start measurement: config=${config}, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local sandbox_name="cold-${config}-run-${i}" + local status="ok" + + log "Run ${i}/${num_runs}: creating sandbox ${sandbox_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! openshell sandbox create --name "$sandbox_name" --from base 2>/dev/null; then + log "WARN: Failed to create sandbox ${sandbox_name}" + status="create-failed" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "$sandbox_name" "$status" + cleanup_sandbox "$sandbox_name" "sandbox" + continue + fi + + if wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_sandbox_pod "$sandbox_name") + + local phase_deltas=",,,," + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-${config}-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + phase_deltas=$(extract_phase_deltas "$(cat "$events_file")" "$create_ts") + fi + + IFS=',' read -r sched_ms pull_ms init_ms sup_ms ssh_ms <<< "$phase_deltas" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" "$sched_ms" "$pull_ms" "$init_ms" "$sup_ms" "$ssh_ms" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown})" + else + status="timeout" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "$sandbox_name" "$status" + log "Run ${i}: TIMEOUT" + fi + + cleanup_sandbox "$sandbox_name" "sandbox" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_vanilla_config() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/cold-start-vanilla.csv" + + log "Starting cold-start measurement: config=vanilla, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local sandbox_name="vanilla-run-${i}" + local status="ok" + + log "Run ${i}/${num_runs}: creating vanilla sandbox ${sandbox_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! kubectl apply -n "$NAMESPACE" -f - < "$events_file" + phase_deltas=$(extract_phase_deltas "$(cat "$events_file")" "$create_ts") + fi + + IFS=',' read -r sched_ms pull_ms init_ms sup_ms ssh_ms <<< "$phase_deltas" + write_csv_row "$csv_file" "$i" "vanilla" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" "$sched_ms" "$pull_ms" "$init_ms" "$sup_ms" "$ssh_ms" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown})" + else + status="timeout" + write_csv_row "$csv_file" "$i" "vanilla" "$create_ts" "" "" "$sandbox_name" "$status" + log "Run ${i}: TIMEOUT" + fi + + cleanup_sandbox "$sandbox_name" "vanilla" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_config() { + local config="$1" + local num_runs="${RUNS:-$(default_runs_for_config "$config")}" + + case "$config" in + prepulled|noprepull) + run_openshell_config "$config" "$num_runs" + ;; + vanilla) + run_vanilla_config "$num_runs" + ;; + *) + log "ERROR: Unknown config: $config" + exit 1 + ;; + esac +} + +case "$CONFIG" in + all) + run_config "prepulled" + run_config "noprepull" + run_config "vanilla" + ;; + prepulled|noprepull|vanilla) + run_config "$CONFIG" + ;; + *) + log "ERROR: Invalid config: $CONFIG (must be prepulled, noprepull, vanilla, or all)" + exit 1 + ;; +esac + +log "Cold-start measurement complete." diff --git a/experiments/measure-combined.sh b/experiments/measure-combined.sh new file mode 100755 index 0000000000..f3c390a025 --- /dev/null +++ b/experiments/measure-combined.sh @@ -0,0 +1,279 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="combined" +RUNS=10 +NAMESPACE="${NAMESPACE:-default}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" +READINESS_METHOD="probe-1s" +INJECT_ENV="true" + +usage() { + cat </dev/null || log "WARN: Could not patch template readiness probe" + ;; + probe-1s) + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge -p '{ + "spec": { + "podTemplateSpec": { + "spec": { + "containers": [{ + "name": "sandbox", + "readinessProbe": { + "tcpSocket": {"port": 2222}, + "initialDelaySeconds": 1, + "periodSeconds": 1 + } + }] + } + } + } + }' 2>/dev/null || log "WARN: Could not patch template readiness probe" + ;; + readiness-gate) + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge -p '{ + "spec": { + "podTemplateSpec": { + "spec": { + "readinessGates": [{"conditionType": "sandbox.agents.x-k8s.io/Ready"}] + } + } + } + }' 2>/dev/null || log "WARN: Could not patch template readiness gates" + ;; + sidecar) + log "Sidecar readiness requires custom template with sidecar container." + log "Ensure your SandboxTemplate already includes the readiness sidecar." + ;; + esac +} + +patch_env_policy() { + if [[ "$INJECT_ENV" == "true" ]]; then + log "Enabling env var injection on template" + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge \ + -p '{"spec":{"envVarsInjectionPolicy":"Allowed"}}' 2>/dev/null || \ + log "WARN: Could not patch env injection policy" + fi +} + +get_claim_pod() { + local claim_name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/claim=$claim_name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +wait_claim_ready() { + local name="$1" + local timeout="${2:-120}" + local elapsed=0 + while (( elapsed < timeout )); do + local status + status=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local phase + phase=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{.status.phase}' 2>/dev/null) || true + + if [[ "$phase" == "Failed" ]]; then + return 1 + fi + + sleep 1 + (( elapsed++ )) + done + return 1 +} + +cleanup_claim() { + local name="$1" + kubectl delete sandboxclaim "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandboxclaim "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + (( elapsed++ )) + done +} + +run_combined() { + local num_runs="$1" + local config_label="${READINESS_METHOD}" + if [[ "$INJECT_ENV" == "true" ]]; then + config_label="${config_label}+env" + fi + + local csv_file="${RESULTS_DIR}/combined.csv" + + log "Starting combined measurement: readiness=${READINESS_METHOD}, env=${INJECT_ENV}, runs=${num_runs}" + ensure_results_dir + + patch_template_for_readiness "$READINESS_METHOD" + patch_env_policy + + sleep 5 + log "Waiting for warm pool to stabilize after template patch..." + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="combined-run-${i}" + local status="ok" + local adoption="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} (${config_label})" + + local create_ts + create_ts=$(capture_timestamp) + + local claim_yaml + if [[ "$INJECT_ENV" == "true" ]]; then + claim_yaml=$(cat </dev/null; then + log "WARN: Failed to create claim ${claim_name}" + status="create-failed" + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},,,,${status},${adoption}" >> "$csv_file" + cleanup_claim "$claim_name" + continue + fi + + if wait_claim_ready "$claim_name" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-combined-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + adoption=$(detect_adoption "$pod_name" "$NAMESPACE") + fi + + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},${status},${adoption}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown}, adoption: ${adoption})" + else + status="timeout" + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},,,,${status},${adoption}" >> "$csv_file" + log "Run ${i}: TIMEOUT" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_combined "$RUNS" + +log "Combined measurement complete (${READINESS_METHOD}, env=${INJECT_ENV})." diff --git a/experiments/measure-env-injection.sh b/experiments/measure-env-injection.sh new file mode 100755 index 0000000000..179fdb621a --- /dev/null +++ b/experiments/measure-env-injection.sh @@ -0,0 +1,291 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="env-injection" +CONFIG="allowed" +RUNS=5 +NAMESPACE="${NAMESPACE:-default}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" + +usage() { + cat </dev/null || { + log "WARN: Could not patch template (may not exist yet or field unsupported)" + return 1 + } +} + +get_claim_pod() { + local claim_name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/claim=$claim_name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +wait_claim_ready() { + local name="$1" + local timeout="${2:-120}" + local elapsed=0 + while (( elapsed < timeout )); do + local status + status=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local phase + phase=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{.status.phase}' 2>/dev/null) || true + + if [[ "$phase" == "Failed" ]]; then + log "Claim $name failed" + return 1 + fi + + sleep 1 + (( elapsed++ )) + done + + log "WARN: Claim $name did not become ready within ${timeout}s" + return 1 +} + +cleanup_claim() { + local name="$1" + kubectl delete sandboxclaim "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandboxclaim "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + (( elapsed++ )) + done +} + +run_allowed() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/env-injection-allowed.csv" + + log "Starting env injection measurement: config=allowed, runs=${num_runs}" + ensure_results_dir + + patch_template_policy "Allowed" || true + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="envinj-allowed-run-${i}" + local status="ok" + local adoption="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} with env vars" + + local create_ts + create_ts=$(capture_timestamp) + + if ! kubectl apply -n "$NAMESPACE" -f - <> "$csv_file" + cleanup_claim "$claim_name" + continue + fi + + if wait_claim_ready "$claim_name" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-envinj-allowed-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + adoption=$(detect_adoption "$pod_name" "$NAMESPACE") + fi + + echo "${i},${EXPERIMENT_ID},allowed,${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},${status},${adoption}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown}, adoption: ${adoption})" + else + status="timeout" + echo "${i},${EXPERIMENT_ID},allowed,${create_ts},,,,${status},${adoption}" >> "$csv_file" + log "Run ${i}: TIMEOUT" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_disallowed() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/env-injection-disallowed.csv" + + log "Starting env injection measurement: config=disallowed, runs=${num_runs}" + ensure_results_dir + + patch_template_policy "Disallowed" || true + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,behavior" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="envinj-disallowed-run-${i}" + local behavior="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} with env vars (policy=Disallowed)" + + local create_ts + create_ts=$(capture_timestamp) + + local apply_output + local apply_exit=0 + apply_output=$(kubectl apply -n "$NAMESPACE" -f - 2>&1 <> "$csv_file" + continue + fi + + sleep 3 + + local phase + phase=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.phase}' 2>/dev/null) || phase="" + + local conditions + conditions=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.conditions[*].message}' 2>/dev/null) || conditions="" + + if [[ "$phase" == "Failed" ]]; then + behavior="failed-phase" + elif wait_claim_ready "$claim_name" 60; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + local adoption + adoption=$(detect_adoption "${pod_name:-}" "$NAMESPACE") + + if [[ "$adoption" == "cold-fallback" ]]; then + behavior="cold-fallback-with-env-stripped" + else + behavior="unexpected-warm-adopted" + fi + + echo "${i},${EXPERIMENT_ID},disallowed,${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},ok,${behavior}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (behavior: ${behavior}, conditions: ${conditions})" + cleanup_claim "$claim_name" + continue + else + behavior="timeout" + fi + + log "Run ${i}: behavior=${behavior}, phase=${phase}, conditions=${conditions}" + echo "${i},${EXPERIMENT_ID},disallowed,${create_ts},,,,${behavior},${behavior}" >> "$csv_file" + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + log "Review ${csv_file} for behavior observations under Disallowed policy." +} + +case "$CONFIG" in + allowed) + run_allowed "$RUNS" + ;; + disallowed) + run_disallowed "$RUNS" + ;; +esac + +log "Env injection measurement complete." diff --git a/experiments/measure-readiness-gates.sh b/experiments/measure-readiness-gates.sh new file mode 100755 index 0000000000..f9ffe672fe --- /dev/null +++ b/experiments/measure-readiness-gates.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="readiness-gates" +RUNS=10 +NAMESPACE="${NAMESPACE:-default}" +MANIFEST="${SCRIPT_DIR}/manifests/readiness-gate-pod.yaml" +CSV_FILE="${RESULTS_DIR}/readiness-gates.csv" + +usage() { + cat </dev/null || echo "") + if [[ "$phase" == "Running" ]]; then + return 0 + fi + if [[ "$phase" == "Failed" || "$phase" == "Succeeded" ]]; then + log "Pod $pod terminated with phase: $phase" + return 1 + fi + sleep 1 + (( elapsed++ )) + done + log "Timeout: pod $pod not Running after ${timeout}s" + return 1 +} + +log "Starting readiness-gate measurement: runs=${RUNS}" +ensure_results_dir +write_csv_header "$CSV_FILE" + +for (( i=1; i<=RUNS; i++ )); do + POD="readiness-gate-run-${i}" + status="ok" + + log "Run ${i}/${RUNS}: creating pod ${POD}" + + sed "s/readiness-gate-test-PLACEHOLDER/${POD}/" "$MANIFEST" | \ + kubectl apply -n "$NAMESPACE" -f - + + if ! wait_pod_running "$POD" 120; then + status="not-running" + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "" "" "" "$POD" "$status" + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + continue + fi + + patch_ts=$(capture_timestamp) + + kubectl patch pod "$POD" -n "$NAMESPACE" \ + --type=json --subresource=status \ + -p '[{"op":"add","path":"/status/conditions/-","value":{"type":"sandbox.openshell.io/claimed","status":"True","lastTransitionTime":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"}}]' + + if wait_for_pod_ready "$POD" "$NAMESPACE" 60; then + ready_ts=$(capture_timestamp) + delta_ms=$(( (ready_ts - patch_ts) / 1000000 )) + + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "$patch_ts" "$ready_ts" "$delta_ms" "$POD" "$status" + log "Run ${i}: ${delta_ms}ms" + else + status="timeout" + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "$patch_ts" "" "" "$POD" "$status" + log "Run ${i}: TIMEOUT" + fi + + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + sleep 2 +done + +log "Results written to ${CSV_FILE}" +compute_stats "$CSV_FILE" +log "Readiness-gate measurement complete." diff --git a/experiments/measure-sidecar-readiness.sh b/experiments/measure-sidecar-readiness.sh new file mode 100755 index 0000000000..cadae7defc --- /dev/null +++ b/experiments/measure-sidecar-readiness.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="sidecar-readiness" +RUNS=10 +NAMESPACE="${NAMESPACE:-default}" +MANIFEST="${SCRIPT_DIR}/manifests/sidecar-readiness.yaml" +CSV_FILE="${RESULTS_DIR}/sidecar-readiness.csv" + +usage() { + cat </dev/null || echo "") + if [[ -n "$running" ]]; then + return 0 + fi + sleep 1 + (( elapsed++ )) + done + log "Timeout: container $container in pod $pod not Running after ${timeout}s" + return 1 +} + +log "Starting sidecar-readiness measurement: runs=${RUNS}" +ensure_results_dir +write_csv_header "$CSV_FILE" + +for (( i=1; i<=RUNS; i++ )); do + POD="sidecar-readiness-run-${i}" + status="ok" + + log "Run ${i}/${RUNS}: creating pod ${POD}" + + sed "s/sidecar-readiness-test-PLACEHOLDER/${POD}/" "$MANIFEST" | \ + kubectl apply -n "$NAMESPACE" -f - + + if ! wait_container_running "$POD" "sandbox" 120; then + status="not-running" + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "" "" "" "$POD" "$status" + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + continue + fi + + signal_ts=$(capture_timestamp) + + kubectl exec "$POD" -n "$NAMESPACE" -c sandbox -- touch /tmp/signal/ready + + if wait_for_pod_ready "$POD" "$NAMESPACE" 60; then + ready_ts=$(capture_timestamp) + delta_ms=$(( (ready_ts - signal_ts) / 1000000 )) + + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "$signal_ts" "$ready_ts" "$delta_ms" "$POD" "$status" + log "Run ${i}: ${delta_ms}ms" + else + status="timeout" + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "$signal_ts" "" "" "$POD" "$status" + log "Run ${i}: TIMEOUT" + fi + + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + sleep 2 +done + +log "Results written to ${CSV_FILE}" +compute_stats "$CSV_FILE" +log "Sidecar-readiness measurement complete." diff --git a/experiments/measure-warm-pool.sh b/experiments/measure-warm-pool.sh new file mode 100755 index 0000000000..434c553bb6 --- /dev/null +++ b/experiments/measure-warm-pool.sh @@ -0,0 +1,416 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="warm-pool" +CONFIG="default" +RUNS="" +NAMESPACE="${NAMESPACE:-default}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +CLAIM_TIMEOUT="${CLAIM_TIMEOUT:-60}" + +usage() { + cat </dev/null; then + log "ERROR: SandboxWarmPool '$WARM_POOL_NAME' not found in namespace '$NAMESPACE'" + log "Create it first: kubectl apply -f ${SCRIPT_DIR}/manifests/warm-pool.yaml" + exit 1 + fi + + local ready_replicas + ready_replicas=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + local desired_replicas + desired_replicas=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0") + + if [[ "$ready_replicas" == "0" || -z "$ready_replicas" ]]; then + log "ERROR: SandboxWarmPool has no ready replicas (desired: ${desired_replicas})" + log "Wait for pool provisioning before running measurements" + exit 1 + fi + + log "Pool status: ${ready_replicas}/${desired_replicas} replicas ready" +} + +wait_pool_replenished() { + local min_ready="${1:-1}" + local timeout="${2:-120}" + local elapsed=0 + + while (( elapsed < timeout )); do + local ready + ready=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + if [[ -n "$ready" ]] && (( ready >= min_ready )); then + return 0 + fi + if (( elapsed % 10 == 0 && elapsed > 0 )); then + log "Waiting for pool replenishment... (${ready:-0} ready, need ${min_ready}, ${elapsed}s/${timeout}s)" + fi + sleep 1 + (( elapsed++ )) + done + + log "WARN: Pool did not replenish to ${min_ready} replicas within ${timeout}s" + return 1 +} + +get_claimed_sandbox_name() { + local claim_name="$1" + kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.sandboxRef.name}' 2>/dev/null || echo "" +} + +get_sandbox_pod_name() { + local sandbox_name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/name=$sandbox_name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +wait_claim_bound() { + local claim_name="$1" + local timeout="${2:-$CLAIM_TIMEOUT}" + local deadline=$(( $(date +%s) + timeout )) + + while (( $(date +%s) < deadline )); do + local phase + phase=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.phase}' 2>/dev/null || echo "") + + if [[ "$phase" == "Bound" ]]; then + return 0 + fi + if [[ "$phase" == "Failed" ]]; then + log "WARN: Claim $claim_name failed" + return 1 + fi + + sleep 1 + done + + local final_phase + final_phase=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.phase}' 2>/dev/null || echo "unknown") + log "WARN: Claim $claim_name not bound after ${timeout}s (phase: ${final_phase})" + return 1 +} + +wait_claimed_sandbox_ready() { + local claim_name="$1" + local timeout="${2:-$CLAIM_TIMEOUT}" + + local sandbox_name + sandbox_name=$(get_claimed_sandbox_name "$claim_name") + if [[ -z "$sandbox_name" ]]; then + log "WARN: No sandbox assigned to claim $claim_name" + return 1 + fi + + wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" "$timeout" +} + +cleanup_claim() { + local claim_name="$1" + kubectl delete sandboxclaim "$claim_name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true +} + +create_claim_yaml() { + local claim_name="$1" + cat </dev/null || echo "10" +} + +run_single_config() { + local config="$1" + local num_runs="$2" + local csv_file="${RESULTS_DIR}/warm-pool-${config}.csv" + + log "Starting warm-pool measurement: config=${config}, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="wp-${config}-run-${i}" + local status="ok" + + # Wait for at least 1 replica before claiming + if ! wait_pool_replenished 1 60; then + log "WARN: Pool exhausted, recording run ${i} as pool-exhausted" + local ts + ts=$(capture_timestamp) + write_csv_row "$csv_file" "$i" "$config" "$ts" "" "" "" "pool-exhausted" + continue + fi + + log "Run ${i}/${num_runs}: creating claim ${claim_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! create_claim_yaml "$claim_name" | kubectl apply -f - 2>/dev/null; then + log "WARN: Failed to create claim ${claim_name}" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "" "create-failed" + cleanup_claim "$claim_name" + continue + fi + + # Wait for claim to bind and sandbox to be ready + if wait_claim_bound "$claim_name" "$CLAIM_TIMEOUT"; then + local sandbox_name + sandbox_name=$(get_claimed_sandbox_name "$claim_name") + + if [[ -n "$sandbox_name" ]] && wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" "$CLAIM_TIMEOUT"; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_sandbox_pod_name "$sandbox_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-wp-${config}-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + fi + + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" + log "Run ${i}: ${delta_ms}ms (sandbox: ${sandbox_name}, pod: ${pod_name:-unknown})" + else + status="sandbox-not-ready" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "${sandbox_name:-unknown}" "$status" + log "Run ${i}: sandbox not ready (${sandbox_name:-unknown})" + fi + else + status="bind-timeout" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "" "$status" + log "Run ${i}: claim bind timeout" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_burst_config() { + local num_runs="$1" + local burst_size=5 + local csv_file="${RESULTS_DIR}/warm-pool-burst.csv" + + log "Starting warm-pool burst measurement: runs=${num_runs}, burst_size=${burst_size}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( round=1; round<=num_runs; round++ )); do + log "Burst round ${round}/${num_runs}: submitting ${burst_size} simultaneous claims" + + # Wait for pool to have enough replicas + if ! wait_pool_replenished "$burst_size" 120; then + log "WARN: Pool has fewer than ${burst_size} ready replicas for round ${round}" + fi + + local pool_ready_before + pool_ready_before=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + log "Pool state before burst: ${pool_ready_before} ready" + + # Create all claims simultaneously + local create_ts + create_ts=$(capture_timestamp) + + declare -a claim_names=() + for (( j=1; j<=burst_size; j++ )); do + local claim_name="wp-burst-r${round}-c${j}" + claim_names+=("$claim_name") + create_claim_yaml "$claim_name" | kubectl apply -f - 2>/dev/null & + done + wait # Wait for all kubectl apply commands + + # Wait for each claim and record individual times + for (( j=0; j/dev/null || echo "unknown") + if [[ "$claim_phase" == "Pending" ]]; then + status="pool-exhausted" + else + status="bind-timeout" + fi + write_csv_row "$csv_file" "$run_label" "burst" "$create_ts" "" "" "" "$status" + log " Claim ${claim_name}: ${status} (phase: ${claim_phase})" + fi + done + + # Clean up all claims from this round + for claim_name in "${claim_names[@]}"; do + cleanup_claim "$claim_name" + done + + # Wait for pool to replenish before next round + if (( round < num_runs )); then + log "Waiting for pool replenishment before next burst round..." + wait_pool_replenished "$burst_size" 180 || true + fi + + unset claim_names + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_config() { + local config="$1" + local num_runs="${RUNS:-$(default_runs_for_config "$config")}" + + case "$config" in + default) + run_single_config "default" "$num_runs" + ;; + aggressive) + local original_period + original_period=$(save_original_probe_period) + log "Saving original readinessProbe periodSeconds: ${original_period}" + + patch_readiness_probe 1 + trap 'patch_readiness_probe '"$original_period"'; _cleanup_on_exit' EXIT + trap 'patch_readiness_probe '"$original_period"'; _cleanup_on_exit; exit 130' INT TERM + + log "Waiting for pool to stabilize with aggressive probes..." + sleep 10 + wait_pool_replenished 1 120 || true + + run_single_config "aggressive" "$num_runs" + + log "Restoring original readinessProbe periodSeconds: ${original_period}" + patch_readiness_probe "$original_period" + trap _cleanup_on_exit EXIT + trap - INT TERM + ;; + burst) + run_burst_config "$num_runs" + ;; + *) + log "ERROR: Unknown config: $config" + exit 1 + ;; + esac +} + +# Main +check_prerequisites + +case "$CONFIG" in + all) + run_config "default" + run_config "aggressive" + run_config "burst" + ;; + default|aggressive|burst) + run_config "$CONFIG" + ;; + *) + log "ERROR: Invalid config: $CONFIG (must be default, aggressive, burst, or all)" + exit 1 + ;; +esac + +log "Warm-pool measurement complete." diff --git a/experiments/results/.gitignore b/experiments/results/.gitignore new file mode 100644 index 0000000000..afed0735dc --- /dev/null +++ b/experiments/results/.gitignore @@ -0,0 +1 @@ +*.csv diff --git a/experiments/sidecar/Dockerfile b/experiments/sidecar/Dockerfile new file mode 100644 index 0000000000..3ae8fabc51 --- /dev/null +++ b/experiments/sidecar/Dockerfile @@ -0,0 +1,9 @@ +FROM golang:1.24-alpine AS builder +WORKDIR /src +COPY go.mod main.go ./ +RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /readiness-sidecar . + +FROM scratch +COPY --from=builder /readiness-sidecar /readiness-sidecar +EXPOSE 8080 +ENTRYPOINT ["/readiness-sidecar"] diff --git a/experiments/sidecar/Makefile b/experiments/sidecar/Makefile new file mode 100644 index 0000000000..bc2d6896c6 --- /dev/null +++ b/experiments/sidecar/Makefile @@ -0,0 +1,14 @@ +IMAGE ?= ghcr.io/nvidia/openshell/sidecar-readiness:latest + +.PHONY: all build image push + +all: build image push + +build: + CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o readiness-sidecar main.go + +image: + podman build -t $(IMAGE) . + +push: + podman push $(IMAGE) diff --git a/experiments/sidecar/go.mod b/experiments/sidecar/go.mod new file mode 100644 index 0000000000..d386c45aef --- /dev/null +++ b/experiments/sidecar/go.mod @@ -0,0 +1,3 @@ +module github.com/nvidia/openshell/experiments/sidecar + +go 1.24 diff --git a/experiments/sidecar/main.go b/experiments/sidecar/main.go new file mode 100644 index 0000000000..7f833bd75d --- /dev/null +++ b/experiments/sidecar/main.go @@ -0,0 +1,46 @@ +package main + +import ( + "fmt" + "log" + "net/http" + "os" + "sync/atomic" + "time" +) + +const signalPath = "/tmp/signal/ready" + +var ready atomic.Bool + +func main() { + go pollSignalFile() + + http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { + if ready.Load() { + w.WriteHeader(http.StatusOK) + fmt.Fprintln(w, "ready") + } else { + w.WriteHeader(http.StatusServiceUnavailable) + fmt.Fprintln(w, "not ready") + } + }) + + log.Println("readiness sidecar listening on :8080") + if err := http.ListenAndServe(":8080", nil); err != nil { + log.Fatalf("server failed: %v", err) + } +} + +func pollSignalFile() { + for { + if _, err := os.Stat(signalPath); err == nil { + if !ready.Load() { + ready.Store(true) + log.Println("signal file detected, transitioning to ready") + } + return + } + time.Sleep(100 * time.Millisecond) + } +} diff --git a/rfc/NNNN-warm-pool-feasibility/README.md b/rfc/NNNN-warm-pool-feasibility/README.md new file mode 100644 index 0000000000..564b03c538 --- /dev/null +++ b/rfc/NNNN-warm-pool-feasibility/README.md @@ -0,0 +1,382 @@ +--- +authors: + - "@rhuss" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/2157 +--- + +# RFC NNNN - Warm Pool Feasibility Study + +## Summary + +This RFC documents the findings of a feasibility study for warm-pooling +sandbox pods in OpenShell's Kubernetes driver. The study evaluates whether +pre-provisioned, idle sandbox pods can reduce sandbox startup latency from +the current 8-12s cold-start baseline to under 2s, and identifies the +architectural changes required to adopt warm pooling in production. + +The study covers claim latency measurements, health check tuning, environment +variable injection behavior, identity binding constraints, and integration +points with the Agent Sandbox operator's `SandboxWarmPool` CRD. + +## Motivation + +OpenShell's Kubernetes driver provisions a fresh sandbox pod for every +`sandbox.create` request. On a typical cluster, this cold-start path takes +8-12s (image pull excluded), which is too slow for interactive agent workflows +where sub-second tool calls are the norm. Agents that create sandboxes +mid-conversation force the user to wait, breaking flow and limiting the +viability of ephemeral sandbox patterns. + +Warm pooling addresses this by maintaining a pool of pre-provisioned, +ready-to-claim sandbox pods. When a sandbox is requested, the driver claims +an existing pod from the pool instead of creating one from scratch. The pool +controller replenishes claimed pods in the background, keeping idle capacity +available for the next request. + +The Agent Sandbox operator already ships a `SandboxWarmPool` CRD (Tech Preview) +that implements pool lifecycle management, including replenishment, health +checks, and idle eviction. This study evaluates whether that CRD meets +OpenShell's requirements or whether gaps exist that would block adoption. + +### Why now + +- Issue [#2157](https://github.com/NVIDIA/OpenShell/issues/2157) tracks the + warm pool integration as a concrete feature request. +- The Agent Sandbox operator's `SandboxWarmPool` CRD reached Tech Preview in + operator version TBD, making it available for evaluation. +- Multiple users have reported cold-start latency as a blocker for + interactive agent workflows. + +## Non-goals + +- **Pool autoscaling.** This study evaluates fixed-size pools. Autoscaling + based on demand signals is a separate concern. +- **Multi-cluster pooling.** Pools are cluster-local. Cross-cluster pool + federation is out of scope. +- **Non-Kubernetes drivers.** Warm pooling applies only to the Kubernetes + compute driver. Docker and VM drivers have different startup + characteristics. +- **Image pre-pull optimization.** Image pull latency is excluded from + measurements. Pre-pull is a separate optimization (DaemonSet or operator + feature). +- **GPU resource pooling.** GPU-attached sandbox pooling has different + economics and constraints. This study covers CPU-only pools. + +## Experiment Setup + +All measurements were collected on a single cluster with the following +configuration: + +| Parameter | Value | +|-----------|-------| +| Cluster type | TBD | +| Kubernetes version | TBD | +| Agent Sandbox operator version | TBD | +| OpenShell version | TBD (commit hash) | +| Node count | TBD | +| Node instance type | TBD | +| Sandbox image | TBD | +| Image pre-pulled | Yes / No | +| Pool size | TBD | +| Measurement tool | TBD | +| Sample count per measurement | TBD | + +### Pool Configuration + +```yaml +# SandboxWarmPool CRD configuration used for measurements +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: warm-pool-feasibility +spec: + templateRef: + name: openshell-warm + replicas: TBD +``` + +## Results + +### Cold-Start Baseline + +Baseline measurements without warm pooling, from `sandbox.create` request +to sandbox ready. + +| Metric | Value | +|--------|-------| +| p50 latency | TBD | +| p90 latency | TBD | +| p99 latency | TBD | +| Min | TBD | +| Max | TBD | +| Sample count | TBD | + +### Warm Pool Claim Latency + +Time from `sandbox.create` request to sandbox ready when claiming from a +warm pool. + +| Metric | Value | +|--------|-------| +| p50 latency | TBD | +| p90 latency | TBD | +| p99 latency | TBD | +| Min | TBD | +| Max | TBD | +| Sample count | TBD | + +### Comparison + +| Scenario | p50 | p90 | Speedup | +|----------|-----|-----|---------| +| Cold start (no pool) | TBD | TBD | baseline | +| Warm pool claim | TBD | TBD | TBD | +| Target | < 2s | < 2s | > 4x | + +### Pool Drain Behavior + +Measurements under sustained load when the pool is fully drained and +requests fall back to cold-start provisioning. + +| Scenario | p50 | p90 | +|----------|-----|-----| +| Pool available (steady state) | TBD | TBD | +| Pool drained (fallback to cold start) | TBD | TBD | +| Pool replenishment time (per pod) | TBD | TBD | + +## Health Check Analysis + +### Probe Configuration Impact + +How different readiness probe configurations affect claim latency and +pool stability. + +| Probe interval | Initial delay | Failure threshold | Claim latency p50 | False-positive eviction rate | +|----------------|---------------|-------------------|--------------------|------------------------------| +| TBD | TBD | TBD | TBD | TBD | +| TBD | TBD | TBD | TBD | TBD | +| TBD | TBD | TBD | TBD | TBD | + +### Readiness Gates + +Whether the `SandboxWarmPool` CRD supports custom readiness gates and how +they interact with the sandbox supervisor's startup sequence. + +| Question | Finding | +|----------|---------| +| Does SandboxWarmPool support readiness gates? | TBD | +| Can the supervisor signal readiness via a gate? | TBD | +| Does gate-based readiness reduce claim latency? | TBD | + +### Sidecar Pattern + +Whether the pool controller supports sidecar containers and how they +affect pool pod lifecycle. + +| Question | Finding | +|----------|---------| +| Are sidecar containers preserved on claim? | TBD | +| Can the privacy router run as a sidecar? | TBD | +| Does the sidecar pattern affect replenishment time? | TBD | + +## Environment Variable Injection + +### Injection Behavior + +How environment variables are injected into claimed pool pods, and +whether late-binding (post-claim injection) is supported. + +| Question | Finding | +|----------|---------| +| Are env vars set at pool creation or claim time? | TBD | +| Can env vars be injected/overridden at claim time? | TBD | +| Are secrets mounted or injected as env vars? | TBD | +| Is there a mutation webhook for late binding? | TBD | + +### Policy Requirements + +Environment variables required by the sandbox policy engine and whether +they can be late-bound. + +| Variable | Purpose | Can be late-bound? | +|----------|---------|--------------------| +| `OPENSHELL_GATEWAY_URL` | Gateway callback URL | TBD | +| `OPENSHELL_SANDBOX_ID` | Sandbox identity | TBD | +| `OPENSHELL_AUTH_TOKEN` | Sandbox authentication | TBD | +| `OPENSHELL_POLICY_*` | Policy configuration | TBD | + +### Identity Binding Constraints + +How sandbox identity (sandbox ID, auth tokens, mTLS certificates) is +bound to a claimed pod. Identity must be unique per sandbox session and +cannot be shared across pool members. + +| Constraint | Approach | Feasible? | +|------------|----------|-----------| +| Unique sandbox ID per session | TBD | TBD | +| Per-session auth token | TBD | TBD | +| mTLS certificate rotation on claim | TBD | TBD | +| Gateway store registration timing | TBD | TBD | + +## Architecture Recommendations + +### Kubernetes Driver Changes + +Changes required in `crates/openshell-driver-kubernetes/` to support +warm pool claiming instead of pod creation. + +- **Pool-aware provisioning path.** TBD: How the driver detects pool + availability and switches from create to claim. +- **Claim API integration.** TBD: Which operator API the driver calls + to claim a pod from the pool. +- **Fallback behavior.** TBD: What happens when the pool is empty. +- **Pool selection.** TBD: How the driver selects the correct pool when + multiple pools exist (e.g., different resource profiles). + +### Supervisor Changes + +Changes required in `crates/openshell-sandbox/` to support late-binding +of identity and configuration on a pre-provisioned pod. + +- **Deferred initialization.** TBD: Whether the supervisor can defer + policy loading until claim-time env vars are available. +- **Identity rebinding.** TBD: How the supervisor acquires a new identity + after being claimed from the pool. +- **Health check endpoint.** TBD: Whether a lightweight health endpoint + is needed for pool readiness probes. + +### Gateway Store Changes + +Changes required in the gateway's sandbox store to support the claim +lifecycle. + +- **Registration timing.** TBD: When the gateway registers the sandbox + in its store (at claim time, not pool creation time). +- **Pool metadata.** TBD: Whether the gateway needs to track pool + membership for cleanup. + +### Identity Binding Mechanism + +The recommended approach for binding sandbox identity to a claimed pod. + +- TBD: Recommended mechanism (env var injection, config file mount, + API callback, or combination). +- TBD: Security properties (token rotation, certificate lifecycle). +- TBD: Operator support level for the chosen mechanism. + +### Recommendation for Issue #2157 + +Based on the findings above, the recommended path forward for +[#2157](https://github.com/NVIDIA/OpenShell/issues/2157). + +- TBD: Whether warm pooling is feasible with the current operator. +- TBD: Whether the sub-2s target is achievable. +- TBD: Recommended implementation phases. +- TBD: Estimated effort and timeline. + +## Gaps and Risks + +### Missing Agent Sandbox Operator Features + +Features that the `SandboxWarmPool` CRD does not currently support but that +OpenShell requires for production use. + +| Gap | Severity | Workaround | +|-----|----------|------------| +| TBD | TBD | TBD | +| TBD | TBD | TBD | + +### Red Hat Tech Preview Coverage + +Areas where the Tech Preview designation limits production adoption. + +| Limitation | Impact | +|------------|--------| +| TBD | TBD | +| TBD | TBD | + +### Pool Replenishment Under Burst + +Risk of pool exhaustion under burst traffic and the resulting fallback +to cold-start latency. + +| Scenario | Pool drain time | Recovery time | User impact | +|----------|-----------------|---------------|-------------| +| TBD | TBD | TBD | TBD | + +### Other Risks + +- **Idle resource cost.** Warm pool pods consume cluster resources while + idle. The cost-benefit tradeoff depends on pool size and claim + frequency. +- **Stale pods.** Long-lived idle pods may accumulate stale state + (expired certificates, outdated images). Eviction and rotation + policies are needed. +- **Operator version coupling.** Tight coupling to a specific operator + version may limit OpenShell's deployment flexibility. + +## Alternatives + +### Do nothing + +Keep the current cold-start provisioning path. Users accept 8-12s +startup latency. This is unacceptable for interactive agent workflows +but acceptable for batch or long-running sandbox use cases. + +### Pre-warm at the container runtime level + +Use container runtime features (e.g., checkpoint/restore with CRIU) to +snapshot a ready sandbox container and restore it on demand. This avoids +the pool management overhead but requires runtime-specific support and +may not preserve network state correctly. + +### Client-side sandbox reuse + +Instead of creating a new sandbox per tool call, reuse an existing +sandbox across multiple tool invocations within the same agent session. +This reduces the number of cold starts but does not eliminate them and +changes the sandbox isolation model. + +## Prior Art + +- **Knative cold-start mitigation.** Knative maintains a configurable + minimum replica count to avoid cold starts for serverless workloads. + The warm pool pattern is analogous. +- **AWS Lambda SnapStart.** Lambda's SnapStart pre-initializes function + instances and restores from snapshots. Similar in intent but uses + checkpoint/restore rather than pool management. +- **Agent Sandbox operator SandboxWarmPool CRD.** The operator's own pool + implementation, which this study directly evaluates. + +## Open Questions + +- What is the minimum pool size required to sustain a typical interactive + agent workload without pool drain? +- Can the operator's claim API be called directly from the Kubernetes + driver, or does it require going through an admission webhook? +- How does the pool controller handle node failures that take pool pods + with them? +- Is there a mechanism to pre-configure pool pods with a base policy + that gets specialized at claim time? +- What is the operator's roadmap for moving `SandboxWarmPool` from Tech + Preview to GA? + +## Next Steps + +### Upstream Contributions + +- TBD: Features or fixes to contribute to the Agent Sandbox operator. + +### Internal Work Items + +- TBD: OpenShell issues to file for implementing warm pool support. + +### Concrete Action Items + +1. TBD: Complete measurements on the test cluster. +2. TBD: Validate identity binding approach with the operator team. +3. TBD: Prototype the claim path in the Kubernetes driver. +4. TBD: Update this RFC with findings and recommendations. +5. TBD: Present findings to the team for review. diff --git a/specs/001-warm-pool-feasibility/REVIEWERS.md b/specs/001-warm-pool-feasibility/REVIEWERS.md new file mode 100644 index 0000000000..ad7795cb41 --- /dev/null +++ b/specs/001-warm-pool-feasibility/REVIEWERS.md @@ -0,0 +1,128 @@ +# Review Guide: Warm Pool Feasibility Study + +**Generated**: 2026-07-09 | **Spec**: [spec.md](spec.md) + +## Why This Change + +OpenShell's Kubernetes driver creates a fresh Sandbox CR for every sandbox request, paying 8-12 seconds for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration. For agent harnesses like OpenClaw that create sandboxes per tool call, this latency is unusable. The upstream Agent Sandbox project (v0.5.0) provides extension CRDs for warm pooling (SandboxTemplate, SandboxWarmPool, SandboxClaim), but OpenShell has no awareness of these. We need to know if warm pooling can deliver sub-2s sandbox startup on OpenShift before investing in integration work. + +## What Changes + +This study will create an `experiments/` directory with shell-based measurement scripts, Kubernetes manifests for warm pool configurations, and a minimal Go sidecar binary for readiness experiments. It will provision a short-lived ROSA HCP cluster, run 6 experiments measuring cold-start vs warm pool latency across readiness configurations, and produce an RFC in `rfc/` with data-backed architectural recommendations for OpenShell warm pool integration. No changes to OpenShell core code. This PR contains the specification artifacts only; implementation follows after spec approval. + +## How It Works + +The study uses a layered measurement approach across 4 execution phases: + +1. **Cluster Setup**: ROSA HCP with 3x m5.2xlarge workers, Red Hat Agent Sandbox tech preview operator (upstream fallback), OpenShell via the OpenShell OpenShift deploy chart, and image pre-pulling via DaemonSet. + +2. **Measurement Library**: Shared shell functions (`experiments/lib/common.sh`) for nanosecond timestamp capture, CSV output, pod event collection, and p50/p90 computation. Each experiment script wraps kubectl with these functions. + +3. **Experiments**: Six experiments progressively build understanding: + - Cold-start baseline (control measurement) + - Warm pool claim-to-ready with default and aggressive readiness probes + - Pod Readiness Gates (KEP-580) as probe-free alternative + - Sidecar readiness pattern (Knative-style, using KEP-753 native sidecars) + - Env var injection at claim time (identity binding feasibility) + - Combined best-configuration measurement + +4. **RFC**: Results compiled into a standalone RFC following the OpenShell RFC template, structured for later distillation into a GitHub Issue #2157 comment. + +## When It Applies + +**Applies when**: +- Evaluating whether OpenShell should adopt Agent Sandbox warm pooling +- Understanding sandbox startup latency bottlenecks on OpenShift +- Making architectural decisions about OpenShell's Kubernetes driver +- Informing Issue #2157 design discussions with measured data + +**Does not apply when**: +- Podman/Docker single-player warm pooling (explicitly out of scope) +- Production OpenShell code changes (this is measurement-only, no core changes) +- Non-OpenShift Kubernetes distributions (results are OpenShift-specific) + +## Key Decisions + +1. **Layered measurement (no OpenShell code changes)**: Start with raw Agent Sandbox measurements, then layer OpenShell-specific concerns. Rejected "modify the Kubernetes driver first" because if the raw K8s layer is too slow, code changes would be wasted. + +2. **Red Hat tech preview operator first, upstream fallback**: The tech preview may lack extension CRDs, but testing the downstream operator path is part of the study's value. Upstream manifests are applied on top only if needed. + +3. **Shell scripts over Go benchmark harness**: N=10-20 per configuration is enough for a feasibility study. Shell scripts with `date +%s%N` and `kubectl wait` provide sufficient precision without over-engineering. + +4. **Results as RFC (not Google Doc)**: The RFC template provides structured review and lives in the public repo. Google Docs are inaccessible to external contributors. + +5. **3x m5.2xlarge workers**: Enough capacity for warm pool replicas and burst tests without hitting resource limits that would skew measurements. Short-lived cluster to control cost. + +## Areas Needing Attention + +- **Readiness probe interval assumption**: The brainstorm identifies the readiness probe interval as the dominant latency bottleneck. If this assumption is wrong (e.g., kubelet overhead dominates), the Readiness Gate and sidecar experiments may not show the expected improvement. +- **Env var injection behavior is undocumented**: The SandboxClaim `envVarsInjectionPolicy` behavior is based on reading the CRD spec, not tested behavior. The actual controller implementation may differ. +- **Sidecar readiness binary**: Building and pushing a custom container image adds complexity. If the sidecar experiment is not needed (readiness gates are sufficient), this work could be skipped. +- **RFC number assignment**: The OpenShell RFC process requires maintainer-assigned numbers from an originating issue. A placeholder number is used until assignment. + +## Open Questions + +- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs, or only the core Sandbox CRD? (Answered by T002/T003 during setup.) +- What happens when the warm pool is exhausted? Does SandboxClaim stay Pending or fall back to cold start? (Answered by T024 during experiments.) +- Is the sidecar experiment necessary if Pod Readiness Gates already eliminate the probe interval bottleneck? (Decided after T033 results.) + +## Review Checklist + +- [ ] Key decisions are justified +- [ ] Measurement methodology is sound (N=10+, per-phase timestamps, CSV output) +- [ ] Scope boundaries are clear (no OpenShell core changes, K8s only) +- [ ] Success criteria are measurable (sub-2s target, p50/p90 computation) +- [ ] Cluster sizing is appropriate for experiment scope +- [ ] RFC structure follows the OpenShell RFC template +- [ ] No Google Drive links referenced in public-facing artifacts + +## Revision History + +### Rev 1 (2026-07-09) - Address Devin and CodeRabbit review feedback + +**Trigger**: PR review feedback from [#2](https://github.com/rhuss/OpenShell/pull/2) (Devin, CodeRabbit) + +**Spec changes**: None (all findings were in non-spec files) + +**Non-spec fixes**: +- AGENTS.md: Restored original OpenShell agent instructions (was replaced entirely by spex plugin description) +- CLAUDE.md: Removed feature-specific SPECKIT block (should not be in committed version) +- quickstart.md: Removed hardcoded AWS account ID +- brainstorm/04: Replaced hardcoded Obsidian vault path with generic placeholder +- brainstorm/02: Added note about pinning deployment chart to a specific commit + +**Quality gates**: +- review-spec: skipped (no spec changes) +- review-plan: skipped (no plan changes) + +**Cascade impact**: +- plan.md: unchanged +- tasks.md: unchanged +- REVIEWERS.md: revision history appended + +### Rev 2 (2026-07-09) - Address Copilot and Devin re-review feedback + +**Trigger**: PR review feedback from [#2](https://github.com/rhuss/OpenShell/pull/2) (Copilot, Devin re-review) + +**Spec changes**: None + +**Non-spec fixes**: +- .gitignore: Narrowed `**/.claude/` to worktree-specific pattern to avoid blocking root `.claude/` tracked files +- quickstart.md: Fixed DaemonSet path to `experiments/manifests/`, clarified `rosa:create` as Claude Code skill wrapper with CLI equivalent +- tasks.md: Clarified T001 `rosa:create` wrapper reference +- checklists/requirements.md: Fixed self-assessment to accurately reflect technical spec content +- brainstorm/04: Aligned decision with RFC approach, removed individual names and meeting references +- REVIEWERS.md: Reworded "What Changes" to describe planned work (not present artifacts) + +**Quality gates**: +- review-spec: skipped (no spec changes) +- review-plan: skipped (no plan changes) + +**Cascade impact**: +- plan.md: unchanged +- tasks.md: T001 wording updated (no structural change) +- REVIEWERS.md: revision history appended, "What Changes" reworded + +--- + + diff --git a/specs/001-warm-pool-feasibility/checklists/requirements.md b/specs/001-warm-pool-feasibility/checklists/requirements.md new file mode 100644 index 0000000000..7db637f3dc --- /dev/null +++ b/specs/001-warm-pool-feasibility/checklists/requirements.md @@ -0,0 +1,35 @@ +# Specification Quality Checklist: Warm Pool Feasibility Study + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-09 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [ ] No implementation details (languages, frameworks, APIs) -- spec references CLI commands and K8s version constraints as domain context for the feasibility study +- [x] Focused on user value and business needs +- [ ] Written for non-technical stakeholders -- spec is technical by nature (feasibility study targeting engineers) +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [ ] No implementation details leak into specification -- CLI commands and K8s constraints are domain context, not implementation details + +## Notes + +- Items marked [ ] are intentional deviations: this is a technical feasibility study spec, not a product feature spec. CLI commands and K8s version constraints are domain context necessary for the study. +- The spec references specific Kubernetes CRD names and KEP numbers as domain terminology, not implementation choices. diff --git a/specs/001-warm-pool-feasibility/data-model.md b/specs/001-warm-pool-feasibility/data-model.md new file mode 100644 index 0000000000..03ea626306 --- /dev/null +++ b/specs/001-warm-pool-feasibility/data-model.md @@ -0,0 +1,75 @@ +# Data Model: Warm Pool Feasibility Study + +## Entities + +### MeasurementRun + +A single sandbox creation or claim with captured timestamps. + +| Field | Type | Description | +|-------|------|-------------| +| run_number | int | Sequential run number within an experiment | +| experiment_id | string | Experiment identifier (e.g., `cold-start`, `warm-pool-default`) | +| config_label | string | Configuration description (e.g., `probe-10s`, `readiness-gate`) | +| create_timestamp | int64 | Nanosecond epoch when kubectl apply was issued | +| ready_timestamp | int64 | Nanosecond epoch when pod/sandbox became Ready | +| delta_ms | float | Latency in milliseconds (ready - create) | +| phases | string | JSON-encoded phase breakdown from events | +| pod_name | string | Name of the pod that was created or claimed | +| status | enum | `success`, `timeout`, `error` | + +### ExperimentConfig + +A specific combination of settings being tested. + +| Field | Type | Description | +|-------|------|-------------| +| experiment_id | string | Unique identifier | +| description | string | Human-readable description | +| readiness_method | enum | `probe-default`, `probe-1s`, `readiness-gate`, `sidecar` | +| pool_size | int | Number of SandboxWarmPool replicas (0 for cold-start) | +| env_injection | bool | Whether SandboxClaim includes env vars | +| target_runs | int | Number of measurement runs to execute | + +### PhaseTimestamp + +Per-phase latency breakdown extracted from pod events. + +| Field | Type | Description | +|-------|------|-------------| +| run_number | int | FK to MeasurementRun | +| phase_name | string | `scheduled`, `image-pulled`, `init-complete`, `supervisor-ready`, `ssh-available` | +| timestamp | int64 | Nanosecond epoch of phase completion | +| delta_from_create_ms | float | Milliseconds from create to this phase | + +## Relationships + +``` +ExperimentConfig 1---* MeasurementRun +MeasurementRun 1---* PhaseTimestamp +``` + +## State Transitions + +### MeasurementRun Lifecycle + +``` +Created --> Running --> Success + \--> Timeout (>60s) + \--> Error +``` + +### Warm Pool Pod Lifecycle + +``` +Provisioned (warm, NotReady) --> Claimed (SandboxClaim created) + --> Ready (readiness condition met) + --> Terminated (cleanup) +``` + +## CSV Output Format + +```csv +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status +1,cold-start,openshell-prepulled,1720000000000000000,1720000010500000000,10500.0,sandbox-abc123,success +``` diff --git a/specs/001-warm-pool-feasibility/plan.md b/specs/001-warm-pool-feasibility/plan.md new file mode 100644 index 0000000000..e503740b3c --- /dev/null +++ b/specs/001-warm-pool-feasibility/plan.md @@ -0,0 +1,153 @@ +# Implementation Plan: Warm Pool Feasibility Study + +**Branch**: `6111-warm-pool-feasibility` | **Date**: 2026-07-09 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `specs/001-warm-pool-feasibility/spec.md` + +## Summary + +Evaluate Kubernetes Agent Sandbox warm pooling on a ROSA HCP OpenShift cluster to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds. The study provisions a short-lived cluster, runs 6 experiments with shell-based measurement scripts, and produces an RFC in `rfc/` with latency data and architectural recommendations for OpenShell integration. + +## Technical Context + +**Language/Version**: Bash (measurement scripts), Go (sidecar readiness binary), Markdown (RFC) +**Primary Dependencies**: ROSA HCP (AAET profile), Agent Sandbox operator + extension CRDs, OpenShell Helm chart (the OpenShift deploy wrapper), kubectl, oc +**Storage**: CSV files for measurement data, RFC markdown for results +**Testing**: Manual validation of measurement scripts, N=10-20 runs per configuration +**Target Platform**: ROSA HCP, OpenShift 4.20+ (K8s 1.33+), us-east-2, 3x m5.2xlarge workers +**Project Type**: Feasibility study / spike +**Performance Goals**: Measure sandbox startup latency, warm pool claim-to-ready target <2s +**Constraints**: Short-lived cluster (tear down after experiments), no code changes to OpenShell core +**Scale/Scope**: 6 experiments, 4+ configurations, 10-20 runs each, ~100 total measurement runs + +## Constitution Check + +*No project constitution defined (template only). Gate passes trivially.* + +## Project Structure + +### Documentation (this feature) + +```text +specs/001-warm-pool-feasibility/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0: resolved unknowns +├── data-model.md # Phase 1: measurement data model +├── quickstart.md # Phase 1: setup and execution guide +└── tasks.md # Phase 2 output (generated by /speckit-tasks) +``` + +### Source Code (repository root) + +```text +experiments/ +├── measure-cold-start.sh # Experiment 1: cold-start baseline +├── measure-warm-pool.sh # Experiment 2: warm pool claim latency +├── measure-readiness-gates.sh # Experiment 3: Pod Readiness Gates +├── measure-sidecar-readiness.sh # Experiment 4: sidecar readiness pattern +├── measure-env-injection.sh # Experiment 5: env var injection +├── measure-combined.sh # Experiment 6: combined best configuration +├── lib/ +│ ├── common.sh # Shared measurement functions (timestamps, CSV output) +│ └── wait-ready.sh # Pod readiness wait with timeout +├── manifests/ +│ ├── sandbox-template.yaml # SandboxTemplate for warm pool experiments +│ ├── warm-pool.yaml # SandboxWarmPool configurations +│ ├── sandbox-claim.yaml # SandboxClaim templates +│ ├── readiness-gate-pod.yaml # Pod with custom ReadinessGate +│ ├── sidecar-readiness.yaml # Sidecar container with readiness endpoint +│ └── image-prepull-daemonset.yaml # DaemonSet for pre-pulling images +├── sidecar/ +│ ├── main.go # Readiness sidecar binary (HTTP 503/200) +│ ├── Dockerfile # Multi-stage build for scratch image +│ └── Makefile # Build and push sidecar image +└── results/ + └── *.csv # Raw measurement data (gitignored) + +rfc/NNNN-warm-pool-feasibility/ +└── README.md # Results RFC (number assigned by maintainers) +``` + +**Structure Decision**: Experiments live in `experiments/` at repo root for easy discovery. The sidecar binary is self-contained in `experiments/sidecar/`. Results CSV files are gitignored (raw data is synthesized into the RFC). The RFC follows the standard `rfc/NNNN-name/README.md` pattern. + +## Global Constraints + +These constraints from the spec apply to all tasks: + +- **Kubernetes version**: 1.33+ (required for KEP-753 native sidecar containers GA) +- **OpenShift version**: 4.20+ (maps to K8s 1.33) +- **Cluster sizing**: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each), us-east-2 +- **Cluster lifecycle**: Short-lived, tear down after experiments +- **Operator preference**: Red Hat Agent Sandbox tech preview first, upstream v0.5.0 manifests as fallback +- **Measurement format**: Shell scripts producing CSV output +- **Scope**: Kubernetes-backed sandboxes only (no Podman/Docker) + +## Measurement Library Interfaces + +Functions exported by `experiments/lib/common.sh` (consumed by all experiment scripts): + +```bash +capture_timestamp() # Returns nanosecond epoch (date +%s%N) +write_csv_header(file) # Writes CSV header row to file +write_csv_row(file, run, config, create_ts, ready_ts, delta_ms, pod, status) +collect_pod_events(pod) # Returns JSON phase breakdown from kubectl events +compute_stats(csv_file) # Computes p50/p90 from CSV delta_ms column, prints summary +``` + +Functions exported by `experiments/lib/wait-ready.sh`: + +```bash +wait_for_ready(resource, timeout_s) # Blocks until resource is Ready or timeout +wait_for_pod_ready(pod, ns, timeout_s) # Blocks until pod condition Ready=True +``` + +## Execution Phases + +### Phase 1: Cluster Provisioning & Setup + +1. Provision ROSA HCP cluster (3x m5.2xlarge, OpenShift 4.20+) +2. Install Red Hat Agent Sandbox operator from OperatorHub (fallback: upstream manifests) +3. Verify extension CRDs: SandboxTemplate, SandboxWarmPool, SandboxClaim +4. Deploy OpenShell via the OpenShell OpenShift deploy wrapper +5. Pre-pull sandbox images on all worker nodes (DaemonSet approach) +6. Validate: cold-start sandbox creation works end-to-end + +### Phase 2: Measurement Script Development + +1. Build shared measurement library (`experiments/lib/common.sh`) +2. Build cold-start measurement script (Experiment 1) +3. Build warm pool measurement script (Experiments 2, 3) +4. Build sidecar readiness binary and container +5. Build sidecar/readiness-gate measurement scripts (Experiments 3, 4) +6. Build env var injection measurement script (Experiment 5) +7. Build combined measurement script (Experiment 6) + +### Phase 3: Experiment Execution + +1. Run Experiment 1: Cold-start baseline (N=10 pre-pulled, N=5 without) +2. Run Experiment 2: Warm pool claim latency (default probes, aggressive probes) +3. Run Experiment 3: Pod Readiness Gates +4. Run Experiment 4: Sidecar readiness pattern +5. Run Experiment 5: Env var injection +6. Run Experiment 6: Combined best configuration + +### Phase 4: Results & RFC + +1. Compile CSV data into summary tables (p50, p90 per configuration) +2. Write RFC following `rfc/0000-template/README.md` format +3. Include: executive summary, experiment setup, results tables, architecture recommendations, gaps/risks, next steps +4. Tear down cluster + +## Risk Mitigations + +| Risk | Mitigation | +|------|------------| +| Red Hat TP operator lacks extension CRDs | Fallback: upstream `extensions.yaml` on top of operator | +| OpenShift 4.20 not available on ROSA HCP | Check `rosa list versions`, use latest available with K8s 1.33+ | +| the OpenShell OpenShift deploy chart incompatible with OpenShift 4.20 | Adjust chart values; worst case deploy OpenShell manually | +| Sidecar readiness requires custom image | Build minimal Go binary, push to ghcr.io | +| Env var injection triggers cold start | Document as constraint, propose alternative identity binding | + +## Complexity Tracking + +No constitution violations to justify. diff --git a/specs/001-warm-pool-feasibility/quickstart.md b/specs/001-warm-pool-feasibility/quickstart.md new file mode 100644 index 0000000000..4e7e4f7e88 --- /dev/null +++ b/specs/001-warm-pool-feasibility/quickstart.md @@ -0,0 +1,89 @@ +# Quickstart: Warm Pool Feasibility Study + +## Prerequisites + +- AWS credentials for AAET profile +- `rosa` CLI authenticated +- `oc` / `kubectl` CLI +- `gh` CLI for GitHub operations +- `jq` for JSON processing + +## Step 1: Provision Cluster + +```bash +# Using the rosa Claude Code skill (cc-rosa-rhoai plugin): +# rosa:create warm-pool-study +# Or equivalently via the rosa CLI: +rosa create cluster --cluster-name warm-pool-study --sts --hosted-cp \ + --replicas 3 --compute-machine-type m5.2xlarge --region us-east-2 +``` + +3 worker nodes, m5.2xlarge, us-east-2. Wait for cluster to be Ready (~15 min). + +## Step 2: Install Agent Sandbox Operator + +```bash +# Check if Red Hat TP is on OperatorHub +oc get packagemanifests -n openshift-marketplace | grep sandbox + +# If available, install via Subscription +# If not, apply upstream manifests +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/manifest.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/extensions.yaml + +# Verify CRDs +kubectl api-resources | grep agents +``` + +## Step 3: Deploy OpenShell + +```bash +# Clone the OpenShell OpenShift deploy repo (pin to a specific commit for reproducibility) +git clone https://github.com/2000krysztof/Openshell-Openshift-Deploy +cd Openshell-Openshift-Deploy +./deploy.sh +openshell status +``` + +## Step 4: Pre-pull Images + +```bash +# DaemonSet that pre-pulls sandbox images on all nodes +kubectl apply -f experiments/manifests/image-prepull-daemonset.yaml +kubectl rollout status ds/image-prepull +``` + +## Step 5: Run Experiments + +```bash +# Cold-start baseline (N=10 pre-pulled, N=5 no pre-pull) +./experiments/measure-cold-start.sh + +# Warm pool with default probes +./experiments/measure-warm-pool.sh --config default + +# Warm pool with 1s probes +./experiments/measure-warm-pool.sh --config aggressive + +# Readiness gates +./experiments/measure-readiness-gates.sh + +# Sidecar readiness +./experiments/measure-sidecar-readiness.sh + +# Env var injection +./experiments/measure-env-injection.sh +``` + +## Step 6: Generate RFC + +Compile CSV data into the RFC at `rfc/NNNN-warm-pool-feasibility/README.md`. + +## Step 7: Tear Down + +```bash +# Using the rosa Claude Code skill: +# rosa:delete warm-pool-study +# Or equivalently: +rosa delete cluster --cluster warm-pool-study +``` diff --git a/specs/001-warm-pool-feasibility/research.md b/specs/001-warm-pool-feasibility/research.md new file mode 100644 index 0000000000..3f29fa77f8 --- /dev/null +++ b/specs/001-warm-pool-feasibility/research.md @@ -0,0 +1,60 @@ +# Research: Warm Pool Feasibility Study + +## R1: ROSA HCP OpenShift Version Availability (K8s 1.33+) + +**Decision**: Target OpenShift 4.20+ which ships K8s 1.33+, providing native sidecar containers (KEP-753 GA) and Pod Readiness Gates (KEP-580 GA since K8s 1.14). + +**Rationale**: K8s 1.33 is required for native sidecar containers (init containers with `restartPolicy: Always`), which is needed for Experiment 4 (sidecar readiness pattern). OpenShift 4.20 maps to K8s 1.33. ROSA HCP versions are checked at provisioning time via `rosa list versions`. + +**Alternatives considered**: +- OpenShift 4.19 (K8s 1.32): Missing native sidecar GA. Rejected because it would block Experiment 4. +- Kind/k3d local cluster: Faster but not representative of enterprise OpenShift. Rejected because the study specifically targets OpenShift compatibility. + +## R2: Red Hat Agent Sandbox Operator (Tech Preview) Availability + +**Decision**: Install the Red Hat Agent Sandbox tech preview from OperatorHub first. If extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) are not included, apply upstream `extensions.yaml` from kubernetes-sigs/agent-sandbox v0.5.0. + +**Rationale**: The tech preview may only include the core Sandbox CRD. The extension CRDs are in a separate API group (`extensions.agents.x-k8s.io/v1beta1`). Verification: `kubectl api-resources | grep agents` should list all four CRD types. + +**Alternatives considered**: +- Upstream manifests only: Skips the operator path entirely. Rejected because testing the Red Hat operator is part of the study's value. +- Wait for Red Hat TP to include extensions: Unknown timeline, would block the study. Rejected. + +## R3: Measurement Script Design + +**Decision**: Shell scripts per experiment using `kubectl` with `date +%s%N` for nanosecond timestamps. Output to CSV with columns: `run,config,create_ts,ready_ts,delta_ms,phases`. + +**Rationale**: Shell scripts are sufficient for N=10-20 per config. `kubectl wait --for=condition=Ready` provides clean blocking semantics. Phase timestamps are extracted from `kubectl get events --field-selector involvedObject.name=$POD` post-hoc. + +**Alternatives considered**: +- Go benchmark harness with client-go watch: More precise but overkill for feasibility study. Rejected. +- Manual copy-paste: Error-prone with 10+ runs. Rejected. + +## R4: RFC Format and Number Assignment + +**Decision**: The results RFC will follow the OpenShell RFC template (`rfc/0000-template/README.md`). The RFC number must be assigned by maintainers from the originating GitHub issue. For this study, the RFC is created in `rfc/NNNN-warm-pool-feasibility/README.md` once a number is assigned. + +**Rationale**: Per `rfc/README.md`, RFCs require an originating GitHub issue and maintainer-assigned number. The study results naturally fit the RFC structure (Summary, Motivation, Design, Alternatives). + +**Alternatives considered**: +- Informal markdown report: Lacks the structured review process. Rejected because warm pool integration is a cross-cutting architectural decision. +- Google Doc: Prohibited from public repo references per CLAUDE.md. Rejected. + +## R5: Sidecar Readiness Binary + +**Decision**: Use a minimal Go binary for the sidecar readiness experiment. The binary serves HTTP on :8080, returns 503 until a signal file (`/tmp/signal/ready`) exists, then returns 200. Build as a static binary in a scratch container. + +**Rationale**: Go produces small static binaries. The signal file mechanism (shared emptyDir volume) is simple to test. This mirrors the Knative queue-proxy pattern. + +**Alternatives considered**: +- Python/bash sidecar: Requires larger base image (python/bash), slower startup. Rejected. +- gRPC readiness: More complex than needed for a readiness probe. Rejected. + +## R6: Pool Exhaustion Behavior + +**Decision**: Document the behavior when all warm pool replicas are claimed. Expected: SandboxClaim stays Pending until pool replenishes (based on upstream Agent Sandbox controller logic). + +**Rationale**: This is an open question in the spec. The experiment will observe actual behavior. If fallback to cold start is available, it would be a useful feature for production. + +**Alternatives considered**: +- Active testing with configured fallback: The v0.5.0 SandboxWarmPool spec may not support fallback configuration. Observation-only is the safe approach. diff --git a/specs/001-warm-pool-feasibility/review-findings.md b/specs/001-warm-pool-feasibility/review-findings.md new file mode 100644 index 0000000000..30ea9dd035 --- /dev/null +++ b/specs/001-warm-pool-feasibility/review-findings.md @@ -0,0 +1,226 @@ +# Deep Review Findings + +**Date:** 2026-07-09 +**Branch:** 6111-warm-pool-feasibility +**Rounds:** 2 +**Gate Outcome:** PASS +**Invocation:** manual + +## Summary + +| Severity | Found | Fixed | Remaining | +|----------|-------|-------|-----------| +| Critical | 1 | 1 | 0 | +| Important | 7 | 5 | 2 | +| Minor | 8 | - | 8 | +| Notable | 6 | - | 6 | +| **Total** | **22** | **6** | **16** | + +**Agents completed:** 5/5 (+ 0 external tools) +**Agents failed:** CodeRabbit (timed out after 5+ minutes) + +## Findings + +### FINDING-1 +- **Severity:** Critical +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:63-91 (defined), experiments/measure-cold-start.sh (not called) +- **Category:** test-quality +- **Source:** test-quality-agent (also reported by: architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`extract_phase_deltas()` was defined in lib/common.sh but never called by any measurement script. The CSV header declares per-phase columns but all scripts passed only 8 positional arguments to `write_csv_row`, leaving phase columns empty. FR-004 requires per-phase timestamps. + +**Why this matters:** +The per-phase breakdown is the entire point of User Story 1. Without it, cold-start measurements produce only a single end-to-end delta_ms, making it impossible to identify whether scheduling, image pulling, or supervisor startup is the bottleneck. + +**How it was resolved:** +Added calls to `extract_phase_deltas()` in measure-cold-start.sh for both prepulled/noprepull and vanilla config paths. Results are split via `IFS=',' read` and passed as positional arguments 9-13 to write_csv_row. + +### FINDING-2 +- **Severity:** Important +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:63-91 +- **Category:** correctness +- **Source:** correctness-agent (also reported by: architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`extract_phase_deltas()` jq timestamp conversion used incompatible reference frames. The `ts_to_epoch` function produced an approximate day-count value (`year*365*86400 + month*30*86400`), while `$cs` was a real Unix epoch. The subtraction produced nonsensical values (off by ~62 billion seconds for 2026 dates). It also ignored the time-of-day component entirely. + +**Why this matters:** +After FINDING-1 was fixed (wiring up the function), this would have written garbage values into the per-phase CSV columns, corrupting the feasibility study data. + +**How it was resolved:** +Rewrote `extract_phase_deltas()` to use shell-based ISO-to-epoch conversion via `gdate -d` (macOS) or `date -d` (Linux), producing correct epoch seconds for proper delta computation. + +### FINDING-3 +- **Severity:** Important +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:141-143 +- **Category:** correctness +- **Source:** correctness-agent +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`compute_stats` awk uses `vals[NR-1]` for array indexing, but `NR-1` creates non-contiguous indices when rows are skipped (non-ok status or empty delta). The insertion sort iterates `1..n` contiguously, reading uninitialized slots (treated as 0 by awk) while actual values at higher indices are orphaned. This corrupts p50/p90 percentile values. + +**Why this matters:** +When any measurement run produces a timeout or failed row, the percentile statistics in the summary output are wrong. Unset array slots read as 0, pulling percentiles down. + +**How it was resolved:** +Changed `vals[NR-1] = v; n++` to `n++; vals[n] = v` so the array is contiguously indexed by the sample counter, not the line number. + +### FINDING-4 +- **Severity:** Important +- **Confidence:** 92 +- **File:** experiments/measure-warm-pool.sh:374-384 +- **Category:** production-readiness +- **Source:** production-agent (also reported by: correctness-agent, architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +The aggressive config's trap on line 374 replaced the common.sh `_cleanup_on_exit` EXIT trap. Line 384 (`trap - EXIT INT TERM`) permanently cleared ALL traps. When running `--config all`, the burst config ran with zero cleanup traps, meaning Kubernetes resources leaked on interrupt. + +**Why this matters:** +Resources registered via `register_cleanup` would not be cleaned up if the script crashed or was interrupted after the aggressive config completed. + +**How it was resolved:** +Changed the trap to chain both handlers: `trap 'patch_readiness_probe $period; _cleanup_on_exit' EXIT`. After aggressive run, restore only the original common.sh trap instead of clearing all traps. + +### FINDING-5 +- **Severity:** Important +- **Confidence:** 90 +- **File:** experiments/measure-combined.sh:165-184, experiments/measure-env-injection.sh:102-125 +- **Category:** architecture +- **Source:** architecture-agent (also reported by: test-quality-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +Two inconsistent `detect_adoption()` implementations: combined.sh checked only Scheduled events, env-injection.sh checked Scheduled AND Pulled/Pulling events. The combined.sh version would misclassify cold-fallback pods as "warm-adopted" when images were cached. + +**Why this matters:** +The combined experiment is designed to produce the "best case" measurement. Incorrect adoption detection means the RFC results could contain falsely inflated warm pool performance data. + +**How it was resolved:** +Extracted the more thorough version (checking Scheduled + Pulled/Pulling) to lib/common.sh as a shared function. Removed both local implementations. Updated all call sites to pass namespace. + +### FINDING-6 +- **Severity:** Important +- **Confidence:** 90 +- **File:** experiments/measure-env-injection.sh, experiments/measure-combined.sh +- **Category:** production-readiness +- **Source:** production-agent +- **Round found:** 1 +- **Resolution:** remaining (low risk) + +**What is wrong:** +Both scripts patch SandboxTemplate envVarsInjectionPolicy and readiness probes but never restore original values on interrupt. If interrupted mid-run, the template remains modified. + +### FINDING-7 +- **Severity:** Important +- **Confidence:** 85 +- **File:** experiments/measure-warm-pool.sh:302-339 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Round found:** 1 +- **Resolution:** remaining (measurement artifact, documented) + +**What is wrong:** +Burst claim latencies are measured sequentially despite being submitted in parallel. All 5 burst claims share a single `create_ts`, but readiness checks run sequentially. Claim j's `ready_ts` is captured only after claims 1 through j-1 have been waited on, inflating later claims' measured latency. + +## Minor Findings + +### FINDING-8 (Minor, correctness) +- **File:** experiments/lib/common.sh:12 +- **Source:** correctness-agent (also: architecture-agent) +- **Description:** `deregister_cleanup()` uses bash pattern substitution `${array[@]/$1}` which does substring matching, not exact match. Currently unused by any script. + +### FINDING-9 (Minor, architecture) +- **File:** experiments/lib/wait-ready.sh:7-10 +- **Source:** architecture-agent +- **Description:** `wait_for_ready()` is defined but never called. Dead code. + +### FINDING-10 (Minor, architecture) +- **File:** experiments/measure-combined.sh, experiments/measure-env-injection.sh +- **Source:** architecture-agent +- **Description:** `wait_claim_ready()`, `cleanup_claim()`, `get_claim_pod()` remain duplicated across two scripts. Should be extracted to lib/. + +### FINDING-11 (Minor, architecture) +- **File:** experiments/measure-combined.sh, experiments/measure-env-injection.sh +- **Source:** architecture-agent +- **Description:** Both scripts bypass `write_csv_header()`/`write_csv_row()`, writing CSV rows directly with `echo` and custom headers (adding adoption/behavior columns). + +### FINDING-12 (Minor, security) +- **File:** experiments/lib/common.sh:17 +- **Source:** security-agent +- **Description:** `kubectl delete $res` in cleanup handler uses unquoted variable expansion. Low risk since resource names are controlled. + +### FINDING-13 (Minor, security) +- **File:** experiments/manifests/sandbox-claim.yaml:14-15 +- **Source:** security-agent +- **Description:** Commented-out example includes predictable synthetic token `tok-xxx`. Clearly test data, no real risk. + +### FINDING-14 (Minor, production-readiness) +- **File:** experiments/manifests/sidecar-readiness.yaml:6-19 +- **Source:** production-agent +- **Description:** Readiness sidecar init container has no resource requests or limits. + +### FINDING-15 (Minor, test-quality) +- **File:** experiments/measure-warm-pool.sh:370-378 +- **Source:** test-quality-agent +- **Description:** Aggressive probe config does not verify that warm pool pods actually have updated probe configuration before measuring. If the pool controller doesn't recycle pods on template changes, measurements may use old probes. + +## Notable Observations + +### NOTABLE-1 +- **File:** experiments/sidecar/main.go:35-46 +- **Category:** architecture +- **Source:** architecture-agent +- **Description:** Sidecar readiness binary exits polling loop after signal file detection, stays "ready" permanently even if file is deleted +- **Rationale:** Adequate for the experiment but worth noting if pattern is promoted to production + +### NOTABLE-2 +- **File:** experiments/measure-warm-pool.sh:303-305 +- **Category:** correctness +- **Source:** correctness-agent +- **Description:** Background `kubectl apply` failures in burst mode are silently swallowed by `wait` with no operands +- **Rationale:** Transient API server errors during burst would be misdiagnosed as bind timeouts + +### NOTABLE-3 +- **File:** experiments/measure-combined.sh:199-296 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Description:** Combined script does not wait for pool replenishment between runs, unlike measure-warm-pool.sh +- **Rationale:** Later runs may measure cold-start fallback latency instead of warm pool claims + +### NOTABLE-4 +- **File:** rfc/NNNN-warm-pool-feasibility/README.md +- **Category:** architecture +- **Source:** architecture-agent +- **Description:** RFC document has TBD placeholders throughout all results and recommendation sections +- **Rationale:** Expected at this stage (pre-experiment), document structure is complete and ready to receive data + +### NOTABLE-5 +- **File:** experiments/manifests/sidecar-readiness.yaml, readiness-gate-pod.yaml +- **Category:** security +- **Source:** security-agent +- **Description:** Pod manifests lack securityContext (no runAsNonRoot, no dropped capabilities) +- **Rationale:** Acceptable for ephemeral experiment pods + +### NOTABLE-6 +- **File:** experiments/measure-cold-start.sh:85-138 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Description:** The `noprepull` config uses the same code path as `prepulled` with no validation that images are actually uncached. If the prepull DaemonSet is still running, results will be indistinguishable. +- **Rationale:** Worth adding a pre-flight warning but not blocking + +## Test Suite Results + +No test command detected; post-fix test step was skipped. diff --git a/specs/001-warm-pool-feasibility/spec.md b/specs/001-warm-pool-feasibility/spec.md new file mode 100644 index 0000000000..ea0d1fd574 --- /dev/null +++ b/specs/001-warm-pool-feasibility/spec.md @@ -0,0 +1,148 @@ +# Feature Specification: Warm Pool Feasibility Study + +**Feature Branch**: `6111-warm-pool-feasibility` +**Created**: 2026-07-09 +**Status**: Draft +**Input**: Evaluate Kubernetes Agent Sandbox warm pooling on OpenShift to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds, and produce architectural recommendations for OpenShell integration. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Measure Cold-Start Baseline on OpenShift (Priority: P1) + +An engineer provisions a ROSA HCP cluster, deploys OpenShell with the Agent Sandbox operator, and measures the current cold-start sandbox creation latency with per-phase timestamps. This establishes the control measurement against which all warm pool experiments are compared. + +**Why this priority**: Without a reliable baseline, no warm pool improvement can be quantified. This is the foundation for every subsequent experiment. + +**Independent Test**: Can be tested by creating 10+ sandboxes on a clean cluster and recording timestamps from API call to SSH availability. Delivers a latency breakdown table. + +**Acceptance Scenarios**: + +1. **Given** a ROSA HCP cluster with OpenShell deployed via the OpenShell OpenShift deploy chart, **When** the engineer runs `openshell sandbox create --from base` 10 times with pre-pulled images, **Then** per-phase timestamps are captured (scheduled, image pulled, init complete, supervisor ready, SSH available) and p50/p90 latencies are computed. +2. **Given** the same cluster, **When** the engineer runs 5 sandbox creates without pre-pulled images, **Then** the image pull contribution to total latency is quantified separately. +3. **Given** the same cluster, **When** the engineer creates a vanilla Agent Sandbox (no OpenShell) 10 times, **Then** the OpenShell overhead vs. raw Kubernetes overhead is isolated. + +--- + +### User Story 2 - Measure Warm Pool Claim Latency (Priority: P1) + +An engineer creates a SandboxWarmPool with pre-provisioned pods, then measures the time from SandboxClaim creation to the claimed sandbox becoming Ready. Tests multiple readiness configurations to identify the dominant bottleneck. + +**Why this priority**: This is the core feasibility question. If raw warm pool claim-to-ready latency exceeds the target, no amount of OpenShell optimization will help. + +**Independent Test**: Can be tested by creating SandboxTemplate + SandboxWarmPool, then issuing SandboxClaims and measuring claim-to-ready timestamps. Delivers a configuration matrix with latency data. + +**Acceptance Scenarios**: + +1. **Given** a SandboxWarmPool with 5 pre-provisioned replicas using the base sandbox image, **When** a SandboxClaim is created, **Then** the claim-to-ready latency is measured with default readiness probe settings (10s periodSeconds). +2. **Given** the same pool, **When** a SandboxClaim is created with aggressive readiness probes (1s periodSeconds), **Then** the latency improvement over default probes is quantified. +3. **Given** the same pool, **When** 5 SandboxClaims are created simultaneously, **Then** burst claim behavior and pool replenishment time are measured. + +--- + +### User Story 3 - Test Health Check Optimization Patterns (Priority: P2) + +An engineer tests Pod Readiness Gates (KEP-580) and a Knative-style sidecar readiness pattern as alternatives to polling-based readiness probes, measuring whether they eliminate the probe interval bottleneck. + +**Why this priority**: Our research identified the readiness probe interval as the dominant latency bottleneck. These optimization patterns could eliminate the bottleneck entirely, but need validation on OpenShift. + +**Independent Test**: Can be tested by deploying pods with custom ReadinessGate conditions and sidecar containers, then measuring the time from condition/signal flip to pod Ready status. + +**Acceptance Scenarios**: + +1. **Given** a warm-pooled pod with a custom ReadinessGate condition (`sandbox.openshell.io/claimed`) set to False, **When** an external controller patches the condition to True, **Then** the time from patch to pod Ready is measured and recorded. +2. **Given** a warm-pooled pod with a sidecar container that controls readiness via an HTTP endpoint, **When** a signal file is created in a shared emptyDir volume, **Then** the sidecar flips its readiness response and the pod transitions to Ready within measured latency. +3. **Given** both patterns tested, **When** results are compared with probe-based readiness (1s and 10s), **Then** the relative improvement is quantified. + +--- + +### User Story 4 - Test Claim-Time Environment Injection (Priority: P2) + +An engineer tests whether SandboxClaim env var injection works without forcing a cold start, validating the mechanism OpenShell would use to bind identity to a pre-provisioned pod. + +**Why this priority**: OpenShell needs to inject sandbox identity (ID, endpoint, credentials) at claim time. If env var injection triggers cold start, the warm pool advantage is lost and a different identity binding mechanism is needed. + +**Independent Test**: Can be tested by creating a SandboxTemplate with `envVarsInjectionPolicy: Allowed` and a SandboxClaim with env vars, then observing whether the claim adopts a warm sandbox or creates a new one. + +**Acceptance Scenarios**: + +1. **Given** a SandboxTemplate with `envVarsInjectionPolicy: Allowed` and a SandboxWarmPool with 3 replicas, **When** a SandboxClaim is created with env vars (`OPENSHELL_SANDBOX_ID=test-123`), **Then** the claim adopts an existing warm sandbox (not cold start). +2. **Given** the same setup, **When** a SandboxClaim is created with env vars and `envVarsInjectionPolicy: Disallowed` on the template, **Then** the behavior (rejection or cold fallback) is documented. + +--- + +### User Story 5 - Produce Results Document with Recommendations (Priority: P1) + +An engineer compiles all measurement data into a structured report comparing cold-start vs. warm pool latencies across configurations, and produces architectural recommendations for how OpenShell should integrate warm pooling. + +**Why this priority**: The deliverable. Without this document, the experiments have no impact on the OpenShell project direction. + +**Independent Test**: Can be validated by checking that the document contains: raw data tables, per-configuration comparisons, a clear recommendation for Issue #2157, and concrete next steps for the OpenShell core team. + +**Acceptance Scenarios**: + +1. **Given** all experiments are complete, **When** the results RFC is written, **Then** it contains latency tables with p50/p90 for each configuration, a comparison chart, and a recommendation for the OpenShell integration approach. +2. **Given** the results RFC is complete, **Then** it is structured so a distilled summary can later be posted to GitHub Issue #2157 as a separate follow-up step. + +--- + +### Edge Cases + +- What happens when the warm pool is exhausted (all replicas claimed)? Does SandboxClaim fall back to cold start or stay Pending? +- What happens when a warm-pooled pod is on a node that goes NotReady during the experiment? +- How does pool replenishment interact with cluster autoscaler if nodes are at capacity? +- What if the Red Hat Agent Sandbox operator tech preview does not include extension CRDs? (Fallback: install upstream extensions.yaml manually) + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: Engineer MUST be able to provision a ROSA HCP cluster with a Kubernetes version supporting native sidecar containers (K8s 1.33+) +- **FR-002**: Engineer MUST be able to install the Agent Sandbox operator with both core and extension CRDs on the cluster +- **FR-003**: Engineer MUST be able to deploy OpenShell on the cluster using the OpenShell OpenShift deploy wrapper and create cold-start sandboxes +- **FR-004**: Measurement scripts MUST capture per-phase timestamps (API call, scheduled, image pulled, init complete, and for OpenShell experiments: supervisor ready, SSH available) +- **FR-005**: Shell-based measurement scripts MUST produce CSV output with run number, configuration label, and per-phase timestamps +- **FR-006**: Engineer MUST be able to create SandboxTemplate, SandboxWarmPool, and SandboxClaim resources for warm pool experiments +- **FR-007**: Engineer MUST be able to configure Pod Readiness Gates on warm-pooled pods and flip them via kubectl patch +- **FR-008**: Engineer MUST be able to deploy a sidecar container with a controllable readiness endpoint for sidecar readiness experiments +- **FR-009**: Results MUST be published as a standalone RFC in `rfc/` containing raw latency data, configuration matrix, and architectural recommendations +- **FR-010**: The RFC MUST be self-contained so it can later be distilled into a GitHub Issue #2157 comment (posting is a separate follow-up step, not part of this study) + +### Key Entities + +- **ROSA HCP Cluster**: The test environment. Provisioned via ROSA plugin, us-east-2 region, AAET profile. +- **Agent Sandbox CRDs**: Sandbox (core), SandboxTemplate, SandboxWarmPool, SandboxClaim (extensions). The primitives being evaluated. +- **Measurement Run**: A single sandbox creation or claim with captured timestamps. N=10-20 per configuration. +- **Configuration**: A specific combination of settings (probe interval, readiness gate, sidecar pattern, env var injection) being tested. +- **Results RFC**: The final deliverable in `rfc/`, synthesizing all measurements into architectural recommendations. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Cold-start baseline latency is measured with per-phase breakdown for at least 10 runs with pre-pulled images and 5 runs without +- **SC-002**: Warm pool claim-to-ready latency is measured across at least 4 configurations (default probes, aggressive probes, readiness gates, sidecar pattern) +- **SC-003**: Each configuration has at least 10 measurement runs with p50 and p90 latencies computed +- **SC-004**: The env var injection experiment conclusively determines whether claim-time injection triggers cold start or not +- **SC-005**: The results document contains a clear, data-backed recommendation for which warm pool integration approach OpenShell should pursue +- **SC-006**: The results RFC contains all required sections (data tables, configuration comparisons, architectural recommendation, next steps) and is structured for later distillation into a GitHub Issue #2157 comment + +## Clarifications + +### Session 2026-07-09 + +- Q: What ROSA HCP cluster sizing should be used for the experiments? → A: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each). Cluster is short-lived (tear down after experiments). +- Q: Where should the full results document be stored? → A: Standalone RFC in `rfc/` directory of the OpenShell repo. +- Q: What level of measurement automation is expected? → A: Shell scripts per experiment that wrap kubectl with timestamp capture (CSV output). +- Q: Should the study prioritize the Red Hat Agent Sandbox tech preview or upstream manifests? → A: Red Hat tech preview first, upstream manifests only as fallback. + +## Assumptions + +- A ROSA HCP cluster with OpenShift 4.20+ (K8s 1.33+) is available for provisioning via the AAET AWS profile +- The cluster uses 3 worker nodes of type m5.2xlarge (8 vCPU, 32 GB each) in us-east-2, provisioned as a short-lived experiment cluster +- The Red Hat Agent Sandbox operator tech preview is installable from OperatorHub (if not, upstream manifests are used as fallback) +- the OpenShell OpenShift deploy wrapper works with the target OpenShift version (may need minor adjustments) +- Pre-pulling images on worker nodes is sufficient to eliminate image pull latency from measurements +- The upstream Agent Sandbox v0.5.0 extension CRDs (v1beta1) are compatible with the Red Hat operator build +- Pod Readiness Gates (KEP-580, GA since K8s 1.14) work on OpenShift without additional configuration +- Native sidecar containers (KEP-753, GA since K8s 1.33) are available on the target OpenShift version +- This study is scoped to Kubernetes-backed sandboxes only; Podman/Docker single-player warm pooling is out of scope diff --git a/specs/001-warm-pool-feasibility/tasks.md b/specs/001-warm-pool-feasibility/tasks.md new file mode 100644 index 0000000000..1df9348780 --- /dev/null +++ b/specs/001-warm-pool-feasibility/tasks.md @@ -0,0 +1,175 @@ +# Tasks: Warm Pool Feasibility Study + +**Input**: Design documents from `specs/001-warm-pool-feasibility/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md + +**Tests**: Not applicable (feasibility study with manual experiment validation). + +**Organization**: Tasks are grouped by user story from spec.md. US1 and US2 are P1 (core feasibility), US3-US4 are P2 (optimization experiments), US5 is P1 (deliverable). + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2) +- Include exact file paths in descriptions + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Cluster provisioning, operator installation, and shared tooling + +- [ ] T001 Provision ROSA HCP cluster with 3x m5.2xlarge workers using AAET profile (via `rosa create cluster` CLI or the `rosa:create` Claude Code skill) +- [ ] T002 Install Red Hat Agent Sandbox operator from OperatorHub (fallback: apply upstream `manifest.yaml` + `extensions.yaml` from kubernetes-sigs/agent-sandbox v0.5.0) +- [ ] T003 Verify all four CRDs are registered: Sandbox, SandboxTemplate, SandboxWarmPool, SandboxClaim via `kubectl api-resources | grep agents` +- [ ] T004 Deploy OpenShell on the cluster using the OpenShift deploy wrapper (github.com/2000krysztof/Openshell-Openshift-Deploy) +- [ ] T005 Validate cold-start sandbox creation works end-to-end via `openshell sandbox create --from base` +- [X] T006 Create image pre-pull DaemonSet to pre-pull sandbox images on all worker nodes in `experiments/manifests/image-prepull-daemonset.yaml` +- [ ] T007 Apply pre-pull DaemonSet and verify images are cached on all 3 nodes + +**Checkpoint**: Cluster running, operator installed, OpenShell functional, images pre-pulled + +--- + +## Phase 2: Foundational (Measurement Library) + +**Purpose**: Shared measurement infrastructure that all experiments depend on + +- [X] T008 Create `experiments/` directory structure per plan.md project structure +- [X] T009 Implement shared measurement functions (timestamp capture, CSV output, pod event collection) in `experiments/lib/common.sh` +- [X] T010 Implement pod readiness wait with configurable timeout in `experiments/lib/wait-ready.sh` +- [X] T011 [P] Create SandboxTemplate manifest for warm pool experiments in `experiments/manifests/sandbox-template.yaml` +- [X] T012 [P] Create SandboxWarmPool manifest (5 replicas, configurable readiness) in `experiments/manifests/warm-pool.yaml` +- [X] T013 [P] Create SandboxClaim manifest template (with/without env vars) in `experiments/manifests/sandbox-claim.yaml` + +**Checkpoint**: Measurement library ready, manifests templated. Experiment scripts can now be built. + +--- + +## Phase 3: User Story 1 - Measure Cold-Start Baseline (Priority: P1) MVP + +**Goal**: Establish cold-start sandbox creation latency with per-phase breakdown as the control measurement. + +**Independent Test**: Run 10+ sandboxes on the cluster and verify CSV output contains per-phase timestamps with computed p50/p90. + +- [X] T014 [US1] Implement cold-start measurement script that creates sandboxes via `openshell sandbox create --from base`, captures per-phase timestamps (scheduled, image pulled, init complete, supervisor ready, SSH available), and outputs CSV in `experiments/measure-cold-start.sh` +- [ ] T015 [US1] Run Experiment 1a: 10 cold-start runs with pre-pulled images, save results to `experiments/results/cold-start-prepulled.csv` +- [ ] T016 [US1] Run Experiment 1b: 5 cold-start runs without pre-pulled images, save results to `experiments/results/cold-start-noprepull.csv` +- [ ] T017 [US1] Run Experiment 1c: 10 vanilla Agent Sandbox creates (no OpenShell) to isolate OpenShell overhead, save results to `experiments/results/cold-start-vanilla.csv` +- [ ] T018 [US1] Compute p50/p90 latencies for all three cold-start configurations and create summary table + +**Checkpoint**: Cold-start baseline established with per-phase breakdown. + +--- + +## Phase 4: User Story 2 - Measure Warm Pool Claim Latency (Priority: P1) + +**Goal**: Measure raw warm pool claim-to-ready latency across probe configurations. Core feasibility answer. + +**Independent Test**: Create SandboxTemplate + SandboxWarmPool, issue SandboxClaims, verify claim-to-ready CSV output with latency data. + +- [X] T019 [US2] Implement warm pool measurement script that creates SandboxClaims against a running warm pool, captures claim-to-ready timestamps, and outputs CSV in `experiments/measure-warm-pool.sh` +- [ ] T020 [US2] Deploy SandboxWarmPool with 5 replicas and verify all replicas reach provisioned state +- [ ] T021 [US2] Run Experiment 2a: 10 claims with default readiness probes (10s periodSeconds), save results to `experiments/results/warm-pool-default.csv` +- [ ] T022 [US2] Run Experiment 2b: 10 claims with aggressive readiness probes (1s periodSeconds), save results to `experiments/results/warm-pool-aggressive.csv` +- [ ] T023 [US2] Run Experiment 2c: 5 simultaneous claims to measure burst behavior and pool replenishment, save results to `experiments/results/warm-pool-burst.csv` +- [ ] T024 [US2] Document pool exhaustion behavior: what happens when all replicas are claimed +- [ ] T025 [US2] Compute p50/p90 for all warm pool configurations and create comparison table vs cold-start + +**Checkpoint**: Core feasibility question answered. Warm pool claim-to-ready latency measured. + +--- + +## Phase 5: User Story 3 - Test Health Check Optimization Patterns (Priority: P2) + +**Goal**: Test Pod Readiness Gates and sidecar readiness as alternatives to polling-based probes. + +**Independent Test**: Deploy pods with ReadinessGate conditions and sidecar containers, measure condition-flip-to-Ready latency. + +- [X] T026 [P] [US3] Create pod manifest with custom ReadinessGate condition (`sandbox.openshell.io/claimed`) in `experiments/manifests/readiness-gate-pod.yaml` +- [X] T027 [P] [US3] Implement sidecar readiness binary (Go, HTTP 503/200, signal file watch) in `experiments/sidecar/main.go` +- [X] T028 [P] [US3] Create Dockerfile for sidecar binary (multi-stage, scratch base) in `experiments/sidecar/Dockerfile` +- [ ] T029 [US3] Build and push sidecar readiness image to ghcr.io via `experiments/sidecar/Makefile` +- [X] T030 [US3] Create sidecar readiness pod manifest (init container with restartPolicy: Always, shared emptyDir) in `experiments/manifests/sidecar-readiness.yaml` +- [X] T031 [US3] Implement readiness gate measurement script that patches ReadinessGate condition and measures flip-to-Ready latency in `experiments/measure-readiness-gates.sh` +- [X] T032 [US3] Implement sidecar readiness measurement script that triggers signal file and measures flip-to-Ready latency in `experiments/measure-sidecar-readiness.sh` +- [ ] T033 [US3] Run Experiment 3a: 10 readiness gate measurements, save to `experiments/results/readiness-gates.csv` +- [ ] T034 [US3] Run Experiment 3b: 10 sidecar readiness measurements, save to `experiments/results/sidecar-readiness.csv` +- [ ] T035 [US3] Create comparison table: default probes vs aggressive probes vs readiness gates vs sidecar pattern + +**Checkpoint**: Health check optimization patterns validated with latency data. + +--- + +## Phase 6: User Story 4 - Test Claim-Time Environment Injection (Priority: P2) + +**Goal**: Determine whether SandboxClaim env var injection triggers cold start or adopts warm sandbox. + +**Independent Test**: Create SandboxTemplate with `envVarsInjectionPolicy: Allowed`, issue SandboxClaim with env vars, observe warm adoption. + +- [X] T036 [US4] Update SandboxTemplate manifest with `envVarsInjectionPolicy: Allowed` variant in `experiments/manifests/sandbox-template.yaml` +- [X] T037 [US4] Implement env var injection measurement script in `experiments/measure-env-injection.sh` +- [ ] T038 [US4] Run Experiment 5a: 5 claims with env vars and Allowed policy, verify warm adoption, save to `experiments/results/env-injection-allowed.csv` +- [ ] T039 [US4] Run Experiment 5b: Document behavior with Disallowed policy (rejection vs cold fallback) +- [X] T040 [US4] Implement combined measurement script (best readiness pattern + env injection) in `experiments/measure-combined.sh` +- [ ] T041 [US4] Run Experiment 6: 10 combined measurements, save to `experiments/results/combined.csv` + +**Checkpoint**: Env var injection behavior conclusively documented. Combined best-case latency measured. + +--- + +## Phase 7: User Story 5 - Produce Results RFC (Priority: P1) + +**Goal**: Compile all measurement data into a structured RFC with architectural recommendations. + +**Independent Test**: RFC contains raw data tables, per-configuration comparisons, and a clear recommendation for OpenShell warm pool integration. + +- [X] T042 [US5] Create RFC directory structure `rfc/NNNN-warm-pool-feasibility/README.md` (number TBD by maintainers, use placeholder) +- [ ] T043 [US5] Write RFC Executive Summary section: can warm pooling hit sub-2s? What is the dominant bottleneck? +- [ ] T044 [US5] Write RFC Experiment Setup section: cluster config, operator version, OpenShell version, image pre-pull status +- [ ] T045 [US5] Write RFC Results section: compile all CSV data into p50/p90 tables per configuration with comparison charts +- [ ] T046 [US5] Write RFC Health Check Analysis section: probe interval impact, readiness gates performance, sidecar pattern performance +- [ ] T047 [US5] Write RFC Env Var Injection section: injection behavior, policy requirements, identity binding constraints +- [ ] T048 [US5] Write RFC Architecture Recommendations section: Kubernetes driver changes, supervisor changes, gateway store changes, identity binding mechanism, Issue #2157 recommendation +- [ ] T049 [US5] Write RFC Gaps and Risks section: missing Agent Sandbox extension features, Red Hat TP coverage gaps, pool replenishment under burst +- [ ] T050 [US5] Write RFC Next Steps section: upstream contributions, internal work items, concrete action items + +**Checkpoint**: RFC complete, ready for review. + +--- + +## Phase 8: Polish & Teardown + +**Purpose**: Cleanup, validation, and cluster teardown + +- [ ] T051 Validate all CSV result files exist and contain expected number of runs +- [ ] T052 Review RFC for completeness: all required sections present, data tables populated, recommendation clear +- [ ] T053 Tear down ROSA HCP cluster via `rosa:delete warm-pool-study` + +--- + +## Dependencies + +```text +Phase 1 (Setup) + └── Phase 2 (Measurement Library) + ├── Phase 3 (US1: Cold-Start Baseline) ─── MVP + │ └── Phase 4 (US2: Warm Pool Claims) + │ ├── Phase 5 (US3: Health Check Optimization) ──┐ + │ └── Phase 6 (US4: Env Var Injection) ──────────┤ + │ └── Phase 7 (US5: Results RFC) + │ └── Phase 8 (Teardown) +``` + +**Critical path**: Setup → Library → Cold-Start → Warm Pool → Health Check Optimization → RFC → Teardown + +**Parallel opportunities**: +- US3 and US4 can run in parallel after US2 completes +- Within US3: readiness gate manifest (T026), sidecar binary (T027), and sidecar Dockerfile (T028) can be built in parallel +- Within US2: warm pool deployment (T020) can begin while measurement script (T019) is being written + +## Implementation Strategy + +**MVP**: Phases 1-4 (Setup + Library + Cold-Start Baseline + Warm Pool Claims). This answers the core feasibility question: can warm pooling deliver sub-2s latency on OpenShift? + +**Full scope**: All 8 phases. The health check optimization and env var injection experiments refine the recommendation, and the RFC captures everything for the upstream community. + +**Incremental delivery**: Each phase produces independently verifiable output (CSV data files). If the warm pool latency exceeds target in Phase 4, Phases 5-6 may pivot to investigating why rather than optimizing further.