test(dsv2lite): add retained HTTP soak gate

docs/index.md

@@ -89,9 +89,9 @@ Organized by domain (model line / subsystem / playbook / lesson) instead of by l
 
 | Path | TL;DR |
 | --- | --- |
-| `models/deepseek-v2-lite/status.md` | DeepSeek-V2-Lite EP2 status ledger: correctness, direct attribution, lifecycle reliability, and the #466 six-child host-staged/NCCL HTTP SLO report remain separate evidence buckets; no soak, parity, or production claim. |
-| `models/deepseek-v2-lite/benchmarking.md` | Verification ladder and commands for DSV2-Lite correctness, direct decode diagnostics, retained host-staged/NCCL HTTP SLO profiles, and the separate soak/production boundary. |
-| `models/deepseek-v2-lite/roadmap.md` | Single-node EP2 roadmap: #466 retained SLO reporting is an evidence layer; soak, device KV/attention, long-prefill scheduling, and Stable promotion remain separate gates. |
+| `models/deepseek-v2-lite/status.md` | DeepSeek-V2-Lite EP2 status ledger: correctness, direct attribution, lifecycle reliability, #466 HTTP SLO reports, and #465 host-staged/NCCL soak evidence remain separate buckets; no parity or production claim. |
+| `models/deepseek-v2-lite/benchmarking.md` | Verification ladder and commands for DSV2-Lite correctness, direct decode diagnostics, retained host-staged/NCCL HTTP SLO profiles, and #465 sustained soak evidence. |
+| `models/deepseek-v2-lite/roadmap.md` | Single-node EP2 roadmap: #466 SLO and #465 short-shape soak are retained evidence layers; device KV/attention, long-prefill scheduling, hard budgets, and Stable promotion remain separate gates. |
 | `models/deepseek-v2-lite/benchmark-artifact-manifest.md` | Issue #467 implemented: the retained DeepSeek-V2-Lite benchmark matrix emits `artifact_manifest.json` and `regression_summary.json`, with CPU-only summarize-only tests. |
 | `models/deepseek-v2-lite/hf-accuracy-gate.md` | DeepSeek-V2-Lite EP2 HF accuracy gate after PR #149/#150/#274: HF `generate(use_cache=true)`, host-staged EP2, and NCCL EP2 are compared across the committed small case set. |
 | `models/deepseek-v2-lite/decode-attribution-gate.md` | Direct diagnostic benchmark for DeepSeek-V2-Lite EP2 `Hello`/16-token batch sizes 1/4/8: structured timing/counters and graph probes with no HTTP SLO or production claim. |

docs/models/deepseek-v2-lite/benchmarking.md

@@ -1,6 +1,6 @@
 # DeepSeek-V2-Lite Verification And Benchmarking
 
-> **TL;DR:** Use `e2e_ep2` for correctness, `dsv2_lite_ep2_decode_attribution` for direct decode diagnostics, and `bench_dsv2lite_http_slo.py` for retained HTTP SLO evidence. The #466 follow-up NCCL readiness fix lets the backend discover a compatible Python-wheel NCCL runtime from `PATH`; these artifacts still answer different questions, and none of them alone proves production readiness.
+> **TL;DR:** Use `e2e_ep2` for correctness, `dsv2_lite_ep2_decode_attribution` for direct decode diagnostics, `bench_dsv2lite_http_slo.py` for retained HTTP SLO evidence, and `bench_dsv2lite_http_soak.py` for issue #465 sustained HTTP soak evidence. These artifacts still answer different questions, and none of them alone proves production readiness.
 >
 > Last touched: 2026-07
 
@@ -11,7 +11,7 @@
 | Correctness / integration | `openinfer-deepseek-v2-lite/tests/e2e_ep2.rs` | EP2 load, host-staged/NCCL generation, request isolation, output tokens/text/hashes, route and collective accounting | Latency, throughput, SLO, soak, production readiness |
 | Direct diagnostic | `dsv2_lite_ep2_decode_attribution` | Fixed-shape CPU/CUDA section timing, route/collective counters, graph-readiness diagnostics | HTTP behavior, client pressure, serving SLO |
 | HTTP serving SLO | `scripts/bench_dsv2lite_http_slo.py` over the shared HTTP harness | Streaming TTFT/TPOT/ITL, request/output throughput, failures/timeouts, server trace coverage, output hashes, repeat spread | Direct-kernel attribution, sustained soak, production readiness |
-| Soak / production readiness | Separate sustained-run gate | Memory drift, long-duration tails, recovery and deployment limits | Out of scope for issue #466 |
+| Soak / production readiness | `scripts/bench_dsv2lite_http_soak.py` | Sustained request completion, first/last-quartile tail and throughput drift, RSS/VRAM drift, terminal reasons, clean follow-up recovery | Direct decode attribution, vLLM parity, multi-node recovery, or production readiness by itself |
 
 ## Correctness Gate
 
@@ -132,20 +132,91 @@ Each `sweep_summary.json` records:
 
 A cell is `noisy` when repeat spread exceeds 10% of the median for TTFT/TPOT/ITL p95, request throughput, or output-token throughput.
 
+## Sustained HTTP Soak
+
+Issue #465 uses `scripts/bench_dsv2lite_http_soak.py`. It reuses the generic streaming `/v1/completions` leaf benchmark inside fixed time buckets, then writes a backend-level `soak_summary.json` and an optional host-staged/NCCL combined report. A failed soak is still useful evidence when the JSON keeps the failing leaf artifact, server log pointer, terminal reasons, and clean follow-up result.
+
+The default retained shape is greedy, ignore-EOS, `prompt_words=64`, `max_tokens=64`, concurrency `4,8`, and no production latency budget. Tune `--duration-s`, `--bucket-s`, and `--num-requests` for the validation host; `--bucket-s` controls the target wall-clock window per bucket, while `--num-requests` controls each leaf chunk launched continuously inside that window. The summary records actual elapsed time, bucket count, and leaf count.
+
+```bash
+# Template: run one backend soak after the server is ready.
+python3 scripts/bench_dsv2lite_http_soak.py run \
+  --backend BACKEND \
+  --base-url http://127.0.0.1:18000 \
+  --server-log artifacts/bench/dsv2-lite/RUN_ID/BACKEND/server.log \
+  --model-path MODEL_PATH \
+  --server-command "SERVER_COMMAND" \
+  --commit COMMIT \
+  --model-revision MODEL_REVISION \
+  --server-binary SERVER_BINARY \
+  --backend-runtime-version BACKEND_RUNTIME_VERSION \
+  --duration-s 1800 \
+  --bucket-s 300 \
+  --num-requests 32 \
+  --concurrency 4,8 \
+  --prompt-words 64 \
+  --max-tokens 64 \
+  --out-dir artifacts/bench/dsv2-lite/RUN_ID/BACKEND/soak
+
+# Template: combine host-staged and NCCL backend summaries.
+python3 scripts/bench_dsv2lite_http_soak.py combine \
+  --summary artifacts/bench/dsv2-lite/RUN_ID/host-staged/soak/soak_summary.json \
+  --summary artifacts/bench/dsv2-lite/RUN_ID/nccl/soak/soak_summary.json \
+  --out artifacts/bench/dsv2-lite/RUN_ID/retained_soak_report.json
+```
+
+Each backend summary records:
+
+- leaf artifact path and SHA-256 for every leaf chunk inside each bucket;
+- completed, failed, timeout, terminal-reason, and error counts;
+- TTFT, TPOT, ITL, request throughput, and output-token throughput per bucket;
+- first/last-quartile drift summaries for tails, throughput, RSS, and VRAM;
+- active-set, pending-queue, decode-batch, token-timing, and missing-trace coverage when server traces are available;
+- output hash distribution and combined hash;
+- process RSS and total device memory samples;
+- post-soak clean follow-up result.
+
+`soak_gate.passed` means every requested concurrency has loaded bucket evidence, leaf commands completed successfully with zero failures/timeouts, optional trace coverage passed for every bucket, and the clean follow-up completed. Numeric drift is reported but not a hard budget until deployment limits are ratified.
+
+The combined host-staged/NCCL report hard-fails when either backend is missing, a child `soak_gate` fails, the child commits differ, model/server provenance differs, or a backend runtime boundary is missing or generic. This keeps NCCL selector evidence attached to the soak result instead of turning a conservative runtime run into a default-runtime claim.
+
 ## Local Tooling Gate
 
-These commands were run on 2026-07-13:
+These commands were run on 2026-07-18:
 
 ```bash
-python3 -m py_compile scripts/bench_http_common.py scripts/bench_http_serving.py scripts/bench_http_sweep.py scripts/bench_dsv2lite_http_slo.py
-python3 -m unittest -v tests/test_bench_http_serving.py tests/test_bench_http_sweep.py tests/test_bench_dsv2lite_http_slo.py
+python3 -m py_compile scripts/bench_http_common.py scripts/bench_http_serving.py scripts/bench_http_sweep.py scripts/bench_dsv2lite_http_slo.py scripts/bench_dsv2lite_http_soak.py
+python3 -m unittest -v tests/test_bench_http_common.py tests/test_bench_http_serving.py tests/test_bench_http_sweep.py tests/test_bench_dsv2lite_http_slo.py tests/test_bench_dsv2lite_http_soak.py
 cargo fmt --all --check
 cargo metadata --locked --no-deps --format-version 1
 ```
 
 ## Current-Source Evidence
 
-Regenerate the retained report when its profile, schema, or measurement contract changes. The current retained 2x RTX 5090 evidence completed all six host-staged/NCCL children with zero failures/timeouts and full required trace coverage. The gitignored local copy is under `artifacts/bench/dsv2-lite/<run-id>/`; aggregate SHA-256: `a7e677c63d1ce92ad0c069f83acfcc8b381e07d06bed9364ab899adecef8d317`.
+Regenerate retained reports when their profile, schema, or measurement contract changes. The gitignored local copies live under `artifacts/bench/dsv2-lite/<run-id>/`; keep public docs to artifact basenames, hashes, contract shape, and claim boundaries.
+
+### Issue #465 Retained HTTP Soak
+
+The retained #465 latest-HEAD rerun used code commit `a5703d0424d917ce99b4bd8691b0b86eecde966f`, model revision `604d5664dddd88a0433dbae533b7fe9472482de0`, 2x RTX 5090, `prompt_words=64`, `max_tokens=64`, greedy sampling, ignore-EOS, concurrency `4,8`, `duration_s=120`, `bucket_s=60`, `num_requests=8`, and full required trace coverage. Later commits may update docs or tooling gates; regenerate the artifacts before claiming evidence for a changed benchmark schema or runtime path. The combined report SHA-256 is `1c06e8825da70888277f1485f54d7f4fb9b2f61d617149d3ac7357cd5a03e7f1`.
+
+| Artifact basename | Backend | SHA-256 | Result | Boundary |
+| --- | --- | --- | --- | --- |
+| `soak_summary.json` | host-staged | `3fe4f163024602a51f10cac0c15cc24a5feffd92bbcb93091e175cd63e49bd33` | `completed=112`, `failed=0`, `timeouts=0`, `soak_gate.passed=true`, clean follow-up passed, combined output hash `6912777bed672f57` | Host-staged sustained HTTP soak evidence |
+| `soak_summary.json` | NCCL | `52c2a70d895fc0c080233961ea06824097fbf5b7241a0983e41c0ca6646e762f` | `completed=128`, `failed=0`, `timeouts=0`, `soak_gate.passed=true`, clean follow-up passed, combined output hash `24f2db9fc47acc10` | NCCL sustained HTTP soak evidence with NCCL `2.26.2`, `NCCL_IB_DISABLE=1`, `NCCL_P2P_DISABLE=1` |
+| `retained_soak_report.json` | combined | `1c06e8825da70888277f1485f54d7f4fb9b2f61d617149d3ac7357cd5a03e7f1` | `coverage_gate.passed=true`, `child_gates={host-staged:true,nccl:true}`, commit/model provenance consistent, runtime boundaries present | Combined host-staged/NCCL #465 report |
+
+Resource and drift fields are retained for diagnosis, not hard budgets. Host-staged max device memory stayed flat in both concurrency buckets, with output-token throughput drift `-1.5%` at c4 and `-0.8%` at c8; RSS moved `+1.6%` at c4 and `+12.1%` at c8. NCCL max device memory stayed flat, with output-token throughput drift `-1.0%` at c4 and `+1.1%` at c8; RSS moved `+1.6%` at c4 and `+3.8%` at c8. The NCCL server used a conservative single-node runtime selection and reached readiness after the NCCL communicator init path completed; do not turn this row into a default-runtime speed claim.
+
+Reviewer evidence summary for issue/PR text:
+
+- Combined artifact basename/SHA: `retained_soak_report.json` / `1c06e8825da70888277f1485f54d7f4fb9b2f61d617149d3ac7357cd5a03e7f1`.
+- Child artifact basenames/SHA: host-staged `soak_summary.json` / `3fe4f163024602a51f10cac0c15cc24a5feffd92bbcb93091e175cd63e49bd33`; NCCL `soak_summary.json` / `52c2a70d895fc0c080233961ea06824097fbf5b7241a0983e41c0ca6646e762f`.
+- Gate result: host-staged and NCCL child gates passed, combined gate passed, commit/model provenance consistent, runtime boundary retained.
+- Claim boundary: short-shape sustained HTTP soak evidence only; no production-readiness, default-runtime NCCL, long/mixed-prompt, or vLLM-parity claim.
+
+### Issue #466 Retained HTTP SLO Evidence
+
+The current retained #466 2x RTX 5090 evidence completed all six host-staged/NCCL children with zero failures/timeouts and full required trace coverage. The gitignored local copy is under `artifacts/bench/dsv2-lite/<run-id>/`; aggregate SHA-256: `a7e677c63d1ce92ad0c069f83acfcc8b381e07d06bed9364ab899adecef8d317`.
 
 ### Issue #466 Follow-Up NCCL Readiness Smoke
 

docs/models/deepseek-v2-lite/roadmap.md

@@ -1,6 +1,6 @@
 # DeepSeek-V2-Lite Serving Roadmap
 
-> **TL;DR:** DeepSeek-V2-Lite has single-node EP2 correctness, request-lifecycle reliability, direct diagnostics, and retained HTTP SLO reporting. Production readiness still requires sustained soak, bounded device KV ownership, long-prefill scheduling, and explicit deployment limits.
+> **TL;DR:** DeepSeek-V2-Lite has single-node EP2 correctness, request-lifecycle reliability, direct diagnostics, retained HTTP SLO reporting, and a retained #465 short-shape soak baseline. Production readiness still requires bounded device KV ownership, long-prefill scheduling, ratified budgets, and explicit deployment limits.
 >
 > Last touched: 2026-07
 
@@ -15,7 +15,7 @@ Evidence stays in four buckets:
 3. HTTP serving SLO: fixed `/v1/completions` contracts, tails, throughput, failures/timeouts, trace coverage, hashes, repeat spread.
 4. Soak and production readiness: sustained memory/tail drift, recovery, capacity, deployment and support limits.
 
-The first three can be green while the fourth remains open.
+The first three can be green while the fourth is only a retained baseline. A passing soak report records sustained behavior for a named contract; it does not define production budgets by itself.
 
 ## Current Gates
 
@@ -25,7 +25,7 @@ The first three can be green while the fourth remains open.
 | Direct decode attribution | Retained | `decode-attribution-gate.md` | Direct diagnostic only |
 | HTTP lifecycle reliability | Retained | `status.md`, issue #453 | Failure isolation and recovery scenarios, no long-duration claim |
 | HTTP SLO report | Retained for #466 | `benchmarking.md`, `bench_dsv2lite_http_slo.py` | Fixed host-staged/NCCL HTTP contracts retained; no soak or production claim |
-| Sustained soak | Open | issue #465 | Required before Stable promotion |
+| Sustained soak | Retained baseline for #465 | `benchmarking.md`, `bench_dsv2lite_http_soak.py` | Fixed short-shape host-staged/NCCL soak retained; runtime/provenance gates are hard, drift is reported, not a ratified production budget |
 | Long-prefill scheduling | Open | issue #452 | Current long smoke records the boundary; it does not close latency work |
 | Device attention and KV | Open | issue #635 | Required for bounded device lifetime and stronger scaling |
 
@@ -35,6 +35,14 @@ Issue #466 is an evidence and reporting milestone. It provides named DSV2-Lite p
 
 This closes the missing report layer. It does not optimize latency or throughput and does not close issue #465.
 
+## Issue #465 Position
+
+Issue #465 is the sustained evidence milestone. The retained 2026-07-18 run covered host-staged and NCCL under the fixed short-shape HTTP soak contract with zero failures/timeouts, full required trace coverage, clean follow-up recovery, output hashes, resource samples, and first/last-quartile drift fields. The NCCL row used NCCL `2.26.2` with `NCCL_IB_DISABLE=1` and `NCCL_P2P_DISABLE=1`; keep that runtime boundary attached to the evidence.
+
+This closes the first retained soak baseline. It does not set hard latency or drift budgets, does not cover long-prefill scheduling, and does not claim production readiness.
+
+The tooling now guards the evidence boundary directly: backend summaries fail on missing bucket coverage, leaf command errors, missing trace coverage, or failed clean follow-up, and the combined report fails on missing host-staged/NCCL children, failed child gates, provenance drift, commit drift, or missing/generic runtime boundaries. Long-duration and long/mixed-prompt soaks still need separate versioned contracts before Stable promotion.
+
 ## Sequence
 
 ### 1. Retain The Serving Contract
@@ -44,13 +52,14 @@ This closes the missing report layer. It does not optimize latency or throughput
 - Fail retained runs on request failures, timeouts, missing traces, or missing active/decode coverage.
 - Keep startup failures as structured artifacts instead of dropping failed cells.
 
-### 2. Close Sustained Availability
+### 2. Extend Sustained Availability
 
 Primary issue: #465.
 
-- Run host-staged and NCCL for ratified short and long durations.
-- Track first/last-quartile tails and throughput, RSS/VRAM drift, active/pending state, terminal reasons, and clean follow-up recovery.
-- Calibrate budgets from retained variance before promoting numeric thresholds to hard gates.
+- Keep the retained host-staged/NCCL short-shape soak reproducible after scheduler, frontend, trace, or backend changes.
+- Extend from the short-shape baseline into ratified long-duration and long/mixed-prompt soaks only when those contracts have explicit support limits.
+- Calibrate latency, throughput, RSS, and VRAM budgets from repeated retained variance before promoting numeric thresholds to hard gates.
+- Keep conservative NCCL runtime selectors in the retained evidence until default-runtime startup is separately proven on the supported hardware matrix.
 
 ### 3. Move Decode State To The Device
 
@@ -77,6 +86,7 @@ Promotion requires all of the following:
 - request and KV capacity are bounded and observable;
 - lifecycle and soak gates recover cleanly with no unexplained drift;
 - short, mixed, and long retained SLO reports pass on the supported hardware/runtime matrix;
+- long-duration and long/mixed-prompt soak profiles have ratified duration, shape, and drift budgets;
 - backend startup failures give actionable version or configuration errors;
 - API, topology, context, sampling, and recovery limits are documented;
 - performance claims use matched repeated HTTP contracts, while direct and profiler data remain diagnostic.