feat(planner): 0.6.0 — SOTA serving model (continuous batching, prefill/decode, Pareto, variance guard)#3
Merged
Merged
Conversation
First SOTA increment from the cold review: model the real concurrency limiter
for batched backends. VRAMModel.kv_cache_gb() + max_concurrent_seqs() compute,
from first principles (no fitting), how many sequences one GPU can hold after
weights + activations, bounded by per-sequence KV-cache. Surfaced as
Candidate.max_concurrent_seqs and in plan output ("Max concurrent/GPU").
Foundation for batched throughput (next: GPU FLOPS + continuous-batching curve,
then engine integration of per-GPU batching vs replica fan-out). Verified:
3B Q4 @2k on 24GB -> 89 seqs; 14B Q4 @4k -> 19; 7B Q4 @2k on 12GB -> 25.
442 tests (+6).
…consoles) The top-level --help showed "optimizer <?>" because the help= string used a non-ASCII em-dash (violates the repo's ASCII-only rule and renders as a replacement char on cp1252 terminals). Replaced em-dashes / x with ASCII in the CLI-facing strings, and dropped literal RST backticks from the catalog help so the command list reads "rebuild it with --build" not "``--build``". No new `help` command needed -- Typer already lists all 10 commands with descriptions (`chimeraforge --help`) and full per-command help (`<cmd> --help`).
Repo standard is ASCII-only for universal compatibility. Section dividers used U+2500 (box-drawing), prose used U+2014 (em dash), one docstring used U+00D7 (multiply). Swept all to ASCII hyphen / 'x' across src/chimeraforge and tests (comments + docstrings only; no logic or output strings changed), including the new 0.6.0 code. Folded into the 0.6.0 branch rather than a separate chore branch to avoid divergence. 442 tests pass; ruff clean on src/chimeraforge; zero non-ASCII remain.
Real SLOs are written against time-to-first-token and time-per-output-token, not one atomic service time. Separate the two: - GPUSpec gains fp16_tflops (dense FP16 Tensor, FP32-accumulate). Datacenter from official datasheets (A100 312, H100 SXM 989, L4 121, T4 65); consumer Ada derived (dense = 2x FP32 shader, cross-checked vs RTX 4090's published 165.2); consumer Ampere on the FP32-acc basis (3090 71, 3080 59.5) for one consistent basis. - LatencyModel.predict_ttft_ms(): prefill is compute-bound -- ~2 FLOPs/param/token over the prompt / (peak TFLOPS * MFU=0.4). Returns 0.0 when GPU compute is unknown (latency falls back to decode-only, no guess). - p95 service time is now prefill (TTFT) + avg_tokens * decode-per-token (TPOT), not decode-only. Candidate carries ttft_ms + tpot_ms; plan shows both plus a --prompt-tokens input. Verified: 7B/512-tok TTFT H100 18ms < 4090 108ms < T4 276ms; 14B = 2x 7B; TPOT ~= 1000/tps. 450 tests (+8).
The docs/ folder and product spec still framed the repo as the Banterhearts research breakout, predating the model-agnostic CLI. Audit + fix: - docs/README.md: rewrite the index product-first (plan/suggest/measure/catalog); agent/Rust/TR guides preserved under a clearly-labeled Research Archive; drop the stale "Last Updated: January 2025" framing. - docs/planning.md: NEW task-oriented planner usage guide (the missing product doc). - Dead links fixed: configuration.md (x2) and production_deployment.md removed, lowercase contributing.md -> ../CONTRIBUTING.md. - docs/API.md: v0.2.1 -> v0.5.0. - docs/ARCHITECTURE.md: scope note (covers the research subsystem, not the CLI). - Linked the orphaned repo_structure.md from the index. PRODUCT_SPEC.md was rewritten lean locally (1140 -> ~110 lines) and a VISION.md split off for the moonshot, but both stay LOCAL: PRODUCT_SPEC.md is gitignored by design (strategy doc), so VISION.md is gitignored to match and no public doc links to either. This push is the public docs realignment only. No code changed; 0 broken markdown links; no tracked cruft found.
…replicate (0.6.0 increment 3) The SOTA centerpiece: model per-GPU continuous batching so vLLM/TGI stop being modelled as replicas-of-single-stream. Aggregate decode throughput rises with batch B up to the KV-cache cap (increment 1), anchored to the measured/roofline single-stream rate so it stays quant-correct: aggregate(B) = B * bw*MBU / (weight_eff + B*kv_per_seq), weight_eff = bw*MBU/n1_tps - ThroughputModel.batched_decode_tps(): KV-amortized saturating curve, capped by the decode compute ceiling (GPU FP16 TFLOPS). - Engine: per backend, search (N replicas x B batch/GPU) for the cheapest config meeting the rate under the latency SLO; B <= max_concurrent_seqs; Ollama B=1. Higher B trades per-request TPOT for aggregate throughput (the real tradeoff). - LatencyModel.predict_p95 gains concurrent_per_agent, so capacity = N*B/service -- decoupling per-request latency from aggregate capacity. - BACKEND_CONTINUOUS_BATCHING (ollama no, vllm/tgi yes); Candidate.effective_batch. Verified: 7B @ 3 req/s on a 4090 -- Ollama needs 5 GPUs ($216/mo, single-stream); vLLM serves it on 1 GPU at batch=8 ($43.2/mo). 453 tests (+4), updated the Phase-0 replica tests to the Ollama path (batching now supersedes replicas for vLLM/TGI).
….0 increment 5) Analytical queueing is conservative for low-variance chat but under-estimates the tail for high-variance/agent traffic (heavy-tailed service holds slots), where it can approve fleets that fail SLOs (the cold-review finding). Generalize the M/D/1 wait to the two-moment (Allen-Cunneen) form (Ca^2+Cs^2)/2 x M/M/1; Cs^2=0 reproduces M/D/1 exactly (no change at the default), higher Cs^2 inflates p95. - LatencyModel.predict_p95: service_cv2 param (default 0 = M/D/1). - WORKLOAD_CV2 presets (steady 0 / chatbot 1 / bursty 4 / agent 8); plan --workload. - Above HIGH_VARIANCE_CV2 (4) every candidate carries a loud warning: analytical p95 under-estimates the tail, validate with a load test. Verified: agent vs steady on an 8B/4090 -- agent raises p95 (2903->3220ms), allocates more batch headroom, and warns. 459 tests (+6).
…u (0.6.0 increment 4) Expose the non-dominated trade-off frontier instead of a single cost-sorted pick (the research's "move beyond single-point estimates"). Now that batching sweeps the throughput<->latency curve, the frontier is the real menu: cheapest, lowest-latency, highest-quality, and the bends between. - engine.pareto_frontier(): non-dominated set on (cost down, p95 down, quality up), sorted by cost. - formatter.format_pareto() / format_pareto_json(): trade-off table tagging the cheapest / fastest / best-quality picks, plus JSON. - plan --pareto (works with --json too). Verified on an 8B/4090: 10 non-dominated configs from $43 vLLM (cheapest+fastest) to the quality/latency extremes. 464 tests (+5).
- bump 0.5.0 -> 0.6.0 (pyproject, __init__, CLAUDE, API, README) - CHANGELOG [0.6.0]: continuous batching, prefill/decode TTFT/TPOT, KV-cache max-batch, Pareto frontier, variance-aware queueing; linear replicas + cost-per-N - CLAUDE.md: 0.6.0 serving-model section; test count 476 - tests/test_accuracy.py: numerical falsifiability gates (throughput reproduces measured, roofline calibration anchor, VRAM formula, TTFT, batching invariants)
Sahil170595
changed the title
Blind multi-agent code audit (23 confirmed findings, 5 refuted). Each finding re-verified against source before fixing. HIGH: - VRAM activation memory is O(context), not O(context^2) -- a quadratic term blew up to ~130 GB at 32k for a 3B model, spuriously failing the VRAM gate and zeroing max_concurrent_seqs. Coefficient re-pinned to preserve the calibrated 2k value. - bench/refit/compare/report --json now emit valid JSON when piped (highlight=False, soft_wrap=True), matching the other six commands. MEDIUM: - quality_tier is family-aware (off-registry tier no longer collapses to unknown, so the concerning-drop advisory can fire) - guard ZeroDivisionError on a degenerate "...0b" identifier - refit --validate gates before the write (no longer persists invalid coefficients ahead of the failing exit) - per-key confidence weighting in refit (each entry blended by its own successful-run count, not the global total) - measure surfaces unknown --backend cleanly; check_model handles timeouts/HTTP errors; resolver/discovery raise ResolverError on a non-JSON 200 response (no raw traceback) - bench --context forwards --quant; non-Ollama context sweeps warn the per-request override was not applied - eval --fp16-baseline exposes tier classification - measured-corpus staleness warning on version mismatch - build floor setuptools>=77 (PEP 639 license form) LOW: - roofline bandwidth-correct above FP16 (FP32 ~ 0.5x); F16/F32 native quant normalized; engine spec=None no longer mislabels source as registry; strengthened tautological/weak tests; doc fix for classify_tier Deferred (low impact, noted in CHANGELOG): GB/GiB unit mix, ambiguous partial GPU-name match, BERTScore-0.0 sentinel. 483 tests pass; lint/format clean; wheel builds under isolation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the capacity planner reflect how LLM inference actually behaves — continuous batching, prefill/decode disaggregation, KV-cache-bound concurrency, goodput/Pareto, variance-aware queueing — instead of modelling every backend as replicas-of-single-stream. Direction validated by a cited deep-research pass over the serving literature.
Increments
max_concurrent_seqs) — the real concurrency limiter.TTFT(compute-bound, GPU FP16 TFLOPS) +TPOT(bandwidth-bound); end-to-end p95 now includes prefill.GPUSpec.fp16_tflops,--prompt-tokens.Candidate.effective_batch.plan --pareto) — the non-dominated cost/latency/quality menu, not one cost-sorted pick.plan --workload steady|chatbot|bursty|agent) — two-moment wait (Cs²=0 reproduces M/D/1); high-variance/agent traffic inflates the tail and warns. Closes the "analytical queueing silently approves broken fleets" gap.Also
cost_per_1m_tokno longer understated by instance count.Testing
Cs²=0reproduces the prior M/D/1 exactly.