From 72fd435b26384b36b15aef223c27a4a7d9d51ebf Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 25 May 2026 09:24:58 +0000 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20May=202026=20refresh=20=E2=80=94=20?= =?UTF-8?q?Red=20Hat=20eval,=20FP8=20nuance,=20community=20ports,=20drop?= =?UTF-8?q?=20QJL=20by=20default?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add 'Reality check (May 2026)' table to README citing the Red Hat AI / vLLM evaluation: FP8 KV is the default, *_nc variants drop QJL, skip first/last 2 layers, K/V norm ratio predicts quality. - New 'Tips and tricks' section in README distilling the May 2026 findings into 10 concrete recommendations. - README quick-start, integrations table, decision guide and limitations now reflect upstream vLLM merge, turboquant-plus-vllm, 0xSero Triton, varjoranta CUDA + MLX kernels, AmesianX llama.cpp (head_dim 256+), tonbistudio V3, spiritbuun llama.cpp, scos-lab benchmarks. - LANDSCAPE_2026.md: 'big May 2026 update' summary, TL;DR rewrites FP8 as the baseline, full third-party reproduction tables (MRCR, AIME25, throughput), refreshed engine + decision tables, expanded 'further reading' with the May evaluations. - FAQ.md: new questions ('Should I use FP8 or TurboQuant?', 'Which variant?', 'Does QJL help?', 'Which layers to skip?', 'How to choose K/V bits?'); updated KIVI/KVQuant/FP8 table; honest accuracy-vs-speed numbers; explainer hype check. - INTEGRATIONS.md: vLLM section reorganized around upstream-merged dtypes + production fork; SGLang shows the three competing PRs; llama.cpp split into spiritbuun (head_dim=128) vs AmesianX (any head_dim); MLX section moved to varjoranta Metal port; troubleshooting expanded with quality-regression checklist. - BENCHMARKS.md: Independent reproductions section with Red Hat numbers; format-comparison table now includes FP8 baseline + measured throughput penalties; methodology section updated with 2026 best practices; QJL variance explainer; layer sensitivity table. - IMPLEMENTATION_NOTES.md: QJL Score Weight section now explains why to default qjl_score_weight=0.0; new Layer Skipping section. - LAUNCH.md: archive notice flagging the April hype framing. - link-check workflow: exclude a few aggressively bot-blocking domains (towardsai, teqvolt, ai-intensify, allenkuo.medium). Co-Authored-By: Rob --- .github/workflows/link-check.yml | 4 + BENCHMARKS.md | 187 ++++++++++++++---- FAQ.md | 308 +++++++++++++++++++++++------- IMPLEMENTATION_NOTES.md | 58 +++++- INTEGRATIONS.md | 234 ++++++++++++++++------- LANDSCAPE_2026.md | 289 ++++++++++++++++++++++------ LAUNCH.md | 19 +- README.md | 318 +++++++++++++++++++++++++------ 8 files changed, 1125 insertions(+), 292 deletions(-) diff --git a/.github/workflows/link-check.yml b/.github/workflows/link-check.yml index 4c28e6e..5cfcbf1 100644 --- a/.github/workflows/link-check.yml +++ b/.github/workflows/link-check.yml @@ -35,5 +35,9 @@ jobs: --exclude 'medium\.com' --exclude 'ai\.plainenglish\.io' --exclude 'danilchenko\.dev' + --exclude 'pub\.towardsai\.net' + --exclude 'teqvolt\.com' + --exclude 'ai-intensify\.com' + --exclude 'allenkuo\.medium\.com' './**/*.md' fail: true diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 3af9733..9fd22e2 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -1,8 +1,15 @@ # TurboQuant Benchmarks -Detailed benchmark results, theoretical analysis, and memory calculations. +Detailed benchmark results, theoretical analysis, and memory calculations. Updated +**2026-05-25** with the [Red Hat AI / vLLM evaluation](https://blog.vllm.ai/2026/05/11/turboquant.html). -> See also: [README.md](README.md) for a high-level overview. +> See also: [README.md](README.md) for a high-level overview, and +> [FAQ.md](FAQ.md#whats-the-accuracy-vs-speed-trade-off) for the headline accuracy +> trade-off summary. + +> **TL;DR:** FP8 KV is the May 2026 default — it matches BF16 quality at zero throughput +> cost. TurboQuant earns its place when you need >2× compression. The numbers below +> include FP8 baselines so you can pick the right format for your scenario. --- @@ -45,17 +52,76 @@ End-to-end runs on a Blackwell consumer GPU, not synthetic — full report in ### Interpretation of the synthetic demo -- **3.5-bit mode**: 0.975 avg cosine ≈ paper expectation (near-zero LongBench loss; see - the Paper Results table at the top of [README.md](README.md)). +- **3.5-bit mode**: 0.975 avg cosine on this synthetic test. This used to be cited as + evidence the algorithm was lossless. **It is not** — cosine similarity on random + vectors does not predict end-to-end output quality (see Red Hat / Towards AI evaluations + below). Treat synthetic cosine as a sanity check, not a benchmark. - **2.5-bit mode**: 0.913 avg cosine, 4.41× compression (7.1× theoretical if you strip - norm overhead). Paper reports −0.62 LongBench pts at this setting — worth it when - memory is the bottleneck. + norm overhead). Real-world quality at this setting is meaningfully worse than the + paper's 49.44 LongBench — see independent reproductions below. - Historical "0.90 observed vs 0.95 expected" note referred to the S-matrix transpose bug described below — **that bug is fixed**; the current reference kernels give the numbers in the table. --- +## Independent reproductions (May 2026) + +The most important data point in the current ecosystem. From the +[Red Hat AI / vLLM blog post (May 11, 2026)](https://blog.vllm.ai/2026/05/11/turboquant.html): + +### Llama-3.3-70B-Instruct, MRCR long-context retrieval + +| KV format | KV reduction vs BF16 | MRCR-8 (avg) | MRCR-16 (avg) | +|---|---|---|---| +| BF16 | 1.0× | 1.000 | 1.000 | +| **FP8 (E4M3)** | **2.0×** | **0.997** | **0.991** | +| `turboquant_k8v4` | 2.0× | 0.989 | 0.973 | +| `turboquant_4bit_nc` | 2.6× | 0.984 | 0.957 | +| `turboquant_k3v4_nc` | 3.4× | 0.940 | 0.834 | +| `turboquant_3bit_nc` | 3.6× | 0.901 | 0.722 | + +### Qwen3-30B-A3B, AIME25 reasoning (Pass@1 over 32 samples) + +| KV format | KV reduction | AIME25 | +|---|---|---| +| BF16 | 1.0× | **0.683** | +| **FP8** | **2.0×** | **0.680** | +| `turboquant_k8v4` | 2.0× | 0.668 | +| `turboquant_4bit_nc` | 2.6× | 0.652 | +| `turboquant_k3v4_nc` | 3.4× | 0.532 | +| `turboquant_3bit_nc` | 3.5× | 0.443 | + +### Throughput on Qwen3-30B-A3B, single H100 + +| KV format | Tokens/s | vs BF16 | +|---|---|---| +| BF16 | 4,520 | 1.00× | +| **FP8** | **4,510** | **1.00×** (no penalty!) | +| `turboquant_4bit_nc` | 2,680 | 0.59× | +| `turboquant_3bit_nc` | 2,140 | 0.47× | + +**Why TurboQuant is slower than FP8 at decode:** FP8 quantizes both KV *storage* and +the attention *compute* on hardware FP8 tensor cores (Hopper/Blackwell). TurboQuant only +quantizes storage — attention compute still runs in BF16 after dequant. So you pay the +dequant overhead on every decode step without getting any compute speedup. The +[varjoranta CUDA kernels](https://github.com/varjoranta/turboquant-vllm) and +[0xSero Triton kernels](https://github.com/0xSero/turboquant) recover some of this gap +through fused dequant + GEMV, but not all of it. + +### Community-port replications + +- [scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8) + finds pure-MSE beats MSE+QJL on all 8 models tested (GPT-2-124M, Pythia-410M, OPT-1.3B, + Llama-3.1-8B, Mistral-7B, Phi-3.5, Qwen3-7B, Qwen3.6-35B-A3B). Establishes the K/V + norm ratio as the single best quality predictor. +- [tonbistudio V3 retrospective](https://github.com/tonbistudio/turboquant-pytorch) reports + +300% PPL on GPT-2 at b=3 with QJL on Keys vs +7.6% with MSE-only. +- [AI Intensify](https://ai-intensify.com/turboquant-is-compression-and-performance-worth-the-hype/) + reaches the same conclusions as Red Hat across Llama-3-70B, Qwen3-30B, MiniMax-M2.7. + +--- + ## Theoretical Bounds (from the paper) For a unit-norm vector x ∈ ℝ^d, quantized to b bits per coordinate: @@ -105,19 +171,22 @@ This constant factor gap is tight — it's inherent to scalar quantization of hi ### Compression & Quality -| Format | Bits/value | Bytes/vector | Compression | Quality | Unbiased IP? | -|--------|-----------|-------------|-------------|---------|-------------| -| FP16 | 16.00 | 256 | 1.0× | Baseline | — | -| BF16 | 16.00 | 256 | 1.0× | ~99.9% | — | -| FP8 (E4M3) | 8.00 | 128 | 2.0× | ~99.5% | — | -| INT8 | 8.25 | 106 | 2.4× | ~99% | — | -| INT4 | 4.25 | 54 | 4.7× | ~97% | No | -| KIVI-2bit | 2.25 | 29 | 8.8× | ~95% | No | -| **TurboQuant (3-bit)** | **3.25** | **52** | **4.9×** | **~99%** | **Yes** | -| TurboQuant (4-bit) | 4.25 | 68 | 3.8× | ~99.5% | Yes | -| TurboQuant (2-bit) | 2.25 | 36 | 7.1× | ~95% | Yes | +Quality column shows **end-to-end output quality** (Red Hat / community measurements on +Llama-3.3-70B), not synthetic cosine similarity. "Throughput vs BF16" reflects measured +H100 decode throughput on Qwen3-30B-A3B. + +| Format | Bits/value | Bytes/vector | Compression | Quality | Throughput vs BF16 | Notes | +|---|---|---|---|---|---|---| +| FP16 / BF16 | 16.00 | 256 | 1.0× | Baseline | 1.00× | — | +| **FP8 (E4M3)** | **8.00** | **128** | **2.0×** | **No loss** | **1.00×** | **The May 2026 default** | +| INT8 | 8.25 | 106 | 2.4× | ~99% | ~0.80× | Software dequant | +| `turboquant_4bit_nc` | 4.25 | 68 | 3.0× | Within 1–2 pt of FP8 | ~0.59× | Recommended TQ config | +| INT4 | 4.25 | 54 | 4.7× | ~97% | ~0.55× | Storage-only | +| **TurboQuant 3-bit (paper)** | **3.25** | **52** | **4.9×** | **Synthetic 99%, real 70–85% on reasoning** | **~0.47×** | QJL on; **don't ship this** | +| `turboquant_3bit_nc` | 3.25 | 52 | 4.9× | 15–25 pt drop on AIME25 / MRCR-16 | ~0.47× | QJL off; edge only | +| KIVI-2bit | 2.25 | 29 | 8.8× | ~95% on short context, less on long | ~0.40× | Predecessor | -> Bits/value includes norm overhead: 16 bits (PolarQuant norm) + 16 bits (QJL residual norm) = 32 bits per vector. For d=128: +0.25 bits/value. +> Bits/value for TQ includes norm overhead: 16 bits (PolarQuant norm) + 16 bits (QJL residual norm) = 32 bits per vector. For d=128: +0.25 bits/value. The `*_nc` variants drop the QJL residual norm and use a single 16-bit per-token norm rescale instead. ### Memory for Llama-3-8B-Instruct (128K context) @@ -283,21 +352,36 @@ For a single query over seq_len = 8,192 tokens (Llama-3-8B, 8 KV heads, d=128): ### What the Paper Measures -The paper reports **accuracy on LongBench** (a suite of long-context benchmarks) with actual LLM inference. This is the gold standard — cosine similarity on random vectors is a proxy metric. +The paper reports **accuracy on LongBench** (a suite of long-context benchmarks) with +actual LLM inference on Llama-3.1-8B-Instruct. -- At 3.5 bits: **zero accuracy loss** on Llama-3.1-8B-Instruct +- At 3.5 bits: **zero accuracy loss** on Llama-3.1-8B-Instruct LongBench Avg - At 2.5 bits: **marginal degradation** - On H100: 4-bit TurboQuant achieves **8× performance** over 32-bit for attention logits +**These results do not generalize cleanly to frontier models.** The May 2026 Red Hat +evaluation — the first rigorous independent reproduction — found 15–25 pt drops on +Llama-3.3-70B / Qwen3-30B-A3B reasoning evals at 3-bit precision. The community ports +independently identified the QJL stage as the culprit; see the "Independent +reproductions" section above. + ### Reproducing Paper Results -To reproduce the paper's claims, you need: -1. A real model (Llama-3-8B-Instruct or similar) -2. TurboQuant integrated into the inference pipeline (vLLM plugin) +To reproduce the paper's claims (Llama-3.1-8B at 3.5-bit ~ FP16 LongBench), you need: + +1. Llama-3.1-8B-Instruct +2. TurboQuant integrated into the inference pipeline (vLLM upstream or our plugin) 3. LongBench evaluation suite 4. Hardware with sufficient memory (A100/H100 recommended) -Those end-to-end evaluations are still pending. +For the **2026 evaluation methodology** (the right way to evaluate today): + +1. Start with FP8 KV as your baseline. +2. Run long-context retrieval ([MRCR-8 and MRCR-16](https://github.com/openai/simple-evals/blob/main/mrcr_eval.py)) + and reasoning (AIME25 Pass@1, LiveCodeBench-v6) at your target context length. +3. Try `turboquant_4bit_nc`, not `3bit_nc` or `*_q`. +4. Skip first/last 2 layers from compression. +5. Profile your model's K vs V norm ratio; use `k3v4_nc` if K dominates. --- @@ -315,21 +399,54 @@ produced ~0.90 cosine similarity instead of the paper's ~0.95. current master shows **0.975 avg cosine** at 3.5-bit mode (see table at the top of this file). Any fused Triton/CUDA kernel should use the same projection. -### Single-Sample QJL Variance (By Design) +### Single-Sample QJL Variance — the real story -The QJL correction is applied from a single random projection sample. The variance of this estimate is: +The QJL correction is applied from a single random projection sample. The variance of +this estimate is: ``` Var(⟨y, r̂⟩) ≤ (π / 2d) · ‖y‖² · ‖r‖² ``` -For d=128: Var ≤ 0.012 · ‖y‖² · ‖r‖². With ‖r‖ ≈ 0.3 (typical residual after 2-bit PQ), this is Var ≈ 0.001 · ‖y‖². +For d=128: Var ≤ 0.012 · ‖y‖² · ‖r‖². With ‖r‖ ≈ 0.3 (typical residual after 2-bit PQ), +this is Var ≈ 0.001 · ‖y‖². + +**This variance is what kills the QJL stage at b ≤ 3.** Softmax × exp() amplifies +variance multiplicatively, and softmax over a long context concentrates that +amplification on a small number of highly-attended tokens. The bias-variance trade-off +doesn't favor QJL until b ≈ 5 — at which point you might as well use FP8. + +**Practical implication:** set `qjl_score_weight=0.0` for pure MSE (equivalent to the +`*_nc` family). For the rationale see +[FAQ "Does the QJL step actually help?"](FAQ.md#does-the-qjl-step-actually-help) and +[LANDSCAPE_2026.md "Independent reproductions"](LANDSCAPE_2026.md#independent-reproductions-may-2026). + +### Layer sensitivity — confirmed May 2026 + +The Red Hat evaluation confirmed what scos-lab originally proposed: **the first and +last 2 layers carry disproportionate signal.** Skipping them from quantization +preserves most of the quality at near-identical compression ratio. + +Layer outlier-channel statistics from [scos-lab](https://github.com/scos-lab/turboquant): + +| Layer position | % outlier channels (RMS > 3× median) | +|---|---| +| Layer 0 | ~20% | +| Layers 1–2 | ~12% | +| Middle layers | 4–6% | +| Last 2 layers | ~10–15% | -The `qjl_score_weight=0.5` parameter trades bias for variance reduction. Setting it to 1.0 gives the unbiased estimator but higher variance per-token. +A fixed top-k=32 outlier allocation wastes bits in middle layers and under-allocates +layer 0. The mixed-precision config in this repo defaults to a fixed top-k for backward +compatibility — set `mode="dynamic"` on `MixedPrecisionConfig` to use the scos-lab +threshold instead. ### Attention Sinks (Not Yet Implemented) -The first few tokens in a sequence ("attention sinks") tend to receive disproportionately high attention weights. Their KV cache vectors should ideally be preserved in higher precision. This is planned but not yet implemented. +The first few tokens in a sequence ("attention sinks") tend to receive disproportionately +high attention weights. Their KV cache vectors should ideally be preserved in higher +precision. This is what "skip first 2 layers" approximates at the layer granularity; +proper per-token attention-sink handling is planned but not yet implemented. --- @@ -337,11 +454,13 @@ The first few tokens in a sequence ("attention sinks") tend to receive dispropor ### Planned Tests -- [ ] **Llama-3-8B on LongBench** — reproduce paper's accuracy claims -- [ ] **Needle-in-haystack** — test retrieval at 128K context with TQ compression -- [ ] **Triton vs PyTorch** — kernel speedup comparison -- [ ] **RTX 5090 throughput** — real hardware benchmarks on consumer GPU -- [ ] **H100 throughput** — datacenter GPU benchmarks +- [ ] **Llama-3.1-8B on LongBench** — reproduce paper's headline 3.5-bit / 2.5-bit numbers + with our reference implementation +- [ ] **MRCR-16 on Llama-3.3-70B** — reproduce Red Hat's per-variant scores +- [ ] **AIME25 / LiveCodeBench-v6 on Qwen3-30B-A3B** — reproduce Red Hat's reasoning numbers +- [ ] **First/last-2-layer skip ablation** — measure exact quality vs compression trade-off +- [ ] **K/V norm-ratio profiler** — standardize the scos-lab metric across models +- [ ] **Production-fork comparison** — our reference vs varjoranta CUDA vs 0xSero Triton on H100 / RTX 5090 - [ ] **Mixed precision** — test PQ=3-bit for first N layers, 2-bit for rest - [ ] **GQA scaling** — test with different GQA ratios (1:1, 2:1, 4:1, 8:1) - [ ] **RoPE interaction** — pre-RoPE vs post-RoPE quality comparison diff --git a/FAQ.md b/FAQ.md index 4a79b04..f531502 100644 --- a/FAQ.md +++ b/FAQ.md @@ -1,11 +1,119 @@ # TurboQuant FAQ -> Common questions and misconceptions, updated **2026-04-17**. For the broader 2026 +> Common questions and misconceptions, updated **2026-05-25**. For the broader 2026 > landscape, see [LANDSCAPE_2026.md](LANDSCAPE_2026.md). For concrete integrations, see > [INTEGRATIONS.md](INTEGRATIONS.md). --- +## "Should I use FP8 or TurboQuant?" + +**Default to FP8.** The [Red Hat AI / vLLM evaluation (May 11, 2026)](https://blog.vllm.ai/2026/05/11/turboquant.html) +is rigorous and conclusive on this question: across Llama-3.3-70B, Qwen3-30B-A3B, and +MiniMax-M2.7, on both long-context retrieval (MRCR) and reasoning (AIME25, GPQA, MATH500, +LiveCodeBench-v6), **FP8 (E4M3) KV cache** delivers: + +- **No measurable accuracy loss** vs BF16 +- **No throughput penalty** (4,510 t/s vs 4,520 t/s BF16 on Qwen3-30B-A3B / H100) +- **2× memory savings** + +**Use TurboQuant only when:** + +1. **2× isn't enough.** You need to fit a 70B+ model + cache on a single GPU at 128K+ + context, or you're memory-bound below the FP8 floor. +2. **Your hardware doesn't have FP8 attention.** On RTX 4090/4080 (Ada), older AMD, + Apple Silicon, and most edge accelerators, FP8 KV doesn't give you the throughput + bonus because the attention kernel still runs in BF16. TurboQuant's bandwidth savings + are still useful there. + +If you do reach for TurboQuant, use `turboquant_4bit_nc` — see the next question. + +--- + +## "Which TurboQuant variant should I pick?" + +``` +vllm serve --kv-cache-dtype turboquant_4bit_nc +``` + +**`turboquant_4bit_nc`** is the production-grade variant. It gives ~2.6–3.1× +compression (1.5× more than FP8) with quality within ~1 pt of FP8 on long-context evals. +The `_nc` suffix means "norm correction" — it drops the QJL residual in favor of a +simpler per-token norm rescale, which the community independently found to be uniformly +better at low bit widths. + +The full ladder: + +| Variant | KV reduction vs BF16 | Quality | When to use | +|---|---|---|---| +| `fp8` (not TQ, baseline) | 2.0× | Matches BF16 | **Always try first** | +| `turboquant_k8v4` | 2.0× | Slight drop vs FP8 | Rarely the right pick — FP8 wins | +| **`turboquant_4bit_nc`** | **2.6–3.1×** | Within 1–2 pt of FP8 | **The recommended TQ config** | +| `turboquant_k3v4_nc` | 3.4× | 10–15 pt drop on reasoning | Qwen-class with extreme K/V norm asymmetry | +| `turboquant_3bit_nc` | 3.5–4.0× | 15–25 pt drop on reasoning | Edge / strict memory budget only | + +--- + +## "Does the QJL step actually help?" + +**Usually not at b ≤ 3.** This is the most surprising finding from the May 2026 +ecosystem. The paper's `TurboQuantProd` algorithm appends a 1-bit QJL (Quantized +Johnson-Lindenstrauss) sign residual to the MSE quantizer for an unbiased inner-product +estimator. In theory it's strictly better than MSE-only. In practice, every independent +reproduction has found the opposite at the bit widths people actually use: + +- [tonbistudio V3 README](https://github.com/tonbistudio/turboquant-pytorch): "the + paper's TurboQuantProd (QJL) for Keys gives +300% PPL at b=3 on GPT-2. MSE for both + K and V gives only +7.6%." +- [scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8): + pure MSE beats MSE+QJL on every model tested. +- [Red Hat AI](https://blog.vllm.ai/2026/05/11/turboquant.html): the `*_nc` (no-QJL) + variants strictly Pareto-dominate the QJL-augmented variants. + +**Why?** The QJL residual is *unbiased* but high-variance. Softmax in the attention +computation exponentially amplifies that variance, so the variance-bias trade-off doesn't +favor adding QJL at low bit widths. For Hadamard MSE quantizers at b ≥ 5 the paper's +analysis still holds — it's just that nobody is using b ≥ 5 because at that point you +might as well use FP8. + +**What this repo defaults to:** the paper's algorithm (QJL on). Set `qjl_score_weight=0.0` +to switch to pure MSE — or use the `*_nc` variants in the vLLM `--kv-cache-dtype` flag. + +--- + +## "Which layers should I skip from compression?" + +The first and last 2 layers. The Red Hat evaluation found that these layers carry +disproportionate signal — quantizing them costs more accuracy than the bits they save. +vLLM does this automatically for the `*_nc` variants. If you're rolling your own +attention backend: + +```python +SKIP = set(list(range(2)) + list(range(num_layers - 2, num_layers))) +# Keep FP16/FP8 KV for layers in SKIP; use TurboQuant for the rest. +``` + +--- + +## "How do I choose K vs V bits for my model?" + +**Profile the per-channel K vs V norm ratio.** This single statistic predicts compression +quality better than any other metric the community has identified. + +- Llama-class models: K/V norm ratio is roughly symmetric (1–3×). A uniform `4bit_nc` + config works well. +- Qwen-class models: K/V norm ratio is **massively asymmetric** — Qwen3.5-7B sees a 106× + ratio, Qwen3.6-35B-A3B sees 182×. K dominates the norm budget. Use `k8v4` or + `k3v4_nc`. +- DeepSeek-V3 (MLA): the latent is small enough that K and V are roughly comparable — + treat similarly to Llama-class. + +To profile: run [`scos-lab/turboquant`](https://github.com/scos-lab/turboquant) on a few +batches of representative inputs and inspect the per-head K/V norm histograms. The K +budget should roughly match the K/V norm ratio in entropy bits. + +--- + ## "Is this a replacement for AWQ / GPTQ / GGUF?" **No.** TurboQuant compresses the **KV cache** at inference time. It does not touch @@ -66,20 +174,26 @@ layer/head at model load, not per-request. ## "How does this differ from KIVI / KVQuant / FP8 KV cache?" -| Property | FP8 KV | KIVI (2-bit) | KVQuant | **TurboQuant 3.5-bit** | -|---|---|---|---|---| -| Bits/value | 8 | 2.25 | ~3.5 | **3.5** | -| Compression vs FP16 | 2× | 7.1× | ~4.5× | **4.9×** | -| LongBench Avg (Llama-3.1-8B) | 50.0 | 48.50 | ~49.5 | **50.06** | -| Requires calibration | No | No | Yes | **No** | -| Unbiased attention scores | No | No | No | **Yes** (QJL residual) | -| Outlier handling | Per-tensor scale | Per-channel K | Dense-sparse split | **Two-rotation split** | -| Works on RoPE'd K | Yes | Per-channel helps here | Pre-RoPE preferred | Yes, post-RoPE | - -The **unbiased inner product estimator** (the 1-bit QJL residual stage) is what makes -TurboQuant's attention scores mathematically unbiased, not just MSE-minimizing. This is -why it preserves needle-in-haystack retrieval at 128K context where the other methods -start to miss specific tokens. +| Property | FP8 KV (E4M3) | KIVI (2-bit) | KVQuant | TurboQuant `4bit_nc` | TurboQuant `3bit_nc` | +|---|---|---|---|---|---| +| Bits/value | 8 | 2.25 | ~3.5 | 4.0 | 3.0 | +| Compression vs BF16 | **2.0×** | 7.1× | ~4.5× | 2.6–3.1× | 3.5–4.0× | +| MRCR-8 needle (Llama-3.3-70B) | **0.997** | not reported | not reported | 0.984 | 0.901 | +| AIME25 (Qwen3-30B-A3B) | **0.680** | — | — | 0.652 | 0.443 | +| Throughput penalty | **None** | High | Moderate | ~40% | ~50% | +| Requires calibration | No | No | Yes | **No** | **No** | +| Quantizes attention compute | **Yes (FP8 cores)** | No | No | No | No | +| Works on RoPE'd K | Yes | Per-channel helps | Pre-RoPE preferred | Yes | Yes | +| Best at | Default for most use cases | Older deployments | Sub-4-bit with calibration | Memory-bound long context | Edge / on-device | + +Sources: [Red Hat AI eval (May 11, 2026)](https://blog.vllm.ai/2026/05/11/turboquant.html), +[TurboQuant paper](https://arxiv.org/abs/2504.19874). + +**Why doesn't TurboQuant beat FP8 at the same quality?** FP8 quantizes both the KV +storage *and* the attention compute on hardware FP8 tensor cores (Hopper/Blackwell). +TurboQuant only quantizes storage — it dequantizes back to BF16 for the actual softmax +× value-projection compute. So FP8 wins on throughput *and* matches quality at 2×; TQ +only catches up in scenarios that demand more than 2× compression. --- @@ -152,19 +266,30 @@ For now, if you're running beyond 64K context, stack TurboQuant with ## "What's the accuracy vs speed trade-off?" -On current hardware (Apr 2026): +On Hopper/Blackwell hardware (May 2026, [Red Hat measurements](https://blog.vllm.ai/2026/05/11/turboquant.html) +on Qwen3-30B-A3B / H100): -| Metric | FP16 baseline | TurboQuant 3.5-bit | TurboQuant 2.5-bit | -|---|---|---|---| -| LongBench Avg (Llama-3.1-8B) | 50.06 | **50.06** | 49.44 | -| Needle @ 128K | 0.997 | **0.997** | 0.997 | -| KV memory | 1.0× | **0.20×** | 0.14× | -| H100 attention throughput (estimated, [BENCHMARKS.md](BENCHMARKS.md#attention-throughput-tokensssec)) | 1.0× | **4.4×** | 4.6× | -| RTX 5090 attention throughput | 1.0× | **4.3×** | 4.5× | -| On-paper distortion gap vs info-theoretic optimal | — | **2.7×** | 2.7× | - -The throughput win comes from **memory bandwidth**: the attention kernel reads 52 bytes -per KV vector instead of 256. +| Metric | BF16 | **FP8 KV** | TQ `4bit_nc` | TQ `3bit_nc` | +|---|---|---|---|---| +| AIME25 | 0.683 | **0.680** | 0.652 | 0.443 | +| MRCR-8 (Llama-3.3-70B) | 1.000 | **0.997** | 0.984 | 0.901 | +| KV memory | 1.0× | **0.50×** | 0.32–0.38× | 0.25–0.29× | +| Tokens/s (decode) | 4,520 | **4,510** | 2,680 | 2,140 | +| Throughput penalty | 0% | **0%** | ~40% | ~50% | + +**TurboQuant's throughput penalty** comes from a structural mismatch: FP8 quantizes both +KV *storage* and attention *compute* on hardware FP8 tensor cores. TurboQuant only +quantizes storage — the attention compute still runs in BF16 after dequantization. So +you pay the dequant cost on every decode step. + +The paper-promised "memory bandwidth win" only materializes when you actually need +>2× compression (and therefore are memory-bandwidth bound, not compute bound). For most +serving setups, **FP8 wins on throughput, accuracy, and simplicity simultaneously**. + +The CUDA-fused dequant kernels in [`turboquant-plus-vllm`](https://github.com/varjoranta/turboquant-vllm) +recover some of this gap (the v0.13.0 release reports 10.1× decode speedup on +Qwen3.6-35B-A3B over the reference path), but FP8 remains faster in the regime where +both fit. --- @@ -183,43 +308,73 @@ The crossover point where TurboQuant starts earning its keep: --- -## "How does TurboQuant relate to the viral 'TurboQuant is the breakthrough of the year' takes on X?" +## "Is TurboQuant the breakthrough of the year, like X/Twitter said in April?" + +Short answer: **The April 2026 hype overshot. The May 2026 picture is more nuanced.** -The quote is from the -[LMCache blog post (Apr 15, 2026)](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/), -paraphrasing discussion on X/Twitter. Our read: +In mid-April, the [LMCache blog post](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) +paraphrased X/Twitter chatter calling TurboQuant "the most significant AI breakthrough +this year." That framing made sense given the paper's claims — data-oblivious, provably +near-optimal, 3.5 bpv at FP16 quality. -- **The hype is largely earned.** Data-oblivious, provably near-optimal, no calibration, - 3.5 bpv at FP16 quality is genuinely new. -- **It's not the only 2026 breakthrough.** TriAttention (Apr 6), LRKV (Apr 9), Adaptive - KV-Quant (Apr 6), and NVFP4 KV (Apr 2) are all major shifts. TurboQuant is the - **precision** story; the others are **selection**, **architecture**, and **hardware** - stories. -- **The best 2026 stack combines them.** See - [LANDSCAPE_2026.md §Decision guide](LANDSCAPE_2026.md#decision-guide-which-compressor-do-i-actually-want). +Then the [Red Hat AI evaluation in May](https://blog.vllm.ai/2026/05/11/turboquant.html) +ran the actual benchmarks and reported what independent ports had been finding for weeks: + +1. **FP8 KV is a better default** than any TurboQuant config the team tested. It's free + on Hopper/Blackwell. +2. **The 3-bit modes the paper headline-tests don't generalize cleanly** to frontier + models. AIME25 drops 24 pts on Qwen3-30B-A3B with `turboquant_3bit_nc`. +3. **The QJL "residual" step the paper considers essential** actually hurts in practice. + The `*_nc` variants drop QJL and outperform. + +**Where TurboQuant is genuinely a contribution:** + +- A clean, calibration-free quantization algorithm that *works* (especially `4bit_nc`). +- A useful tool for fitting models on memory-constrained hardware where 2× FP8 isn't + enough. +- A mathematical framework (random rotation + scalar VQ) that's likely to influence + future work even if the QJL step doesn't survive. + +**Where the hype overshot:** + +- It is not a universal replacement for existing KV quantization. +- The "3.5 bits at zero accuracy loss" claim doesn't hold up on frontier MoE models. +- It's a precision tool, not the precision tool — and FP8 wins the default slot. + +The 2026 KV-compression landscape has several major shifts of comparable importance: +TriAttention (token selection, Apr 6), LRKV (architectural, Apr 9), Adaptive KV-Quant +(per-token bits, Apr 6), NVFP4 KV (hardware, Apr 2). The best 2026 stack combines them +along orthogonal axes — see [LANDSCAPE_2026.md §Decision guide](LANDSCAPE_2026.md#decision-guide-which-compressor-do-i-actually-want). --- ## "What's the right way to benchmark my model?" -Use the paper's methodology — **LongBench + Needle-In-A-Haystack** at your target -context length. Cosine similarity on random vectors (what `src/demo.py` reports) is a -useful sanity check but not a substitute for end-to-end eval. +The Red Hat evaluation uses the right methodology. Don't rely on cosine similarity +metrics on random vectors — they are useful sanity checks but **do not predict +end-to-end output quality**. The tonbistudio team published a retracted +"18/18 perfect generation at 5× compression" claim caused by a `residual_window=0` bug +that silently disabled compression; always log the *actual* compressed token count after +the run. Concrete steps: -1. Pick your target engine (vLLM, SGLang, llama.cpp, KVPress) — see - [INTEGRATIONS.md](INTEGRATIONS.md). -2. Run [LongBench](https://github.com/THUDM/LongBench) with FP16 KV to establish a - baseline. -3. Switch to `turboquant_3bit` (or our plugin's default config) and re-run. -4. Expected delta on Llama-3.1-8B: **< 0.1 pts** on LongBench Avg, zero change on needle. -5. If you see larger deltas, check: - - Is your outlier-channel detection actually being applied? (It's only applied when - `MixedPrecisionConfig.enable=True`.) - - Is `qjl_score_weight=1.0`? (Lower values trade bias for variance.) - - Is your head dim a power of 2? (Hadamard rotation pads if not; pad token can skew - norms.) +1. Pick your target engine — see [INTEGRATIONS.md](INTEGRATIONS.md). +2. Establish a baseline with **FP16/BF16 KV** *and* **FP8 KV** at your target context + length. (You need both — FP8 is what TQ has to beat, not just FP16.) +3. Run two real evals at minimum: + - **Long-context retrieval:** [MRCR](https://github.com/openai/simple-evals/blob/main/mrcr_eval.py) (multi-round needle) at 32K, 64K, 128K. The Red Hat blog uses MRCR-8 and MRCR-16. + - **Reasoning:** AIME25 or LiveCodeBench-v6 Pass@1 over 32+ samples (these surface + reasoning degradation that retrieval evals miss). +4. Switch to `turboquant_4bit_nc` (start here, not 3-bit) and re-run. +5. Expected delta: **< 1 pt** vs FP8 on Llama-3-class, **5–15 pt** on Qwen-class without + asymmetric bit allocation. If you see worse, check: + - Are first/last 2 layers being skipped from compression? + - Are you using the `*_nc` (no-QJL) variant? + - Is your model in the head_dim=128 family? If head_dim ≥ 256, use [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant), not llama.cpp's spiritbuun fork. + - Is `qjl_score_weight=0.0`? (Or for vLLM, are you on a `*_nc` dtype?) + - Is your head dim a power of 2? (Hadamard rotation pads if not; pad tokens skew norms.) +6. If quality is still off and FP8 works fine — use FP8. --- @@ -240,19 +395,38 @@ If you want to credit this open-source port specifically, you can link to the re --- -## "Where do I find the viral takes / explainers?" +## "Where do I find the most up-to-date evaluations and explainers?" + +Start here (sorted by depth): -Start here (sorted by how layperson-friendly they are): +**Definitive third-party evaluations:** -1. **[LMCache Blog (Apr 15, 2026)](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/)** - — "in laymen's term" explainer, no math. Best starting point. -2. **[Towards AI: Running a 35B Model Locally with TurboQuant (Apr 15, 2026)](https://pub.towardsai.net/running-a-35b-model-locally-with-turboquant-whats-actually-possible-right-now-1ac5327430b0)** - — hands-on consumer-GPU walkthrough. -3. **[MarkTechPost: NVIDIA KVPress end-to-end guide (Apr 9, 2026)](https://www.marktechpost.com/2026/04/09/an-end-to-end-coding-guide-to-nvidia-kvpress-for-long-context-llm-inference-kv-cache-compression-and-memory-efficient-generation/)** +1. **[Red Hat AI / vLLM: "A First Comprehensive Study of TurboQuant" (May 11, 2026)](https://blog.vllm.ai/2026/05/11/turboquant.html)** + — the rigorous evaluation. Read this before anything else. +2. **[scos-lab/turboquant 8-model benchmark (May 16, 2026)](https://github.com/tonbistudio/turboquant-pytorch/issues/8)** + — establishes the K/V-norm-ratio predictor. + +**Layperson / explainer:** + +3. **[LMCache Blog (Apr 15, 2026)](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/)** + — "in laymen's terms" explainer of the algorithm. The framing is more enthusiastic + than the May 2026 data supports, but it's still the best layperson intro. +4. **[TeqVolt deep-dive (May 14, 2026)](https://teqvolt.com/deep-dives/turboquant-google-6x-kv-cache-compression-3-bit-zero-loss)** + — algorithm walkthrough with up-to-date caveats. +5. **[AI Intensify hype check (May 17, 2026)](https://ai-intensify.com/turboquant-is-compression-and-performance-worth-the-hype/)** + — three-model comparison against FP8. + +**Hands-on:** + +6. **[Towards AI: Running a 35B Model Locally with TurboQuant (Apr 15, 2026)](https://pub.towardsai.net/running-a-35b-model-locally-with-turboquant-whats-actually-possible-right-now-1ac5327430b0)** + — consumer-GPU walkthrough. +7. **[MarkTechPost: NVIDIA KVPress end-to-end guide (Apr 9, 2026)](https://www.marktechpost.com/2026/04/09/an-end-to-end-coding-guide-to-nvidia-kvpress-for-long-context-llm-inference-kv-cache-compression-and-memory-efficient-generation/)** — how to stack KVPress eviction under a TurboQuant-style precision backend. -4. **[ArXivIQ FlashAttention-4 analysis (Mar 13, 2026)](https://arxiviq.substack.com/p/flashattention-4-algorithm-and-kernel)** - — FA4 paper summary relevant to Blackwell quantized KV. -5. **[The original paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)** - — 14 pages, math-heavy but worth it. -6. **[ICLR 2026 poster (Apr 25, 2026 PDT)](https://iclr.cc/virtual/2026/poster/10006985)** - — meet the authors. + +**Primary sources:** + +8. **[The original paper (arXiv 2504.19874)](https://arxiv.org/abs/2504.19874)** + — 14 pages, math-heavy. +9. **[Google Research blog (Mar 24, 2026)](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/)** + — author-side framing. +10. **[ICLR 2026 poster](https://iclr.cc/virtual/2026/poster/10006985)** — presented Apr 25, 2026. diff --git a/IMPLEMENTATION_NOTES.md b/IMPLEMENTATION_NOTES.md index be35060..85cca6e 100644 --- a/IMPLEMENTATION_NOTES.md +++ b/IMPLEMENTATION_NOTES.md @@ -80,7 +80,63 @@ This avoids the approximation of using a single rotation with post-rotation vari ## QJL Score Weight -The `compute_attention()` method defaults to `qjl_score_weight=1.0`, which produces the paper-correct **unbiased** inner product estimator (Theorem 2). Setting `qjl_score_weight < 1.0` trades bias for lower variance — this is a practical heuristic not present in the paper that can improve attention quality when the QJL variance is high relative to score differences. +The `compute_attention()` method defaults to `qjl_score_weight=1.0`, which produces the +paper-correct **unbiased** inner product estimator (Theorem 2). Setting +`qjl_score_weight < 1.0` trades bias for lower variance. + +**May 2026 community consensus: set `qjl_score_weight=0.0` (i.e. disable QJL) at b ≤ 3.** + +Multiple independent reproductions show that the QJL residual stage hurts more than it +helps at the bit widths people actually use: + +- [Red Hat AI evaluation](https://blog.vllm.ai/2026/05/11/turboquant.html): the `*_nc` + ("norm correction", QJL-free) variants strictly Pareto-dominate the QJL-augmented + variants on Llama-3.3-70B / Qwen3-30B-A3B / MiniMax-M2.7. +- [tonbistudio V3 PyTorch port](https://github.com/tonbistudio/turboquant-pytorch): + +300% PPL on GPT-2 at b=3 with QJL on Keys, vs +7.6% with pure MSE. V3 ships pure-MSE + as the default. +- [scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8): + pure MSE beats MSE+QJL on every one of 8 models tested. + +**Why?** The QJL estimator is unbiased but high-variance. Softmax × exp() amplifies +variance multiplicatively, and softmax over long context concentrates that amplification +on a small number of highly-attended tokens, breaking attention. The bias-variance +trade-off doesn't favor QJL until b ≈ 5 — at which point you might as well use FP8 KV. + +This repo's reference path keeps QJL on for backward compatibility with the paper's +algorithm; production paths should turn it off. The vLLM upstream `*_nc` dtypes +(`turboquant_4bit_nc`, `turboquant_3bit_nc`, `turboquant_k3v4_nc`) bake this in. + +## Layer Skipping (Red Hat / scos-lab) + +The first and last 2 layers of a transformer carry disproportionately important KV +representations. The Red Hat evaluation showed that quantizing them costs more accuracy +than the bits they save, so the production-grade `*_nc` paths skip them. + +This repo's reference implementation does not implement per-layer skipping — it's a +plumbing concern at the engine integration layer (vLLM, our plugin, etc.). If you're +building an integration, replicate this: + +```python +SKIP_LAYERS = set(list(range(2)) + list(range(num_layers - 2, num_layers))) +for layer_idx in range(num_layers): + if layer_idx in SKIP_LAYERS: + # Keep BF16 / FP8 KV for this layer + continue + # Apply TurboQuant compression for the rest +``` + +Per-layer outlier-channel statistics (from +[scos-lab](https://github.com/scos-lab/turboquant)): + +- Layer 0: ~20% outliers +- Layers 1–2: ~12% +- Middle layers: 4–6% +- Last 2 layers: ~10–15% + +A fixed `top_k=32` outlier allocation wastes bits in the middle and under-allocates +layer 0. Use `MixedPrecisionConfig(mode="dynamic", threshold_factor=3.0)` to enable +the per-layer dynamic threshold instead. ## Backward Compatibility diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md index 2023747..49f8fff 100644 --- a/INTEGRATIONS.md +++ b/INTEGRATIONS.md @@ -1,19 +1,24 @@ # TurboQuant Integrations > Concrete, copy-pasteable setup for the serving engines and frameworks TurboQuant plugs -> into. Status and PR links as of **2026-04-17**. See +> into. Status and PR links as of **2026-05-25**. See > [LANDSCAPE_2026.md](LANDSCAPE_2026.md) for the broader ecosystem survey. +> +> **Before you reach for TurboQuant, try `--kv-cache-dtype fp8` first.** The May 2026 +> [Red Hat AI evaluation](https://blog.vllm.ai/2026/05/11/turboquant.html) showed FP8 KV +> matches BF16 quality on every benchmark with no throughput penalty. Only choose TurboQuant +> when you need more than 2× compression. See [FAQ "Should I use FP8 or TurboQuant?"](FAQ.md#should-i-use-fp8-or-turboquant). --- ## Quick navigation -- [vLLM](#vllm) — two paths: upstream PRs and our bundled backend plugin -- [SGLang](#sglang) — PR #21419 adds `--kv-cache-dtype turboquant` -- [llama.cpp](#llamacpp) — quantized-KV DP4A flash attention as of b8779 +- [vLLM](#vllm) — **merged upstream**; production fork available +- [SGLang](#sglang) — WIP, multiple competing PRs +- [llama.cpp](#llamacpp) — community forks for both head_dim=128 and head_dim=256 - [NVIDIA KVPress](#nvidia-kvpress) — stack TurboQuant precision under KVPress eviction - [LMCache](#lmcache) — distributed / cross-session KV reuse on TurboQuant-compressed tensors -- [MLX (Apple Silicon)](#mlx-apple-silicon) — pure-PyTorch path for M-series Macs +- [MLX (Apple Silicon)](#mlx-apple-silicon) — Metal kernel port from varjoranta - [Transformers / HuggingFace](#transformers--huggingface) — direct PyTorch monkey-patch - [Docker + Helm](#docker--helm) — ship TurboQuant-serving rigs @@ -21,18 +26,56 @@ ## vLLM -**Status (Apr 17, 2026):** two upstream PRs are in motion. This repo also ships a -self-contained backend plugin under [`vllm_plugin/`](vllm_plugin/) that works against -vLLM ≥ 0.4.0 without waiting for upstream merge. +**Status (May 25, 2026):** TurboQuant is **merged upstream** in vLLM 0.20.x. You can +also install a production fork with CUDA dequant kernels, or use this repo's +self-contained plugin if you want a `pip install`-able path without patching vLLM. | Path | Status | Use when | |---|---|---| -| [vLLM PR #39890 — official grouped `turboquant_3bit` / `turboquant_4bit`](https://github.com/vllm-project/vllm/pull/39890) | Open (2026-04-15) | You want first-class KV-cache-dtype support and are willing to track a PR branch | -| [vLLM PR #38662 — initial TurboQuant attention backend](https://github.com/vllm-project/vllm/pull/38662) | Open (2026-03-31) | You want the full fused attention backend, not just a cache dtype | -| [vLLM PR #39008 — legacy `tq4`](https://github.com/vllm-project/vllm/pull/39008) | **Closed** | Superseded by #39890 — avoid | -| Our [`vllm_plugin/`](vllm_plugin/) backend plugin | Scaffold, works on vLLM ≥ 0.4.0 | You want a plugin that installs with `pip install -e .` and does not require patching vLLM | +| **Upstream `vllm>=0.20.2` `--kv-cache-dtype turboquant_*`** | **Merged** ([docs.vllm.ai](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/)) | **Default. Use this.** | +| [`turboquant-plus-vllm`](https://pypi.org/project/turboquant-plus-vllm/) production fork | v0.13.5 (May 13, 2026) | You need CUDA dequant kernels, MoE support, or native-packed TQ checkpoints | +| [0xSero/turboquant](https://github.com/0xSero/turboquant) Triton reference | 1.1K stars | Research / kernel development; benchmarked on RTX 5090 and 8×RTX 3090 | +| Our [`vllm_plugin/`](vllm_plugin/) backend plugin | Scaffold, works on vLLM ≥ 0.4.0 | You're stuck on an older vLLM and can't patch it | -### Option A — Our bundled plugin (no upstream patching) +### Option A — Upstream vLLM (recommended) + +```bash +pip install "vllm>=0.20.2" + +# Use the recommended TQ variant — 4-bit with norm correction (no QJL) +vllm serve meta-llama/Llama-3.3-70B-Instruct \ + --kv-cache-dtype turboquant_4bit_nc \ + --max-model-len 131072 +``` + +Four variants ship in `vllm>=0.20.2`: + +| Variant | Bits per value | Quality | When to use | +|---|---|---|---| +| `turboquant_k8v4` | 8 K / 4 V | Small drop vs FP8 | Almost always dominated by `fp8` — rarely the right pick | +| `turboquant_4bit_nc` | 4 (no QJL) | Within ~1 pt of FP8 | **The recommended TQ config** | +| `turboquant_k3v4_nc` | 3 K / 4 V (no QJL) | 10–15 pt drop on reasoning | Qwen-class K-dominant models with extreme K/V norm asymmetry | +| `turboquant_3bit_nc` | 3 (no QJL) | 15–25 pt drop on reasoning at long context | Edge / strict memory budget only | + +### Option B — `turboquant-plus-vllm` (production fork with CUDA kernels) + +Use this when you want the 10.1× decode speedup on Qwen3.6-35B-A3B, or when you need +the `varjosoft/Qwen3.6-35B-A3B-TQ3-native` checkpoint (the first frontier MoE model with +weights natively packed in TQ format). + +```bash +pip install turboquant-plus-vllm # bundles vLLM 0.20.2 + FlashInfer 0.5 + +vllm serve varjosoft/Qwen3.6-35B-A3B-TQ3-native \ + --kv-cache-dtype turboquant_3bit_native \ + --max-model-len 131072 +``` + +The `*_native` dtypes assume the model checkpoint already has its weights rotated into +TQ-friendly layout; the per-expert rotation is baked into the weight tensor, which is +what lets the CUDA kernel hit ~49 t/s on RTX 5090 with Qwen3.6-35B-A3B. + +### Option C — Our bundled plugin (no upstream patching) ```bash git clone https://github.com/OnlyTerp/turboquant.git @@ -48,7 +91,7 @@ Environment-variable overrides (all prefixed `TQ_`): ```bash # Precision knobs export TQ_B_MSE=2 # PolarQuant bits per coordinate (default 2 → "3.5-bit" mode) -export TQ_B_QJL=1 # QJL bits per coordinate +export TQ_B_QJL=0 # QJL bits per coordinate (recommend 0 — May 2026 finding) export TQ_FLUSH_INTERVAL=128 # Raw buffer size before compression # GQA / model geometry (vLLM usually populates these automatically) @@ -63,55 +106,51 @@ export TQ_DEVICE=cuda See [`vllm_plugin/README.md`](vllm_plugin/README.md) for the Python API and architecture details. -### Option B — Upstream PR #39890 (grouped official modes) - -```bash -# Check out the PR branch -gh pr checkout 39890 --repo vllm-project/vllm - -pip install -e . --no-build-isolation - -# Run -vllm serve meta-llama/Llama-3.1-8B-Instruct \ - --kv-cache-dtype turboquant_3bit \ - --max-model-len 131072 -``` - -Also supports `turboquant_4bit` (higher quality, larger cache) and the legacy `*_nc` -presets for backward compatibility. - -### Option C — Compose with NVFP4 weights on Blackwell +### Option D — Compose with NVFP4 weights on Blackwell On SM100/SM120 (B100, B200, RTX 5090, RTX PRO 6000) the typical serving config is **NVFP4 -weights × TurboQuant KV**: +weights × FP8 KV**. Switch the KV to TurboQuant only if FP8 doesn't give you enough +headroom: ```bash -# Weights: NVFP4 (requires pre-converted checkpoint) -# KV: TurboQuant 3.5-bit -vllm serve nvidia/Llama-3.1-8B-Instruct-NVFP4 \ +# Default Blackwell config: NVFP4 weights + FP8 KV +vllm serve nvidia/Llama-3.3-70B-Instruct-NVFP4 \ --quantization nvfp4 \ - --kv-cache-dtype turboquant_3bit + --kv-cache-dtype fp8 + +# Switch to TurboQuant only if FP8 KV doesn't fit +vllm serve nvidia/Llama-3.3-70B-Instruct-NVFP4 \ + --quantization nvfp4 \ + --kv-cache-dtype turboquant_4bit_nc ``` -> As of Apr 17, 2026 the NVFP4-KV combination on desktop Blackwell (SM120) requires +> As of May 2026 the NVFP4-KV combination on desktop Blackwell (SM120) still requires > tracking open upstream issues in flashinfer and CUTLASS; see > [Allen Kuo's Apr 16 writeup](https://allenkuo.medium.com/finishing-what-we-started-gemma-4-nvfp4-on-vllm-desktop-blackwell-wsl2-b2088c34815a) -> for the current workarounds on RTX PRO 6000 + WSL2. +> for current workarounds on RTX PRO 6000 + WSL2. --- ## SGLang -**Status (Apr 17, 2026):** first-class `--kv-cache-dtype turboquant` via -[PR #21419](https://github.com/sgl-project/sglang/pull/21419). Ships with Triton kernels -for FWHT rotation, quantize/dequantize, and a fused 4-bit dequant kernel. +**Status (May 25, 2026):** SGLang has **three competing PRs**, none merged. Watch this +space — the SGLang maintainers have stated they will pick one approach once the design +shakes out. For production today, use vLLM instead. + +| PR | Approach | Last activity | +|---|---|---| +| [#21419](https://github.com/sgl-project/sglang/pull/21419) | First-class `--kv-cache-dtype turboquant` with Triton kernels for FWHT + dequant | Mar 30, 2026 — stale | +| [#22048](https://github.com/sgl-project/sglang/pull/22048) | Cleaner abstraction, integrates with SGLang's strategy pattern | May 4, 2026 — active | +| [#23135](https://github.com/sgl-project/sglang/pull/23135) | Triton kernels only (no integration), waiting for the strategy PR | May 19, 2026 — active | + +If you want to experiment today, the most-active branch is #22048: ```bash -gh pr checkout 21419 --repo sgl-project/sglang +gh pr checkout 22048 --repo sgl-project/sglang pip install -e "python[all]" python -m sglang.launch_server \ - --model-path meta-llama/Llama-3.1-8B-Instruct \ + --model-path meta-llama/Llama-3.3-70B-Instruct \ --kv-cache-dtype turboquant \ --host 0.0.0.0 --port 30000 ``` @@ -125,18 +164,30 @@ strategy-pattern abstraction. ## llama.cpp -**Status (Apr 17, 2026):** no TurboQuant-specific path upstream yet, but -[b8779 (Apr 13, 2026)](https://github.com/ggml-org/llama.cpp/releases/tag/b8779) shipped a -Vulkan **DP4A shader for quantized KV flash attention**, making quantized-KV decoding -fast enough to be the default on recent llama.cpp builds. +**Status (May 25, 2026):** Two community forks support native TurboQuant in llama.cpp. +The choice between them is driven by your model's head_dim: + +| Fork | Head dim support | Best at | +|---|---|---| +| [**spiritbuun/llama-cpp-turboquant-cuda**](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) | 128 only (Llama, Qwen2.5, Qwen3 ≤ 14B, Mistral) | Fast on Llama-3 family, smaller binary | +| [**AmesianX/TurboQuant**](https://github.com/AmesianX/TurboQuant) | **Any** (128, 192, 256, 384) | Required for Qwen3.5/3.6, Gemma 4 (head_dim=256), DeepSeek-V3 (head_dim=192) | -The closest off-the-shelf approximation to TurboQuant 3.5-bit today: +The AmesianX fork is the safer default — its P1→P5 head_dim detection cascade gracefully +handles models where `n_embd/n_head` lies (Qwen3.5, Qwen3.6, Gemma 4), and as of +[v1.4.2](https://github.com/AmesianX/TurboQuant/releases/tag/v1.4.2) (Apr 25, 2026) it +has MMA tensor-core acceleration on the inverse-WHT dequant path (49 t/s on RTX 5090 + +Qwen3.6-35B-A3B). ```bash -# Quantize both K and V caches (q4_0 is the widely-supported option) +# Build the AmesianX fork with CUDA +git clone https://github.com/AmesianX/TurboQuant +cd TurboQuant +make GGML_CUDA=1 -j + +# Serve. Use tq3/tq4 cache types (TurboQuant-native) or fall back to q4_0 for compat ./llama-cli \ - -m models/llama-3.1-8b-instruct.Q4_K_M.gguf \ - -ctk q4_0 -ctv q4_0 \ + -m models/qwen3.6-35b-a3b.gguf \ + -ctk tq3 -ctv tq3 \ -fa \ -c 131072 \ -p "your prompt" @@ -144,15 +195,13 @@ The closest off-the-shelf approximation to TurboQuant 3.5-bit today: Flags: -- `-ctk` / `-ctv` — cache type for K and V respectively. Options include `q4_0`, `q4_1`, - `q5_0`, `q5_1`, `q8_0`, `iq4_nl`, `fp16`. -- `-fa` — flash attention, which in b8779+ has the DP4A path for quantized KV. +- `-ctk` / `-ctv` — cache type. The TurboQuant-native types are `tq3` and `tq4`. Standard + `q4_0` / `q4_1` / `q5_0` / `q5_1` / `q8_0` / `iq4_nl` / `fp16` are also supported as + fallbacks. +- `-fa` — flash attention. In [b8779+](https://github.com/ggml-org/llama.cpp/releases/tag/b8779) + upstream this uses a Vulkan DP4A shader for quantized KV; both forks override with their own CUDA path. - `-c` — context length. -For a native TurboQuant path, watch the project -[github.com/hackimov/turboquant-kv](https://github.com/hackimov/turboquant-kv) — the -sibling port has been prototyping a ggml integration, though it is not yet upstream. - --- ## NVIDIA KVPress @@ -250,9 +299,24 @@ context-independent document caching. ## MLX (Apple Silicon) -**Status (Apr 17, 2026):** MLX has no native FP4 tensor cores, but the TurboQuant -pure-PyTorch path runs on M-series Macs via the MPS backend. Quality matches the CUDA -path; throughput is CPU/GPU-bandwidth bound. +**Status (May 25, 2026):** MLX has no native FP4 tensor cores, but the +[varjoranta/turboquant-vllm MLX port](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) +adds native Metal kernels for the FWHT rotation and 3/4-bit dequant. This is faster +than the pure-PyTorch MPS path that this repo ships and is what the community converges +on for M-series Macs. + +```bash +# Install the varjoranta MLX port +pip install "turboquant-mlx" # PyPI package from the varjoranta repo + +# Run a chat loop with TurboQuant KV compression on M3/M4/M5 +python -m turboquant.mlx.chat \ + --model mlx-community/Llama-3.3-70B-Instruct-4bit \ + --kv-bits 4 \ + --kv-mode nc +``` + +If you want to stay on this repo's pure-PyTorch path (slower, but uses the public API): ```bash pip install -e . # installs the pure-PyTorch TurboQuant @@ -320,7 +384,7 @@ docker run --gpus all --ipc=host \ -p 8000:8000 \ -e HF_TOKEN="$HF_TOKEN" \ -e VLLM_ATTENTION_BACKEND=turboquant \ - -e TQ_B_MSE=2 -e TQ_B_QJL=1 \ + -e TQ_B_MSE=2 -e TQ_B_QJL=0 \ turboquant-vllm \ --model meta-llama/Llama-3.1-8B-Instruct \ --max-model-len 131072 @@ -336,7 +400,7 @@ extraEnv: - name: TQ_B_MSE value: "2" - name: TQ_B_QJL - value: "1" + value: "0" # disable QJL per May 2026 community consensus - name: TQ_FLUSH_INTERVAL value: "128" ``` @@ -349,11 +413,13 @@ extraEnv: |---|---|---| | Python | 3.10, 3.11, 3.12 | `match` statements used internally | | PyTorch | 2.1+ | MPS works on Apple Silicon | -| CUDA | 11.8, 12.1, 12.4, 12.6 | 12.6+ recommended for Blackwell | +| CUDA | 11.8, 12.1, 12.4, 12.6, 12.8 | 12.8+ recommended for Blackwell SM120 | | Triton | 2.2+ (optional) | Enables GPU kernels in [`src/kernels.py`](src/kernels.py) | -| vLLM | 0.4.0+ | 0.9+ recommended for the plugin entry point | -| SGLang | main (PR #21419 branch) | Wait for merge for a stable tag | -| llama.cpp | b8779+ | DP4A flash-attn for quantized KV | +| vLLM (upstream) | **0.20.2+** | Merged `--kv-cache-dtype turboquant_*` | +| vLLM (production fork) | turboquant-plus-vllm 0.13.5 | Bundles CUDA dequant + FlashInfer CUTLASS MoE | +| vLLM (plugin path) | 0.4.0+ | 0.9+ recommended for the plugin entry point | +| SGLang | main — PRs [#21419](https://github.com/sgl-project/sglang/pull/21419), [#22048](https://github.com/sgl-project/sglang/pull/22048), [#23135](https://github.com/sgl-project/sglang/pull/23135) | None merged as of May 2026 | +| llama.cpp | AmesianX v1.4.2 / spiritbuun latest | Use AmesianX for head_dim ≥ 192 | | KVPress | 0.4.0 | Stack eviction × precision | | LMCache | latest | Cross-process cache sharing | @@ -391,6 +457,32 @@ Fall back to the Triton backend: `--attention-backend TRITON`. ### "Throughput is slow on H100" -The pure-PyTorch path is a reference, not a production kernel. On H100, use the Triton -kernels (`src/kernels.py`) or SGLang's fused kernels from PR #21419. The PyTorch path is -roughly 25× slower than FP16 attention; Triton closes ~80% of that gap. +The pure-PyTorch path is a reference, not a production kernel. For production on H100: + +1. **First, try `--kv-cache-dtype fp8`.** It matches BF16 quality at zero throughput + cost on Hopper/Blackwell. If that fits your memory budget, you don't need TurboQuant. +2. If you need >2× compression, use + [`turboquant-plus-vllm`](https://pypi.org/project/turboquant-plus-vllm/) (CUDA dequant + kernels, 10.1× over the reference path) or + [0xSero/turboquant](https://github.com/0xSero/turboquant) (Triton kernels). +3. The plain PyTorch path is roughly 25× slower than FP16 attention; the Triton kernels + in `src/kernels.py` close ~80% of that gap but still aren't competitive with the + production forks. + +### "Output quality is worse than the paper claims" + +The most common culprits, in order: + +1. **You're using a QJL-augmented variant at b ≤ 3.** Switch to a `*_nc` (no-QJL) variant + or set `qjl_score_weight=0.0`. See [FAQ "Does the QJL step actually help?"](FAQ.md#does-the-qjl-step-actually-help). +2. **You're not skipping the first / last 2 layers.** vLLM's `*_nc` paths do this + automatically; custom integrations need to add it. +3. **Your model has head_dim > 128 and you're using the spiritbuun llama.cpp fork.** + Switch to [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant), which detects + head_dim correctly for Qwen3.5, Qwen3.6, Gemma 4. +4. **You're using 3-bit modes on reasoning workloads.** The Red Hat eval showed 15–25 + pt drops on AIME25 at long context with `turboquant_3bit_nc`. Use `turboquant_4bit_nc` + or fall back to FP8 KV. +5. **You're benchmarking with cosine similarity on random vectors instead of MRCR / + LongBench / AIME25.** Cosine similarity doesn't predict end-to-end quality — see + [FAQ "What's the right way to benchmark my model?"](FAQ.md#whats-the-right-way-to-benchmark-my-model). diff --git a/LANDSCAPE_2026.md b/LANDSCAPE_2026.md index ef46888..107fd61 100644 --- a/LANDSCAPE_2026.md +++ b/LANDSCAPE_2026.md @@ -1,13 +1,45 @@ # The 2026 KV-Cache Compression Landscape -> **Why this file exists.** TurboQuant is one entry in a fast-moving ecosystem. In the 10 -> weeks around ICLR 2026 alone, at least a dozen new KV-cache compression methods shipped -> code or papers — many of them specifically benchmarked *against* TurboQuant. This guide -> orients you across the landscape so you can pick the right tool (or combine them) for -> your hardware, context length, and quality budget. +> **Why this file exists.** TurboQuant is one entry in a fast-moving ecosystem. In the +> 14 weeks between the ICLR 2026 poster and now, the community has gone from "TurboQuant +> is SOTA, use it everywhere" to a much more textured picture in which TurboQuant is the +> right tool only in a specific regime. This guide orients you across the landscape so +> you can pick the right tool (or combine them) for your hardware, context length, and +> quality budget. > -> Last refreshed: **2026-04-17** (Apr 14–17 = past 72 hours were especially busy — see -> [What's New](#whats-new-april-2026)). +> Last refreshed: **2026-05-25**. The biggest update vs the April 17 revision: the +> [Red Hat AI / vLLM evaluation (May 11)](https://blog.vllm.ai/2026/05/11/turboquant.html) +> and the community consensus around dropping QJL. + +--- + +## The big May 2026 update + +Three things changed since the previous refresh: + +1. **The Red Hat AI / vLLM team published the first comprehensive third-party evaluation** + of TurboQuant in [a May 11 blog post](https://blog.vllm.ai/2026/05/11/turboquant.html). + Across Llama-3.3-70B, Qwen3-30B-A3B, and MiniMax-M2.7 on long-context retrieval (MRCR) + and reasoning (AIME25, GPQA, MATH500, LiveCodeBench-v6), they conclude: + - **FP8 (E4M3) KV is the best default** — no measurable accuracy loss, no throughput hit. + - **TurboQuant `4bit_nc`** is the only TQ variant viable for production. + - **TurboQuant `3bit_nc` / `k3v4_nc`** drop 15–25 pts on reasoning at long context. + - TurboQuant `k8v4` is dominated by FP8. + - Skipping the first and last two layers from compression preserves most of the + quality at almost the same compression ratio. +2. **Community ports independently converged on dropping QJL.** [tonbistudio's V3 + PyTorch port](https://github.com/tonbistudio/turboquant-pytorch), the + [scos-lab benchmark across 8 models](https://github.com/tonbistudio/turboquant-pytorch/issues/8), + and [0xSero's Triton kernels](https://github.com/0xSero/turboquant) all find that + the 1-bit QJL residual *adds* error at b ≤ 3 because softmax amplifies its variance. + The `*_nc` ("norm correction") variants that landed in vLLM are the official answer. +3. **The serving stack consolidated.** vLLM merged + `--kv-cache-dtype turboquant_{k8v4, 4bit_nc, k3v4_nc, 3bit_nc}` upstream. A production + fork ([`turboquant-plus-vllm`](https://pypi.org/project/turboquant-plus-vllm/)) ships + CUDA dequant kernels with **10.1× decode speedup on Qwen3.6-35B-A3B-TQ3**. llama.cpp + has two forks ([spiritbuun](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) + for head_dim=128, [AmesianX](https://github.com/AmesianX/TurboQuant) for any head_dim). + SGLang remains WIP with three competing PRs. --- @@ -15,35 +47,75 @@ | Method | Axis | Best at | TurboQuant relationship | |---|---|---|---| -| **TurboQuant** (this repo) | **Rotation + scalar VQ + QJL residual** | **Data-oblivious, online, near information-theoretic optimal at 2.5–4 bpv** | — | -| **NVFP4 / MXFP4 KV** | Hardware-native FP4 block quant | Blackwell/SM100/SM120 throughput on large serving rigs | **Complementary** — TQ is the algorithm, NVFP4 is the container. They can coexist in one engine | -| **TriAttention** (Apr 2026) | Pre-RoPE Q/K concentration → trigonometric importance scoring | **Long reasoning (AIME-style 32K CoT)** via eviction, up to 10.7× KV memory reduction | **Orthogonal** — TQ compresses every retained token, TriAttention decides *which* tokens to retain | +| **FP8 (E4M3) KV** | Hardware FP8 container | **Best default in May 2026** — 2× compression, no measurable loss | **Strictly better than TQ when 2× is enough** | +| **TurboQuant `4bit_nc`** (this repo) | **Rotation + scalar VQ, no QJL** | **Data-oblivious 2.6–3.1× compression** when FP8 doesn't fit | The recommended TQ variant | +| **TurboQuant `3bit_nc`** | Rotation + scalar VQ, no QJL, 3 bpv | Memory-bound edge / consumer GPUs | Accept 15–25 pt drops on reasoning at long context | +| **TurboQuant (paper, w/ QJL)** | Rotation + scalar VQ + 1-bit QJL residual | The published algorithm | Worse than `*_nc` in independent reproductions | +| **NVFP4 / MXFP4 KV** | Hardware-native FP4 block quant | Blackwell/SM100/SM120 throughput on large serving rigs | **Complementary** — TQ is the algorithm, NVFP4 is the container | +| **TriAttention** (Apr 2026) | Pre-RoPE Q/K concentration → trigonometric importance scoring | **Long reasoning (AIME-style 32K CoT)** via eviction, up to 10.7× KV memory reduction | **Orthogonal** — TQ compresses every retained token, TriAttention decides which ones | | **Adaptive KV-quant** (Apr 6, 2026) | Learned per-token bit-width {2, 4, 8, FP16} | On-device / edge LLMs where static bits waste budget | Can wrap TQ as the 2-bit/4-bit backend | | **LRKV (Low-Rank KV)** (Apr 9, 2026) | Shared full-rank basis + head-specific low-rank residual | Training-time architectural fix (45–53% KV reduction) | **Architectural** — applies at pretraining; TQ applies at inference | | **DeepSeek MLA** | Multi-head Latent Attention: down-projection of K/V | Architectural compression, native to the model weights | Model-side; TQ still compresses the residual latent KV | -| **KIVI** (ICLR'24) | Per-channel K + per-token V asymmetric 2-bit | Simple, calibration-free 2-bit baseline | Superseded on LongBench by TQ at 3.5 bpv | +| **KIVI** (ICLR'24) | Per-channel K + per-token V asymmetric 2-bit | Simple, calibration-free 2-bit baseline | Predecessor; TQ `4bit_nc` is better on long-context | | **KVQuant** | Per-channel + pre-RoPE K + dense-sparse split | Sub-4-bit with outlier preservation | Outlier handling similar in spirit to TQ's two-rotation split | -| **ParoQuant** (ICLR'26, Apr 2026) | Scaled pairwise rotation for **weight** quant | INT4 **weights** with +2.4% over AWQ on reasoning | **Weight-side, not KV-side** — stack with TurboQuant for W4/KV3.5 | +| **ParoQuant** (ICLR'26, Apr 2026) | Scaled pairwise rotation for **weight** quant | INT4 **weights** with +2.4% over AWQ on reasoning | **Weight-side, not KV-side** — stack with TurboQuant | | **SnapKV / H2O / PyramidKV / StreamingLLM** | Importance-based token eviction | High-ratio drop for long context (>100K) | **Orthogonal** — combine with TQ for multiplicative savings | | **KV Packet** (Apr 14, 2026) | Context-independent cache reuse via soft-token adapters | Cross-document KV reuse in RAG without recompute | Orthogonal — caches TQ-compressed packets across sessions | -| **NVIDIA KVPress** | Framework of press strategies (ExpectedAttention, ThinK, etc.) | Plug-in eviction & budget control | Orthogonal — KVPress can drive *which* tokens TQ keeps | +| **NVIDIA KVPress** | Framework of press strategies (ExpectedAttention, ThinK, etc.) | Plug-in eviction & budget control | Orthogonal — KVPress can drive which tokens TQ keeps | | **[kvtc](https://github.com/OnlyTerp/kvtc)** (sibling project) | PCA-decorrelated channel rotation + per-channel scalar quant | Calibration-based KV compression when a few samples of real activations are available | **Sibling** — same authors, data-**aware** variant; TurboQuant stays data-*oblivious* for zero-setup deployments | The short summary: **TurboQuant is a per-token precision compressor; the other axis is token selection/eviction; the third axis is hardware format (FP4/FP8/INT4).** The three -compose. +compose — but in May 2026, FP8 is the precision-axis default unless you specifically need +more than 2× compression. --- -## What's new (April 2026) - -### Past 72 hours (Apr 14–17, 2026) +## What's new (May 2026) + +### May 11–18, 2026 + +- **May 18 — [Towards AI long-context follow-up](https://pub.towardsai.net/)** independent + verification on Llama-3.1-70B that 3-bit TQ degrades dramatically beyond 32K context. +- **May 17 — [AI Intensify: "TurboQuant — is compression and performance worth the hype?"](https://ai-intensify.com/turboquant-is-compression-and-performance-worth-the-hype/)** + Evaluation against FP8 across three models. Same conclusion as Red Hat: FP8 is the + better default; TQ wins only when 2× isn't enough. +- **May 16 — [scos-lab/turboquant 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8)** + Pure-MSE beats MSE+QJL on every model tested. Establishes the K/V-norm-ratio metric as + the single best predictor of TQ compression quality. +- **May 14 — [TeqVolt: "TurboQuant — Google's 6× KV Cache Compression"](https://teqvolt.com/deep-dives/turboquant-google-6x-kv-cache-compression-3-bit-zero-loss)** + Third-party deep-dive on the algorithm. Notes the paper's "zero loss" claim doesn't + generalize cleanly to frontier MoE models. +- **May 13 — [varjoranta/turboquant-vllm v0.13.5](https://github.com/varjoranta/turboquant-vllm/releases/tag/v0.13.5)** + End-to-end Qwen3.6-35B-A3B-TQ3-native support through vLLM 0.20.2 + FlashInfer CUTLASS + MoE. First native-packed TQ checkpoint for a frontier model. +- **May 11 — [**Red Hat AI / vLLM blog: "A First Comprehensive Study of TurboQuant"**](https://blog.vllm.ai/2026/05/11/turboquant.html)** + by Eldar Kurtić, Michael Goin, Alexandre Marques. The most important data point in the + current ecosystem. Frames TurboQuant against the FP8 KV baseline rigorously for the + first time. + +### Late April 2026 + +- **Apr 23 — [varjoranta/turboquant-vllm v0.13.0](https://github.com/varjoranta/turboquant-vllm/releases/tag/v0.13.0)** + Block-diagonal WHT CUDA kernel: 10.1× decode speedup on Qwen3.6-35B-A3B-TQ3 versus the + reference PyTorch path. +- **Apr 22 — [vLLM merges canonical TurboQuant KV dtypes](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/)** + `--kv-cache-dtype turboquant_{k8v4, 4bit_nc, k3v4_nc, 3bit_nc}` ships in vLLM 0.20.x. + These names supersede the earlier `tq3` / `tq4` / `pq4` proposals. +- **Apr 18 — [AmesianX/TurboQuant v1.3.0](https://github.com/AmesianX/TurboQuant/releases/tag/v1.3.0)** + P1→P5 head_dim detection cascade lands; fixes head_dim=128 / 256 / 384 confusion on + Qwen3.5, Qwen3.6, Gemma 4. Critical finding: `n_embd/n_head` lies for those models. +- **Apr 25 — [AmesianX v1.4.2](https://github.com/AmesianX/TurboQuant/releases/tag/v1.4.2)** + MMA tensor-core acceleration on the inverse-WHT dequant path: 30 → 49 t/s (+63%) on + GeForce RTX 5090 with Qwen3.6-35B-A3B. + +### Apr 14–17, 2026 - **Apr 15 — [LMCache blog: "What is TurboQuant and why it matters for LLM inference"](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/)** - by Kuntai Du (Tensormesh). Notes that TurboQuant is "really hot these days" and that - some X/Twitter commentators are calling it "the most significant AI breakthrough this - year." Frames TurboQuant as **the** answer to Jensen Huang's repeated GTC 2026 warnings - about KV cache memory dominating inference cost. + by Kuntai Du (Tensormesh). The best layperson explainer of the algorithm. Frames + TurboQuant as an answer to Jensen Huang's GTC 2026 warnings about KV cache memory + dominating inference cost. Worth reading even though the May 2026 Red Hat evaluation + has since added important caveats. - **Apr 15 — [vLLM PR #39890: "Add official 3-bit and 4-bit grouped TurboQuant modes"](https://github.com/vllm-project/vllm/pull/39890)** by @erhan1209. Adds canonical `turboquant_4bit` / `turboquant_3bit` KV-cache dtypes with grouped Triton store/decode paths, moving beyond the earlier `*_nc` legacy presets. @@ -161,15 +233,24 @@ suddenly plausible in 2026 — no single axis gets you there. ### TurboQuant (this repo) - **Algorithm:** random rotation (Hadamard or Haar) → scalar Lloyd-Max per coordinate → - QJL 1-bit residual for unbiased inner product. + optionally a QJL 1-bit residual for unbiased inner product. - **Strengths:** **data-oblivious**, no calibration, zero compounding across layers (scalar per-coordinate, no coupling), provably within 2.7× of information-theoretic optimal distortion. -- **Benchmarks:** 3.5 bpv = identical LongBench on Llama-3.1-8B; 2.5 bpv = marginal - degradation (from the paper). +- **Benchmarks (paper):** 3.5 bpv = identical LongBench on Llama-3.1-8B; 2.5 bpv = + marginal degradation. **But:** the [Red Hat May 2026 evaluation](https://blog.vllm.ai/2026/05/11/turboquant.html) + finds 3-bit configurations drop 15–25 pts on reasoning + long-context retrieval, and + the `*_nc` (no QJL) variants are uniformly better than the paper's `*_q` variants. +- **Variants shipped in vLLM** (May 2026): + - `turboquant_k8v4` — 8-bit K, 4-bit V. Dominated by FP8 KV; rarely the right choice. + - `turboquant_4bit_nc` — 4 bpv MSE-only, no QJL. **The recommended TQ config.** + - `turboquant_k3v4_nc` — 3-bit K, 4-bit V, no QJL. For Qwen-class K-dominant models. + - `turboquant_3bit_nc` — 3 bpv MSE-only. Edge / memory-bound only. - **Weaknesses:** pure PyTorch reference is ~25× slower than FP16 on CPU; Triton kernels experimental; outlier handling differs slightly from paper (see - [IMPLEMENTATION_NOTES.md](IMPLEMENTATION_NOTES.md)). + [IMPLEMENTATION_NOTES.md](IMPLEMENTATION_NOTES.md)). For production, use + [`turboquant-plus-vllm`](https://pypi.org/project/turboquant-plus-vllm/) or + [0xSero/turboquant](https://github.com/0xSero/turboquant). ### TriAttention (arXiv 2604.04921, Apr 2026) @@ -281,20 +362,24 @@ suddenly plausible in 2026 — no single axis gets you there. ## Decision guide: which compressor do I actually want? > Use this to pick a starting configuration. All of these can be **stacked** — this chart -> just tells you where to start. +> just tells you where to start. Updated May 2026 to reflect the +> [Red Hat AI evaluation findings](https://blog.vllm.ai/2026/05/11/turboquant.html). | Scenario | Start with | Then add | Why | |---|---|---|---| -| **Datacenter Blackwell (B100/B200), max throughput** | NVFP4 weights + NVFP4 KV | — | Hardware-native FP4 path is the throughput king | -| **Datacenter Hopper (H100/H200), long context (>128K)** | FP8 weights + **TurboQuant 3.5-bit KV** | SnapKV / KVPress if you're beyond 1M tokens | TQ beats KIVI-2bit quality at matched memory; H100 FP8 is native | -| **Consumer Blackwell (RTX 5090, RTX PRO 6000 96GB)** | NVFP4 weights + **TurboQuant 3.5-bit KV** | — | RTX PRO 6000 96GB + TQ = Llama-4-Maverick-class on one card | -| **Consumer Ada (RTX 4090/4080)** | AWQ-INT4 weights + **TurboQuant 3.5-bit KV** | TriAttention / SnapKV for 32K+ reasoning | Matches the [TriAttention single-4090 OpenClaw result](https://arxiv.org/abs/2604.04921) | -| **Apple Silicon (M3/M4/M5)** | MLX-INT4 weights + TurboQuant (CPU path) | — | MLX lacks NVFP4; TQ pure-PyTorch path works | -| **On-device / edge (phone, Jetson)** | Adaptive KV-Quant or TurboQuant-2.5-bit | Token eviction (H2O / StreamingLLM) | Per-token bit allocation is essential at these budgets | -| **RAG systems, high cache reuse** | TurboQuant 3.5-bit KV | **KV Packet** or **LMCache** | Stack across-session reuse on top of per-token compression | -| **Reasoning / long CoT (AIME, AIME25, math)** | TurboQuant 3.5-bit KV | **TriAttention** | Token selection dominates at 32K+ CoT; TQ handles the precision | +| **I just want something that works today** | **FP8 KV** (`--kv-cache-dtype fp8`) | — | Best default in May 2026 — no measurable accuracy loss, no throughput hit | +| **Datacenter Hopper/Blackwell, FP8 fits** | FP8 weights + **FP8 KV** | SnapKV / KVPress beyond 1M | Hardware-native, simplest, fastest | +| **Datacenter Hopper, FP8 doesn't fit** | FP8 weights + **`turboquant_4bit_nc`** | Skip first/last 2 layers | TQ earns its keep when you need >2× compression | +| **Datacenter Blackwell (B100/B200), max throughput** | NVFP4 weights + **FP8 KV** | — | Hardware-native FP4 path is the throughput king | +| **Single-card 70B+ at 128K+** | AWQ/NVFP4 + **`turboquant_4bit_nc`** | — | TQ 4bit_nc is roughly 1.5× more compression than FP8, just enough for these configurations | +| **Consumer Blackwell (RTX 5090, RTX PRO 6000 96GB)** | NVFP4 weights + **`turboquant_4bit_nc`** | — | RTX PRO 6000 96GB + TQ = Llama-4-Maverick-class on one card | +| **Consumer Ada (RTX 4090/4080) — no FP8 attention** | AWQ-INT4 weights + **`turboquant_4bit_nc`** | TriAttention / SnapKV for 32K+ reasoning | Ada lacks FP8 attention, so FP8 KV is no longer "free" | +| **Apple Silicon (M3/M4/M5)** | MLX-INT4 weights + TurboQuant via [varjoranta MLX port](https://github.com/varjoranta/turboquant-vllm) | — | MLX lacks NVFP4 + FP8 attention; TQ Metal kernels work | +| **On-device / edge (phone, Jetson)** | Adaptive KV-Quant or `turboquant_3bit_nc` | Token eviction (H2O / StreamingLLM) | Per-token bit allocation is essential at these budgets | +| **RAG systems, high cache reuse** | `turboquant_4bit_nc` KV | **KV Packet** or **LMCache** | Stack across-session reuse on top of per-token compression | +| **Reasoning / long CoT (AIME, AIME25, math)** | **FP8 KV** (3-bit TQ drops accuracy) | **TriAttention** | Red Hat eval shows 3-bit TQ degrades hard at long context; use FP8 + token selection instead | +| **Qwen3 / Qwen3.5 / Qwen3.6 (asymmetric K/V norms)** | **`turboquant_k3v4_nc`** | head_dim=256? use [AmesianX llama.cpp](https://github.com/AmesianX/TurboQuant) | Asymmetric K-heavy bit allocation matters for Qwen-family | | **Pretraining a new model** | **LRKV** attention or **MLA** | TurboQuant at inference time | Architectural fix + inference compression multiply | -| **I just want something that works today** | **TurboQuant 3.5-bit KV** (this repo) | — | Calibration-free, data-oblivious, 3 lines to integrate | --- @@ -302,14 +387,18 @@ suddenly plausible in 2026 — no single axis gets you there. See [INTEGRATIONS.md](INTEGRATIONS.md) for concrete commands and links. Summary: -| Engine | Status (Apr 17, 2026) | How to use TurboQuant | +| Engine | Status (May 25, 2026) | How to use TurboQuant | |---|---|---| -| **vLLM** | WIP (PRs [#38662](https://github.com/vllm-project/vllm/pull/38662), [#39890](https://github.com/vllm-project/vllm/pull/39890)) | `--kv-cache-dtype turboquant_3bit` (PR #39890); or our `vllm_plugin/` backend plugin | -| **SGLang** | WIP (PR [#21419](https://github.com/sgl-project/sglang/pull/21419)) | `--kv-cache-dtype turboquant` | -| **llama.cpp** | DP4A flash-attn for quantized KV in [b8779 (Apr 13)](https://github.com/ggml-org/llama.cpp/releases/tag/b8779); TQ-specific path not yet upstream | Use `-ctk q4_0 -ctv q4_0` today for a close proxy | -| **LMCache** | First-class coverage in [their Apr 15 blog post](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) | Compose TQ-compressed tensors inside LMCache cross-session cache | +| **vLLM (upstream)** | **Merged** — see [docs.vllm.ai](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/) | `--kv-cache-dtype turboquant_{k8v4, 4bit_nc, k3v4_nc, 3bit_nc}` | +| **vLLM (production fork)** | [`turboquant-plus-vllm` v0.13.5](https://pypi.org/project/turboquant-plus-vllm/) — CUDA dequant kernels, MoE support, native checkpoints | `pip install turboquant-plus-vllm` | +| **vLLM (Triton reference)** | [0xSero/turboquant](https://github.com/0xSero/turboquant) | clone + `pip install -e .` | +| **SGLang** | WIP — competing PRs [#21419](https://github.com/sgl-project/sglang/pull/21419), [#22048](https://github.com/sgl-project/sglang/pull/22048), [#23135](https://github.com/sgl-project/sglang/pull/23135); none merged | `--kv-cache-dtype turboquant` on a PR branch | +| **llama.cpp (head_dim=128)** | [spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) | `-ctk tq3 -ctv tq3 -fa` | +| **llama.cpp (any head_dim)** | [AmesianX/TurboQuant v1.4.2](https://github.com/AmesianX/TurboQuant) — supports head_dim=256+ (Qwen3.5, Qwen3.6, Gemma 4) | `-ctk tq3 -ctv tq3 -fa` | +| **PyTorch reference (QJL-free)** | [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) V3 | `pip install turboquant-pytorch` | +| **LMCache** | First-class; [explainer blog](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) | Compose TQ-compressed tensors inside LMCache cross-session cache | | **NVIDIA KVPress** | Eviction/selection framework | Use KVPress for token eviction, TQ for per-token precision | -| **MLX (Apple)** | No native FP4; use TQ PyTorch path | `pip install -e .` and run `src/demo.py` on Apple Silicon | +| **MLX (Apple)** | [varjoranta Metal kernels](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) | `python -m turboquant.mlx.chat --model ` | --- @@ -334,26 +423,108 @@ becomes dominant, and precision methods alone under-perform. See the --- +## Independent reproductions (May 2026) + +The May 2026 evaluations are markedly less rosy than the paper. Headline numbers from +the [Red Hat AI / vLLM blog post (May 11)](https://blog.vllm.ai/2026/05/11/turboquant.html): + +### Llama-3.3-70B-Instruct, MRCR long-context retrieval + +| KV format | KV reduction vs BF16 | MRCR-8 needle (avg) | MRCR-16 needle (avg) | Comment | +|---|---|---|---|---| +| BF16 | 1.0× | 1.000 | 1.000 | baseline | +| **FP8 (E4M3)** | **2.0×** | **0.997** | **0.991** | **no measurable loss** | +| `turboquant_k8v4` | 2.0× | 0.989 | 0.973 | dominated by FP8 | +| `turboquant_4bit_nc` | 2.6× | 0.984 | 0.957 | the practical TQ choice | +| `turboquant_k3v4_nc` | 3.4× | 0.940 | 0.834 | usable, with care | +| `turboquant_3bit_nc` | 3.6× | 0.901 | 0.722 | large degradation | + +### Qwen3-30B-A3B, AIME25 reasoning (Pass@1 over 32 samples) + +| KV format | KV reduction | AIME25 acc | Comment | +|---|---|---|---| +| BF16 | 1.0× | **0.683** | baseline | +| **FP8** | **2.0×** | **0.680** | matches BF16 | +| `turboquant_k8v4` | 2.0× | 0.668 | small drop | +| `turboquant_4bit_nc` | 2.6× | 0.652 | tolerable | +| `turboquant_k3v4_nc` | 3.4× | 0.532 | ~15 pt drop | +| `turboquant_3bit_nc` | 3.5× | 0.443 | ~24 pt drop | + +### Throughput on Qwen3-30B-A3B, single H100 + +| KV format | Tokens/s | vs BF16 | +|---|---|---| +| BF16 | 4,520 | 1.00× | +| **FP8** | **4,510** | **1.00×** (no penalty!) | +| `turboquant_4bit_nc` | 2,680 | 0.59× | +| `turboquant_3bit_nc` | 2,140 | 0.47× | + +> **Why TurboQuant is slower than FP8 at decode:** FP8 quantizes both the KV cache *and* +> the attention compute on hardware FP8 tensor cores. TurboQuant only quantizes storage — +> the attention compute still happens in BF16/FP16 after dequantization. So you pay the +> dequant cost on every decode step without getting the compute speedup. The +> [varjoranta CUDA kernels](https://github.com/varjoranta/turboquant-vllm) and +> [0xSero Triton kernels](https://github.com/0xSero/turboquant) recover some of this gap +> through fused dequant + GEMV, but not all. + +The community-port findings reinforce the same picture from a different angle: + +- [tonbistudio V3 README](https://github.com/tonbistudio/turboquant-pytorch): "We + initially shipped the paper's QJL-augmented MSE+sign quantizer. Our V3 ships pure MSE + because, on every model we tested, the QJL step amplifies softmax noise and hurts + end-to-end output quality." +- [scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8): + "The K/V norm ratio is the single best predictor of compression quality. GPT-2-124M + shows a 6× ratio; Qwen3.5-7B shows 106×; Qwen3.6-35B shows 182×. Models with extreme + ratios benefit from asymmetric K-heavy bit allocation." +- [Towards AI (May 18)](https://pub.towardsai.net/): "Independent measurement across + three implementations — sharpner/MLX, tonbistudio/PyTorch, 0xSero/Triton — found the + same result: PolarQuant works, QJL doesn't help at b ≤ 3." + +--- + ## Further reading +### Primary sources + - **Original paper:** [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/abs/2504.19874) (Zandieh, Daliri, Hadian, Mirrokni; ICLR 2026). - **ICLR 2026 poster:** [iclr.cc/virtual/2026/poster/10006985](https://iclr.cc/virtual/2026/poster/10006985) - (Sat, Apr 25, 2026, 11:15 AM PDT). -- **Companion papers:** - - [QJL (AAAI 2025)](https://dl.acm.org/doi/10.1609/aaai.v39i24.34773) — the 1-bit - sign-based JL transform TurboQuant uses for its residual. - - [PolarQuant (arXiv 2502.02617)](https://arxiv.org/abs/2502.02617) — the recursive - polar-coordinate predecessor TurboQuant deliberately avoids. -- **Related implementations:** - - [hackimov/turboquant-kv](https://github.com/hackimov/turboquant-kv) — sibling - open-source port with alternative kernel design. -- **Our own:** - - [README.md](README.md) — quickstart and headline numbers. - - [BENCHMARKS.md](BENCHMARKS.md) — memory tables, throughput targets, and methodology. - - [INTEGRATIONS.md](INTEGRATIONS.md) — vLLM / SGLang / llama.cpp / MLX / KVPress - concrete commands. - - [FAQ.md](FAQ.md) — common misconceptions ("does this replace AWQ?", etc.). - - [IMPLEMENTATION_NOTES.md](IMPLEMENTATION_NOTES.md) — rotation modes, outlier - channels, QJL projection, Blackwell stacking. - - [pseudocode.md](pseudocode.md) — line-by-line paper pseudocode for re-implementers. + (archive — presented April 25, 2026). +- **Google Research blog:** ["TurboQuant: Redefining AI efficiency with extreme compression"](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) + (March 24, 2026). + +### Companion papers + +- [QJL (AAAI 2025)](https://dl.acm.org/doi/10.1609/aaai.v39i24.34773) — the 1-bit + sign-based JL transform TurboQuant *can* use for its residual. Independent reproductions + in May 2026 find this stage frequently hurts at b ≤ 3 (see above). +- [PolarQuant (arXiv 2502.02617)](https://arxiv.org/abs/2502.02617) — the recursive + polar-coordinate predecessor TurboQuant deliberately avoids. + +### Independent evaluations (May 2026) + +- [Red Hat AI / vLLM: "A First Comprehensive Study of TurboQuant"](https://blog.vllm.ai/2026/05/11/turboquant.html) (May 11) — the definitive third-party eval. +- [TeqVolt: "TurboQuant — Google's 6× KV Cache Compression"](https://teqvolt.com/deep-dives/turboquant-google-6x-kv-cache-compression-3-bit-zero-loss) (May 14) — algorithm deep-dive. +- [AI Intensify: "Is compression and performance worth the hype?"](https://ai-intensify.com/turboquant-is-compression-and-performance-worth-the-hype/) (May 17) — three-model evaluation. +- [scos-lab/turboquant 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8) — establishes the K/V-norm-ratio predictor. + +### Production implementations + +- [vLLM upstream `turboquant_*` dtypes](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/) — merged April 2026. +- [0xSero/turboquant](https://github.com/0xSero/turboquant) — Triton kernels, 1.1K stars. +- [varjoranta/turboquant-vllm](https://github.com/varjoranta/turboquant-vllm) — CUDA kernels, MoE support, MLX port. +- [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant) — llama.cpp with head_dim=256+ support. +- [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) — V3 PyTorch port that drops QJL. +- [spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) — llama.cpp CUDA, head_dim=128. + +### Our own + +- [README.md](README.md) — quickstart and headline numbers. +- [BENCHMARKS.md](BENCHMARKS.md) — memory tables, throughput targets, and methodology. +- [INTEGRATIONS.md](INTEGRATIONS.md) — vLLM / SGLang / llama.cpp / MLX / KVPress + concrete commands. +- [FAQ.md](FAQ.md) — common misconceptions ("does this replace AWQ?", "FP8 vs TQ?", etc.). +- [IMPLEMENTATION_NOTES.md](IMPLEMENTATION_NOTES.md) — rotation modes, outlier + channels, QJL projection, Blackwell stacking. +- [pseudocode.md](pseudocode.md) — line-by-line paper pseudocode for re-implementers. diff --git a/LAUNCH.md b/LAUNCH.md index 7d6a483..3284f16 100644 --- a/LAUNCH.md +++ b/LAUNCH.md @@ -1,4 +1,21 @@ -# TurboQuant Launch Kit +# TurboQuant Launch Kit (Archive — April 2026) + +> **Archive notice (May 2026):** This file collected ready-to-post content for the +> **ICLR 2026 poster week** launch window in April 2026. It's preserved as a snapshot +> of the project's marketing materials at that moment. The framing below reflects the +> *pre-Red Hat-evaluation* messaging; for the current accurate framing see +> [README.md](README.md) and [LANDSCAPE_2026.md](LANDSCAPE_2026.md). In particular: +> +> - The "near-zero accuracy loss" hook overstates the case for 3-bit modes (see +> [Red Hat May 11 eval](https://blog.vllm.ai/2026/05/11/turboquant.html)). +> - Use `turboquant_4bit_nc` for production, not the QJL-augmented variants the hooks +> below imply. +> - FP8 KV is the better default in May 2026; TurboQuant earns its place when +> compression > 2× is required. +> +> If you reuse any of this content, update the headline claim accordingly. + +--- This file collects ready-to-post content for the **ICLR 2026 poster week** launch window (Zandieh et al. poster on **Sat Apr 25, 11:15 AM PDT**, session 10006985). diff --git a/README.md b/README.md index e0e2556..0355397 100644 --- a/README.md +++ b/README.md @@ -9,46 +9,102 @@ [![PyTorch](https://img.shields.io/badge/PyTorch-2.1+-ee4c2c.svg)](https://pytorch.org/) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/OnlyTerp/turboquant/blob/master/notebooks/demo.ipynb) -**Compress your LLM's KV cache by 5–7× with near-zero accuracy loss.** Run longer contexts, serve more users, use less GPU memory. +**A data-oblivious KV-cache compressor.** Random rotation + scalar Lloyd-Max quantizer +that shrinks the KV cache to 3–4× smaller than FP16, no calibration data required. +Useful when GPU memory is the binding constraint on your serving rig. -> First open-source implementation of [Google's TurboQuant](https://arxiv.org/abs/2504.19874) (ICLR 2026). 3.5 bits/value = near-identical quality to FP16. Provably within 2.7× of information-theoretic optimal. +> Open-source implementation of [Google's TurboQuant](https://arxiv.org/abs/2504.19874) (ICLR 2026). Provably within 2.7× of the information-theoretic distortion bound. -### Verified on real hardware (Apr 2026) +### Reality check (May 2026) -| Where | Headline result | +A lot has happened since the ICLR poster. **TurboQuant is not always the right answer** +and the community has converged on a more nuanced picture than the initial hype suggested. +Read this table before you start: + +| Finding | Source | What it means for you | +|---|---|---| +| **FP8 KV cache is the best default** on Hopper/Blackwell — 2× capacity, no measurable accuracy loss, no throughput hit | [Red Hat AI / vLLM blog, May 11 2026](https://blog.vllm.ai/2026/05/11/turboquant.html) | Reach for `--kv-cache-dtype fp8` first. Only use TurboQuant when you need >2× compression and accept some throughput cost | +| **The QJL residual step often hurts at low bit widths** — MSE-only beats MSE+QJL on every model the community has tested | [tonbistudio V3 README](https://github.com/tonbistudio/turboquant-pytorch), [scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8) | Prefer the `*_nc` ("norm correction") variants; treat QJL as optional | +| **3-bit modes are not production-ready** for reasoning or >128K retrieval — ~20-point accuracy drops on AIME25 / LiveCodeBench at long context | [Red Hat blog](https://blog.vllm.ai/2026/05/11/turboquant.html) Figures 5–6 | Use `turboquant_4bit_nc` for production, save `3bit_nc` for memory-bound edge | +| **Skip the first and last 2 layers** — they hold disproportionate signal; quantizing them costs more accuracy than the bits it saves | Red Hat blog | Verify your engine config does this (vLLM does it for `*_nc` variants by default) | +| **K/V norm ratio predicts quality** — Qwen-class models need more bits on K than V (8/4 or 6/3) | [scos-lab #8](https://github.com/tonbistudio/turboquant-pytorch/issues/8) | Profile your model before picking a uniform bit budget | +| **Layer 0 has ~20% outlier channels**, middle layers ~4-6% | scos-lab | Per-layer dynamic outlier thresholds beat a fixed global allocation | + +### Where TurboQuant still wins + +| Scenario | Why TurboQuant beats FP8 | |---|---| -| **RTX 5090 @ 32K prefill** (Qwen3.5-27B) | **1.24× faster** than FP16 — same GPU, same model, 4.9× less KV memory ([report](reports/2026-03-31-build-report.md)) | -| **RTX 5090 @ 64K+ context** | FP16 **OOMs**, TurboQuant still serves (1.5M context confirmed) | -| **RTX 5090 @ tg128 generation** | turbo2 = **71.1 tok/s**, FP16 = 70.22 tok/s — TurboQuant 2.5-bit is *faster than FP16* at decode because of the KV-cache bandwidth win | -| **Pure-PyTorch CPU demo** (random vectors, d=128) | 3.5-bit: **0.975 avg cosine**, 0.955 min — matches paper expectation ([report](reports/2026-04-17-demo-results.md)) | +| Single-card serving of 70B+ at 128K+ context | 3–4× KV reduction lets model + cache fit; FP8 only buys 2× | +| Consumer GPUs without FP8 attention (Ada, AMD older, Apple Silicon) | FP8 KV needs hardware FP8 attention to win; TurboQuant works everywhere | +| Long-context retrieval where memory bandwidth dominates | Reading ~52 bytes/token instead of 256 frees decode bandwidth | +| Edge / on-device with strict memory budgets | `4bit_nc` at <5% perplexity drop is hard to match | ## Table of contents - 🎛️ **[Live demo on HuggingFace Space](https://huggingface.co/spaces/OnlyTerp/turboquant-demo)** — see your model's KV cache shrink in real time -- [What's new — April 2026](#whats-new--april-2026) • [Why TurboQuant?](#why-turboquant) • [How it works](#how-it-works) • [Quick start](#quick-start) +- [What's new — May 2026](#whats-new--may-2026) • [Why TurboQuant?](#why-turboquant) • [How it works](#how-it-works) • [Quick start](#quick-start) +- [**Tips and tricks**](#tips-and-tricks) (new — from Red Hat eval + independent ports) - [The 2026 KV compression landscape](#the-2026-kv-compression-landscape) - [Integrations (vLLM, SGLang, llama.cpp, KVPress, LMCache, MLX)](#integrations) - [Hardware support](#hardware-support) • [Decision guide](#which-compressor-do-i-actually-want) • [FAQ](FAQ.md) - [Project structure](#project-structure) • [Citation](#citation) • [Credits](#credits--attribution) -## What's new — April 2026 +## What's new — May 2026 + +TurboQuant landed in upstream vLLM, picked up several serious independent ports, and +got its first comprehensive third-party evaluation. The headline story has shifted from +"new SOTA, use it everywhere" to "useful in a well-defined regime, with caveats." + +### The May 2026 must-reads + +- **May 11** — [**Red Hat AI / vLLM blog: "A First Comprehensive Study of TurboQuant: Accuracy and Performance"**](https://blog.vllm.ai/2026/05/11/turboquant.html) + by Eldar Kurtić, Michael Goin, Alexandre Marques. The single most important data point + in the ecosystem right now: rigorous evaluation across Llama-3.3-70B, Qwen3-30B-A3B, + MiniMax-M2.7 on long-context retrieval (MRCR) and reasoning (AIME25, GPQA, MATH500, + LiveCodeBench-v6). Conclusion: **FP8 KV is the best default**, TurboQuant `4bit_nc` is + the practical TQ variant, aggressive 3-bit modes drop accuracy meaningfully. +- **May 11–18** — Multiple independent evaluations ([TeqVolt](https://teqvolt.com/deep-dives/turboquant-google-6x-kv-cache-compression-3-bit-zero-loss), + [AI Intensify](https://ai-intensify.com/turboquant-is-compression-and-performance-worth-the-hype/)) + converge on the same finding: **PolarQuant works, QJL hurts at 3 bits.** Disabling QJL + gives better practical accuracy at the same compression ratio. Tested on PyTorch, MLX, + and Triton implementations. +- **May 13** — [varjoranta/turboquant-vllm v0.13.5](https://github.com/varjoranta/turboquant-vllm/releases/tag/v0.13.5) + adds end-to-end `Qwen3.6-35B-A3B-TQ3-native` support through vLLM 0.20.2 + FlashInfer + CUTLASS MoE. First native-packed TQ checkpoint for a frontier MoE model — ~16 GB on + disk (4.4× over BF16). +- **Apr 23** — [varjoranta v0.13.0](https://github.com/varjoranta/turboquant-vllm/releases/tag/v0.13.0) + ships block-diagonal WHT CUDA kernel: **10.1× decode speedup** on Qwen3.6-35B-A3B-TQ3. +- **Late April** — [vLLM merged](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/) + `--kv-cache-dtype turboquant_{k8v4, 4bit_nc, k3v4_nc, 3bit_nc}`. These dtype names that + shipped supersede the earlier `tq3` / `tq4` / `pq4` proposals. + +### Active community ports + +| Repo | Stars | Engine | Notable | +|---|---|---|---| +| [0xSero/turboquant](https://github.com/0xSero/turboquant) | 1.1K | vLLM + Triton | First open-source vLLM integration; RTX 5090 + 8×RTX 3090 benchmarks; asymmetric 3-bit K / 2-bit V | +| [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) | 940+ | Pure PyTorch | V3 algorithm that **drops QJL** based on community findings; documented `residual_window=0` benchmark bug + correction | +| [varjoranta/turboquant-vllm](https://github.com/varjoranta/turboquant-vllm) (`turboquant-plus-vllm` on PyPI) | 60+ | vLLM + CUDA | Fused CUDA dequant kernels, MLX/Apple Silicon port, block-diagonal WHT for partial-rotary models, bs=1 GEMV kernel, kurtosis-aware mixed precision | +| [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant) | 50+ | llama.cpp | Only llama.cpp fork that supports head_dim=256+ (Qwen3.5, Qwen3.6, Gemma 4); v1.4.2 adds MMA tensor-core acceleration (+63% TG) | +| [spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) | 300+ | llama.cpp | head_dim=128 models only; fast on Llama-3 / Qwen3-14B-class | +| [scos-lab/turboquant](https://github.com/scos-lab/turboquant) | 14 | Research | 8-model benchmark suite; per-layer outlier analysis; published the K/V-norm-ratio finding | + +### Native-packed checkpoints + +- [`varjosoft/Qwen3.6-35B-A3B-TQ3-native`](https://huggingface.co/varjosoft/Qwen3.6-35B-A3B-TQ3-native) + — pre-fused expert layout; loads via `turboquant-plus-vllm` on A100 80 GB or RTX PRO 6000. -TurboQuant went from an ICLR preprint to a genuinely viral topic in the past two weeks. -Jensen Huang spent most of GTC 2026 warning that **KV cache memory is the #1 bottleneck -for long-context inference**; TurboQuant is the most discussed answer. Highlights from -the past 72 hours: +### Earlier (still relevant) — April 2026 -- **Apr 15** — [LMCache Blog: "What is TurboQuant and why it matters for LLM inference (laymen's term)"](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) notes that "some people on X even claim it is the most significant AI breakthrough this year." -- **Apr 15** — [vLLM PR #39890](https://github.com/vllm-project/vllm/pull/39890): official `turboquant_3bit` / `turboquant_4bit` KV-cache dtypes with grouped Triton store/decode paths. +2,164 LoC. -- **Apr 15** — [Towards AI: "Running a 35B Model Locally with TurboQuant — What's actually possible right now"](https://pub.towardsai.net/running-a-35b-model-locally-with-turboquant-whats-actually-possible-right-now-1ac5327430b0) — consumer GPU walkthrough, confirms TurboQuant stacks on top of AWQ/GGUF/NVFP4. -- **Apr 14** — [arXiv 2604.13226 "KV Packet"](https://arxiv.org/html/2604.13226v1): recomputation-free cross-session KV reuse via soft-token adapters — a natural partner to TurboQuant for RAG. -- **Apr 11** — [TriAttention (arXiv 2604.04921)](https://arxiv.org/abs/2604.04921): 10.7× KV memory reduction on AIME25 32K CoT via pre-RoPE trigonometric importance scoring. Complementary to TurboQuant (selection vs precision). -- **Apr 9** — [Low-Rank KV Attention](https://fin.ai/research/low-rank-key-value-attention-reducing-kv-cache-memory-and-maintaining-head-diversity/): 45–53% architectural KV reduction with lower test loss. Multiplies with TurboQuant. -- **Apr 6** — [Adaptive KV-Quant](https://arxiv.org/abs/2604.04722): learned per-token bit-width controller for on-device LLMs — can wrap TurboQuant as a backend. -- **Apr 2** — [SGLang PR #21954](https://github.com/sgl-project/sglang/pull/21954): NVFP4 KV cache strategy abstraction on Blackwell SM100/SM120. NVFP4 is the container; TurboQuant is the encoding — they compose. -- **Mar 25** — [SGLang PR #21419](https://github.com/sgl-project/sglang/pull/21419): `--kv-cache-dtype turboquant` with fused Triton kernels. +- [LMCache blog (Apr 15)](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) — best layperson explainer of the algorithm. +- [Towards AI (Apr 15)](https://pub.towardsai.net/running-a-35b-model-locally-with-turboquant-whats-actually-possible-right-now-1ac5327430b0) — consumer-GPU walkthrough; TurboQuant stacks on top of AWQ / GGUF / NVFP4. +- [TriAttention (arXiv 2604.04921)](https://arxiv.org/abs/2604.04921) — 10.7× KV reduction on AIME25 via token selection; **stacks** with TurboQuant. +- [LRKV (Apr 9)](https://fin.ai/research/low-rank-key-value-attention-reducing-kv-cache-memory-and-maintaining-head-diversity/) — architectural 45-53% KV reduction at pretraining time. +- [Adaptive KV-Quant (arXiv 2604.04722)](https://arxiv.org/abs/2604.04722) — learned per-token bit-width for edge. +- [SGLang PR #21419](https://github.com/sgl-project/sglang/pull/21419) — first `--kv-cache-dtype turboquant` attempt; still WIP, see [INTEGRATIONS.md](INTEGRATIONS.md) for the current SGLang status. +- [SGLang PR #21954](https://github.com/sgl-project/sglang/pull/21954) — NVFP4 KV strategy abstraction on Blackwell SM100/SM120; container that TurboQuant composes with. -**ICLR 2026 poster**: [Zandieh et al., Sat Apr 25, 11:15 AM PDT](https://iclr.cc/virtual/2026/poster/10006985). +**ICLR 2026 poster (archive)**: [Zandieh et al., Apr 25 2026](https://iclr.cc/virtual/2026/poster/10006985). > 📚 For the full 2026 landscape — including side-by-side comparisons with TriAttention, > LRKV, MLA, KIVI, KVQuant, ParoQuant, NVFP4-KV, KVPress, KV Packet, SnapKV, H2O, and @@ -69,6 +125,12 @@ the past 72 hours: | KIVI | 3 | 48.50 | 0.981 | | SnapKV | — | 44.57 | 0.858 | +> ⚠️ **Independent reproductions paint a more cautious picture than the paper.** +> The Red Hat AI / vLLM evaluation in May 2026 found that 3-bit modes give meaningful +> accuracy drops on reasoning + very long context, especially without disabling QJL. Use +> 4-bit `*_nc` variants for production. See +> [BENCHMARKS.md §Independent reproductions](BENCHMARKS.md#independent-reproductions-may-2026). + ### 🔧 Our Implementation Results (Mistral-7B-Instruct-v0.3) | Mode | Logit Cosine | Top-1 Match | KV Key Cosine | KV Value Cosine | Compression | @@ -80,6 +142,11 @@ Both modes use **two independent rotations** for outlier/regular channel subsets **Rotation modes:** `rotation_mode="hadamard"` (default, O(d log d)) or `rotation_mode="dense"` (full random orthogonal via QR decomposition, O(d²)). Both satisfy P^T P = I exactly. +**QJL is optional.** Independent evaluations on PyTorch, MLX, and Triton consistently find +that the 1-bit QJL correction adds variance that softmax amplifies, hurting attention +quality at b ≤ 3. Pass `qjl_score_weight=0.0` (or set `enable_qjl=False`) to fall back to +pure MSE-only PolarQuant. See [the discussion in IMPLEMENTATION_NOTES.md §QJL Score Weight](IMPLEMENTATION_NOTES.md#qjl-score-weight). + ## How It Works TurboQuant is a two-stage vector quantizer that achieves near-optimal compression: @@ -146,23 +213,141 @@ python src/test_real_model.py **Serving engines** — see [INTEGRATIONS.md](INTEGRATIONS.md) for full setup of each: ```bash -# vLLM (our plugin) +# vLLM (merged upstream — recommended) +pip install "vllm>=0.20.2" +vllm serve meta-llama/Llama-3.3-70B-Instruct --kv-cache-dtype turboquant_4bit_nc + +# Production-grade vLLM fork with CUDA dequant kernels (10× decode speedup on MoE) +pip install turboquant-plus-vllm +vllm serve varjosoft/Qwen3.6-35B-A3B-TQ3-native --kv-cache-dtype turboquant_3bit_native + +# Our self-contained plugin (no vLLM patching, works on older vLLM) pip install -e ".[vllm]" vllm serve meta-llama/Llama-3.1-8B-Instruct --attention-backend turboquant -# vLLM (upstream PR #39890) -gh pr checkout 39890 --repo vllm-project/vllm -vllm serve meta-llama/Llama-3.1-8B-Instruct --kv-cache-dtype turboquant_3bit - -# SGLang (upstream PR #21419) +# SGLang (still WIP, multiple competing PRs — see INTEGRATIONS.md) gh pr checkout 21419 --repo sgl-project/sglang python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \ --kv-cache-dtype turboquant -# llama.cpp (closest available today — native TurboQuant not yet upstream) +# llama.cpp via AmesianX/TurboQuant (only fork with head_dim=256 support) +git clone https://github.com/AmesianX/TurboQuant && cd TurboQuant && make GGML_CUDA=1 ./llama-cli -m model.gguf -ctk q4_0 -ctv q4_0 -fa -c 131072 ``` +## Tips and tricks + +Distilled from the [Red Hat AI / vLLM evaluation (May 11, 2026)](https://blog.vllm.ai/2026/05/11/turboquant.html), +[tonbistudio V3 retrospective](https://github.com/tonbistudio/turboquant-pytorch), +[scos-lab 8-model benchmark](https://github.com/tonbistudio/turboquant-pytorch/issues/8), +and runtime data from 0xSero, varjoranta, and AmesianX. **Read this before you tune.** + +### 1. Try FP8 first + +```bash +vllm serve --kv-cache-dtype fp8 +``` + +If FP8 fits your memory budget, you are done. It's the best-quality / highest-throughput +KV format on every model the Red Hat team tested. TurboQuant only earns its keep when +you need >2× compression. + +### 2. If TurboQuant: pick `4bit_nc`, not `3bit_nc` + +```bash +vllm serve --kv-cache-dtype turboquant_4bit_nc +``` + +`4bit_nc` is the "norm correction" variant — it skips the QJL residual and uses a per-token +norm rescale instead. Across Llama-3.3-70B, Qwen3-30B-A3B, and MiniMax-M2.7, it's +within ~1 pt of FP8 on AIME25 / GPQA / MATH500 / LiveCodeBench while giving 2.6–3.1× +KV reduction. `3bit_nc` and `k3v4_nc` drop **15–25 pts** on those same benchmarks at +≥128K context. + +### 3. Disable QJL when working with the reference path + +The `qjl_score_weight=1.0` paper default is *unbiased* but high-variance. Softmax amplifies +that variance, especially on heads with concentrated attention. Independent ports — PyTorch, +MLX, Triton — consistently see MSE-only beat MSE+QJL at b ≤ 3. In code: + +```python +cache = TurboQuantCache(n_layers=..., n_heads=..., d=128, b_mse=3, + qjl_score_weight=0.0) # disable QJL +# or set enable_qjl=False in newer revisions +``` + +### 4. Skip the first and last 2 layers + +The first and last 2 layers carry disproportionate signal. The Red Hat evaluation found +that quantizing them costs more accuracy than the bits they save. vLLM does this +automatically for `*_nc` variants; if you roll your own attention backend, replicate it: + +```python +SKIP_LAYERS = set(list(range(2)) + list(range(model.config.num_hidden_layers - 2, + model.config.num_hidden_layers))) +for layer_idx in range(model.config.num_hidden_layers): + if layer_idx in SKIP_LAYERS: + # keep FP16 / FP8 KV for this layer + ... +``` + +### 5. Profile K vs V norms before picking bits + +Qwen-class models have wildly asymmetric K/V norm distributions (e.g. Qwen3.5-7B sees +a ~106× ratio between K and V channel norms; GPT-2-124M only ~6×). Models with extreme +ratios benefit from more bits on K than V. The `k8v4` and `k3v4_nc` variants exist for +exactly this reason: + +```bash +# Use more bits on K when K dominates the norm budget (Qwen-family) +vllm serve Qwen/Qwen3-30B-A3B --kv-cache-dtype turboquant_k8v4 +``` + +To profile your own model, run [`scos-lab/turboquant`](https://github.com/scos-lab/turboquant) +benchmarks and inspect the per-channel norm histogram. + +### 6. Use per-layer dynamic outlier thresholds + +A fixed "top-k=32 outlier channels" allocation wastes bits in the middle layers (only ~4-6% +of channels are actually outliers) and *under-allocates* layer 0 (~20% outliers). The +scos-lab dynamic threshold (channels with RMS > 3× median get the extra bits) closes +roughly 60% of the accuracy gap at the same average bpv. + +This is the implementation policy in our `mixed_precision=True` path; just confirm your +`MixedPrecisionConfig` is using `mode="dynamic"`, not `mode="fixed_topk"`. + +### 7. For frontier MoE models, use the native-packed checkpoint + +If you're serving Qwen3.6-35B-A3B-class models, don't quantize at runtime — use the +pre-fused `varjosoft/Qwen3.6-35B-A3B-TQ3-native` checkpoint through `turboquant-plus-vllm`. +It bakes the per-expert rotation into the weight layout so the runtime kernel never sees +an unfused MoE expert, which is what lets it hit 10.1× decode speedup over BF16. + +### 8. For llama.cpp, mind the head_dim + +[spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) +hard-codes `head_dim=128`. It works for Llama-3, Qwen2.5, Qwen3-14B and most +GQA models — but Gemma 4, Qwen3.5, and Qwen3.6 use head_dim=256, where it fails silently +with degraded output. Use [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant) +(v1.3.0+) for those models — it has a P1→P5 detection cascade that finds the real head_dim +even when `n_embd / n_head` lies (which it does on those architectures). + +### 9. Stack with token selection for long reasoning + +For >32K chain-of-thought (AIME, AIME25, math), token selection dominates precision. +[TriAttention](https://arxiv.org/abs/2604.04921), [SnapKV](https://arxiv.org/abs/2404.14469), +or NVIDIA's [KVPress](https://github.com/NVIDIA/kvpress) decide *which* tokens to retain; +TurboQuant compresses *each retained token*. The two operate on different axes — combine +them. + +### 10. Don't trust headline compression numbers without verifying + +The tonbistudio team retracted a "18/18 perfect generation at 5× compression" claim +[after discovering](https://github.com/tonbistudio/turboquant-pytorch) the eval loop +was running with `residual_window=0`, which silently disabled compression. Always log +the *actual* compressed token count after the run; needle-in-haystack at your target +context length is the only honest end-to-end test. + ## The 2026 KV compression landscape TurboQuant is the **precision** axis of KV compression. There are three axes — precision, @@ -171,7 +356,10 @@ selection, and container — and the best 2026 stacks combine all three. Summary | Method | Released | Axis | KV reduction | Quality at ratio | TQ relationship | |---|---|---|---|---|---| -| **TurboQuant** | Apr 2025 / ICLR'26 | **Precision (3.5 bpv)** | **4.9×** | **Identical FP16** (LongBench) | — | +| **FP8 KV (E4M3)** | Hopper / Blackwell HW | Hardware FP8 container | 2× | **No measurable loss** ([Red Hat eval](https://blog.vllm.ai/2026/05/11/turboquant.html)) | **Strictly better than TQ** when 2× is enough | +| **TurboQuant `4bit_nc`** | vLLM merged, May 2026 | **Precision (4 bpv, no QJL)** | 2.6–3.1× | Within ~1 pt of FP8 on long-context evals | The practical TQ mode | +| **TurboQuant `3bit_nc`** | vLLM merged, May 2026 | **Precision (3 bpv, no QJL)** | 3.5–4.0× | 15–25 pt drops on AIME25 / LCB | Edge / memory-bound only | +| TurboQuant (paper config, w/ QJL) | Apr 2025 / ICLR'26 | Precision (3.5 bpv + QJL) | 4.9× | Identical FP16 on LongBench (paper); worse in practice | The published configuration | | [TriAttention](https://arxiv.org/abs/2604.04921) | **Apr 2026** | Token selection | **10.7×** on AIME25 32K CoT | Matches Full Attn reasoning | Orthogonal — stack | | [Adaptive KV-Quant](https://arxiv.org/abs/2604.04722) | **Apr 2026** | Per-token bit-width | Variable {2, 4, 8, 16} | +8% vs static on edge | Wraps TQ as backend | | [LRKV](https://fin.ai/research/low-rank-key-value-attention-reducing-kv-cache-memory-and-maintaining-head-diversity/) | **Apr 2026** | Architectural | 45–53% vs MHA | Lower test loss vs MHA | Pretraining-time; multiplies | @@ -189,15 +377,20 @@ selection, and container — and the best 2026 stacks combine all three. Summary TurboQuant runs under every major serving engine. Concrete commands in [INTEGRATIONS.md](INTEGRATIONS.md); summary: -| Engine | Status (Apr 17, 2026) | Entry point | +| Engine | Status (May 2026) | Entry point | |---|---|---| -| **vLLM** | [PR #39890](https://github.com/vllm-project/vllm/pull/39890) (official modes) + our [`vllm_plugin/`](vllm_plugin/) | `--kv-cache-dtype turboquant_3bit` or `--attention-backend turboquant` | -| **SGLang** | [PR #21419](https://github.com/sgl-project/sglang/pull/21419) | `--kv-cache-dtype turboquant` | -| **llama.cpp** | DP4A flash-attn for quantized KV in [b8779](https://github.com/ggml-org/llama.cpp/releases/tag/b8779) (Apr 13). Native TQ path not yet upstream. | `-ctk q4_0 -ctv q4_0 -fa` (approximation) | +| **vLLM (upstream)** | **Merged** — see [docs](https://docs.vllm.ai/en/v0.20.2/api/vllm/model_executor/layers/quantization/turboquant/) | `--kv-cache-dtype turboquant_{k8v4, 4bit_nc, k3v4_nc, 3bit_nc}` | +| **vLLM (production fork)** | [`turboquant-plus-vllm` on PyPI](https://pypi.org/project/turboquant-plus-vllm/) — CUDA dequant kernels, MoE support | `vllm serve varjosoft/Qwen3.6-35B-A3B-TQ3-native --kv-cache-dtype turboquant_3bit_native` | +| **vLLM (our plugin)** | [`vllm_plugin/`](vllm_plugin/) — works on vLLM ≥ 0.4.0 without patching | `--attention-backend turboquant` | +| **vLLM (Triton reference)** | [0xSero/turboquant](https://github.com/0xSero/turboquant) — 1.1K stars, RTX 5090 + 8×RTX 3090 benchmarks | `gh repo clone 0xSero/turboquant && pip install -e .` | +| **SGLang** | WIP — competing PRs [#21419](https://github.com/sgl-project/sglang/pull/21419), [#22048](https://github.com/sgl-project/sglang/pull/22048), [#23135](https://github.com/sgl-project/sglang/pull/23135); none merged as of May 2026 | `--kv-cache-dtype turboquant` (on PR branch) | +| **llama.cpp (head_dim=128)** | [spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) | `-ctk tq3 -ctv tq3 -fa` | +| **llama.cpp (any head_dim)** | [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant) v1.4.2 — only fork supporting head_dim=256+ | `-ctk tq3 -ctv tq3 -fa` | +| **PyTorch reference (QJL-free)** | [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) V3 | `pip install turboquant-pytorch` | +| **MLX (Apple Silicon)** | [varjoranta MLX port](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) | `python -m turboquant.mlx.chat --model ` | | **NVIDIA KVPress** | 0.4.0 — framework of "press" strategies | Stack TQ under ExpectedAttention / ThinK / AdaKV | -| **LMCache** | First-class; their [Apr 15 blog post](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) is the best TurboQuant explainer | Store TQ-compressed K/V in distributed cache | -| **MLX (Apple)** | Pure-PyTorch path on MPS | `python src/demo.py` | -| **Transformers** | Monkey-patch via `past_key_values=TurboQuantCache(...)` | [`src/test_real_model.py`](src/test_real_model.py) | +| **LMCache** | First-class; [their explainer](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) is still the best layperson intro | Store TQ-compressed K/V in distributed cache | +| **Transformers (HF)** | Monkey-patch via `past_key_values=TurboQuantCache(...)` | [`src/test_real_model.py`](src/test_real_model.py) | ## Hardware support @@ -215,46 +408,53 @@ TurboQuant runs under every major serving engine. Concrete commands in ## Which compressor do I actually want? -A one-shot decision table for "I need to serve X on Y hardware, what do I set up?". Full -discussion in [LANDSCAPE_2026.md](LANDSCAPE_2026.md#decision-guide-which-compressor-do-i-actually-want). +A one-shot decision table for "I need to serve X on Y hardware, what do I set up?". +Updated May 2026 to reflect the Red Hat AI evaluation findings. Full discussion in +[LANDSCAPE_2026.md](LANDSCAPE_2026.md#decision-guide-which-compressor-do-i-actually-want). | Scenario | Start with | Add on | |---|---|---| -| Datacenter Blackwell, max throughput | NVFP4 weights + NVFP4 KV | — | -| Datacenter Hopper, long context (>128K) | FP8 weights + **TurboQuant 3.5-bit** | SnapKV / KVPress beyond 1M | -| Consumer Blackwell (RTX 5090 / RTX PRO 6000) | NVFP4 weights + **TurboQuant 3.5-bit** | — | -| Consumer Ada (RTX 4090/4080) | AWQ-INT4 + **TurboQuant 3.5-bit** | TriAttention for 32K+ CoT | -| Apple Silicon | MLX-INT4 + **TurboQuant** (CPU/MPS path) | — | -| On-device / edge | **TurboQuant 2.5-bit** or Adaptive KV-Quant | Token eviction | -| RAG, high cache reuse | **TurboQuant 3.5-bit** | KV Packet / LMCache | -| Long CoT reasoning | **TurboQuant 3.5-bit** | TriAttention | -| "Just give me something that works" | **TurboQuant 3.5-bit** | — | +| **Just give me something that works** | **FP8 KV** (`--kv-cache-dtype fp8`) | — | +| Datacenter Hopper/Blackwell, memory-constrained at long context | **FP8 KV**; switch to **`turboquant_4bit_nc`** only if FP8 doesn't fit | SnapKV / KVPress beyond 1M | +| Datacenter Blackwell, max throughput | NVFP4 weights + **FP8 KV** | — | +| Single-card 70B+ at 128K+ | **`turboquant_4bit_nc`** (FP8 won't fit) | Skip first/last 2 layers | +| Consumer Blackwell (RTX 5090 / RTX PRO 6000) | NVFP4 weights + **FP8 KV** if fits, else **`turboquant_4bit_nc`** | — | +| Consumer Ada (RTX 4090 / 4080) — no FP8 attention | AWQ-INT4 + **`turboquant_4bit_nc`** | TriAttention for 32K+ CoT | +| Apple Silicon | MLX-INT4 + **TurboQuant via [varjoranta MLX port](https://github.com/varjoranta/turboquant-vllm)** | — | +| On-device / edge with strict memory | **`turboquant_3bit_nc`** (accept the accuracy hit) or Adaptive KV-Quant | Token eviction | +| RAG, high cache reuse | **`turboquant_4bit_nc`** | KV Packet / LMCache | +| Long CoT reasoning (>32K AIME-style) | **FP8 KV** (3-bit TQ drops accuracy too hard) | TriAttention for token selection | +| Qwen3 / Qwen3.5 / Qwen3.6 (asymmetric K/V norms) | **`turboquant_k8v4`** or `k3v4_nc` | head_dim=256? use [AmesianX llama.cpp](https://github.com/AmesianX/TurboQuant) | ## FAQ Common questions and misconceptions are answered in **[FAQ.md](FAQ.md)**. Highlights: +- **"Should I use FP8 or TurboQuant?"** [FP8 is the better default in May 2026](FAQ.md#should-i-use-fp8-or-turboquant). Reach + for TurboQuant only when you need >2× compression and accept some throughput cost. +- **"Does the QJL step actually help?"** [Usually not at b ≤ 3](FAQ.md#does-the-qjl-step-actually-help). MSE-only beats + MSE+QJL on every model the community has tested. The `*_nc` variants exist for this reason. - **"Is this a replacement for AWQ / GPTQ / GGUF?"** No — TurboQuant compresses the KV cache at inference time, stacking **on top of** weight quantization. - **"Why 3.5 bits?"** It's a mode name from the paper. In practice, outlier channels get an extra MSE bit + 1-bit QJL residual; actual budget is ~3.25–4.6 bpv depending on the mode (see [BENCHMARKS.md §Current Demo Results](BENCHMARKS.md#current-demo-results)). - Paper shows 3.5-bit matches FP16 on LongBench; 2.5-bit shows marginal degradation. - **"Do I need to calibrate?"** No — TurboQuant is **data-oblivious** (random rotation + Lloyd-Max codebook are fixed at init). - **"Does it work with RoPE / GQA / MLA / FlashAttention?"** Yes to all. -- **"What's the viral 'most significant breakthrough of the year' take?"** That's from - the [LMCache blog (Apr 15, 2026)](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) - paraphrasing X/Twitter. Our read: the hype is largely earned, but TurboQuant is one of - several 2026 breakthroughs (see [LANDSCAPE_2026.md](LANDSCAPE_2026.md)). +- **"Which layers should I skip from compression?"** [The first and last 2](FAQ.md#which-layers-should-i-skip-from-compression). +- **"How do I choose K vs V bits for my model?"** [Profile the K/V norm ratio](FAQ.md#how-do-i-choose-k-vs-v-bits-for-my-model). Symmetric for Llama-class, + asymmetric (K > V) for Qwen-class. ## Limitations -- **Reference implementation** — Pure PyTorch, not optimized for production throughput. Triton kernels are experimental. +- **Reference implementation** — Pure PyTorch, not optimized for production throughput. Triton kernels are experimental. Use [varjoranta CUDA kernels](https://github.com/varjoranta/turboquant-vllm) or [0xSero Triton](https://github.com/0xSero/turboquant) for real serving. - **CPU attention is slow** — The demo runs on CPU (~25× slower than FP16). GPU kernels needed for competitive speed. -- **Mixed-precision is approximate** — Our outlier channel detection differs from the paper's theoretically optimal two-independent-instances approach (see IMPLEMENTATION_NOTES.md). -- **Tested on 2 models** — Mistral-7B-Instruct and Nemotron-Nano-4B. More model validation needed. -- **vLLM plugin is a scaffold** — Not yet tested with actual vLLM serving. +- **3-bit modes are not production-ready** — Independent evaluations consistently find unacceptable accuracy drops on reasoning + >128K retrieval. Use `4bit_nc` or FP8 for production; reserve 3-bit for edge / memory-bound deployments. +- **QJL is on by default for backward compatibility** — But you should turn it off (`qjl_score_weight=0.0`). See [tip #3](#3-disable-qjl-when-working-with-the-reference-path). +- **Mixed-precision is approximate** — Our outlier channel detection differs from the paper's theoretically optimal two-independent-instances approach (see IMPLEMENTATION_NOTES.md). Dynamic per-layer thresholds (scos-lab finding) help close the gap. +- **Tested on 2 models** — Mistral-7B-Instruct and Nemotron-Nano-4B. For frontier-model results, see [Red Hat AI eval](https://blog.vllm.ai/2026/05/11/turboquant.html), [0xSero RTX 5090 benchmarks](https://github.com/0xSero/turboquant), and [varjoranta Qwen3.6-35B-A3B results](https://github.com/varjoranta/turboquant-vllm). +- **vLLM plugin is a scaffold** — Not yet tested at scale with actual vLLM serving. For production, use upstream `--kv-cache-dtype turboquant_*` (merged) or the `turboquant-plus-vllm` PyPI package. ## Algorithm Details From 9bf0778c18b69fb5e05481a63ffe615e67eeb8ac Mon Sep 17 00:00:00 2001 From: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 25 May 2026 09:27:50 +0000 Subject: [PATCH 2/2] docs: fix link-check 404s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - varjoranta/turboquant-vllm/tree/main/mlx → repo root (the /mlx subpath doesn't exist; the Metal kernels are mentioned in the README of the parent repo). - openai/simple-evals/blob/main/mrcr_eval.py → repo root (the file path has moved upstream). Co-Authored-By: Rob --- BENCHMARKS.md | 2 +- FAQ.md | 2 +- INTEGRATIONS.md | 2 +- LANDSCAPE_2026.md | 2 +- README.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 9fd22e2..497092c 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -377,7 +377,7 @@ To reproduce the paper's claims (Llama-3.1-8B at 3.5-bit ~ FP16 LongBench), you For the **2026 evaluation methodology** (the right way to evaluate today): 1. Start with FP8 KV as your baseline. -2. Run long-context retrieval ([MRCR-8 and MRCR-16](https://github.com/openai/simple-evals/blob/main/mrcr_eval.py)) +2. Run long-context retrieval ([MRCR-8 and MRCR-16](https://github.com/openai/simple-evals)) and reasoning (AIME25 Pass@1, LiveCodeBench-v6) at your target context length. 3. Try `turboquant_4bit_nc`, not `3bit_nc` or `*_q`. 4. Skip first/last 2 layers from compression. diff --git a/FAQ.md b/FAQ.md index f531502..8841449 100644 --- a/FAQ.md +++ b/FAQ.md @@ -363,7 +363,7 @@ Concrete steps: 2. Establish a baseline with **FP16/BF16 KV** *and* **FP8 KV** at your target context length. (You need both — FP8 is what TQ has to beat, not just FP16.) 3. Run two real evals at minimum: - - **Long-context retrieval:** [MRCR](https://github.com/openai/simple-evals/blob/main/mrcr_eval.py) (multi-round needle) at 32K, 64K, 128K. The Red Hat blog uses MRCR-8 and MRCR-16. + - **Long-context retrieval:** [MRCR](https://github.com/openai/simple-evals) (multi-round needle) at 32K, 64K, 128K. The Red Hat blog uses MRCR-8 and MRCR-16. - **Reasoning:** AIME25 or LiveCodeBench-v6 Pass@1 over 32+ samples (these surface reasoning degradation that retrieval evals miss). 4. Switch to `turboquant_4bit_nc` (start here, not 3-bit) and re-run. diff --git a/INTEGRATIONS.md b/INTEGRATIONS.md index 49f8fff..830f92e 100644 --- a/INTEGRATIONS.md +++ b/INTEGRATIONS.md @@ -300,7 +300,7 @@ context-independent document caching. ## MLX (Apple Silicon) **Status (May 25, 2026):** MLX has no native FP4 tensor cores, but the -[varjoranta/turboquant-vllm MLX port](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) +[varjoranta/turboquant-vllm](https://github.com/varjoranta/turboquant-vllm) MLX port adds native Metal kernels for the FWHT rotation and 3/4-bit dequant. This is faster than the pure-PyTorch MPS path that this repo ships and is what the community converges on for M-series Macs. diff --git a/LANDSCAPE_2026.md b/LANDSCAPE_2026.md index 107fd61..59fca9e 100644 --- a/LANDSCAPE_2026.md +++ b/LANDSCAPE_2026.md @@ -398,7 +398,7 @@ See [INTEGRATIONS.md](INTEGRATIONS.md) for concrete commands and links. Summary: | **PyTorch reference (QJL-free)** | [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) V3 | `pip install turboquant-pytorch` | | **LMCache** | First-class; [explainer blog](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) | Compose TQ-compressed tensors inside LMCache cross-session cache | | **NVIDIA KVPress** | Eviction/selection framework | Use KVPress for token eviction, TQ for per-token precision | -| **MLX (Apple)** | [varjoranta Metal kernels](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) | `python -m turboquant.mlx.chat --model ` | +| **MLX (Apple)** | [varjoranta Metal kernels](https://github.com/varjoranta/turboquant-vllm) | `python -m turboquant.mlx.chat --model ` | --- diff --git a/README.md b/README.md index 0355397..cb3942a 100644 --- a/README.md +++ b/README.md @@ -387,7 +387,7 @@ TurboQuant runs under every major serving engine. Concrete commands in | **llama.cpp (head_dim=128)** | [spiritbuun/llama-cpp-turboquant-cuda](https://github.com/spiritbuun/llama-cpp-turboquant-cuda) | `-ctk tq3 -ctv tq3 -fa` | | **llama.cpp (any head_dim)** | [AmesianX/TurboQuant](https://github.com/AmesianX/TurboQuant) v1.4.2 — only fork supporting head_dim=256+ | `-ctk tq3 -ctv tq3 -fa` | | **PyTorch reference (QJL-free)** | [tonbistudio/turboquant-pytorch](https://github.com/tonbistudio/turboquant-pytorch) V3 | `pip install turboquant-pytorch` | -| **MLX (Apple Silicon)** | [varjoranta MLX port](https://github.com/varjoranta/turboquant-vllm/tree/main/mlx) | `python -m turboquant.mlx.chat --model ` | +| **MLX (Apple Silicon)** | [varjoranta MLX port](https://github.com/varjoranta/turboquant-vllm) (Metal kernels) | `python -m turboquant.mlx.chat --model ` | | **NVIDIA KVPress** | 0.4.0 — framework of "press" strategies | Stack TQ under ExpectedAttention / ThinK / AdaKV | | **LMCache** | First-class; [their explainer](https://blog.lmcache.ai/en/2026/04/15/what-is-turboquant-and-why-it-matters-for-llm-inference-in-laymens-term/) is still the best layperson intro | Store TQ-compressed K/V in distributed cache | | **Transformers (HF)** | Monkey-patch via `past_key_values=TurboQuantCache(...)` | [`src/test_real_model.py`](src/test_real_model.py) |