vllm-project · Roderick-Wu · Jul 22, 2026 · Jul 23, 2026 · Jul 23, 2026 · Jul 23, 2026
@@ -0,0 +1,41 @@
+# See https://help.github.com/articles/about-codeowners/
+# for more info about CODEOWNERS file
+
+/src/speculators/ @fynnsu @shanjiaz @rahul-tuli @orestis-z
+/src/speculators/convert/ @shanjiaz @fynnsu @rahul-tuli
+/src/speculators/convert/mtp/ @rahul-tuli @fynnsu
+/src/speculators/convert/eagle/ @shanjiaz @fynnsu
+/src/speculators/data_generation/ @shanjiaz @fynnsu @rahul-tuli
+/src/speculators/models/ @fynnsu @shanjiaz @rahul-tuli @orestis-z
+/src/speculators/models/mtp/ @rahul-tuli @fynnsu
+/src/speculators/models/peagle/ @orestis-z @shanjiaz
+/src/speculators/models/eagle3/ @fynnsu @shanjiaz @orestis-z
+/src/speculators/models/dflash/ @fynnsu @shanjiaz @orestis-z
+/src/speculators/train/ @fynnsu @shanjiaz
+
+/tests/ @fynnsu @shanjiaz @rahul-tuli @orestis-z
+/tests/unit/convert/ @shanjiaz @rahul-tuli @fynnsu
+/tests/unit/models/ @fynnsu @rahul-tuli @shanjiaz
+/tests/unit/train/ @fynnsu @shanjiaz
+/tests/integration/ @fynnsu @shanjiaz @orestis-z
+/tests/e2e/ @shanjiaz @fynnsu @rahul-tuli
+/tests/e2e/regression/ @shanjiaz @fynnsu @rahul-tuli @dsikka
+/tests/e2e/smoke/ @shanjiaz @rahul-tuli @dsikka
+
+/scripts/ @fynnsu @shanjiaz @rahul-tuli @orestis-z
+
+/hs_connectors/ @fynnsu @shanjiaz
+
+/.buildkite/ @dsikka @dhuangnm @deepak-kumar-neu
+
+/.github/ @dsikka @fynnsu @shanjiaz
+
+/README.md @dsikka @fynnsu @shanjiaz @rahul-tuli
+/CONTRIBUTING.md @dsikka @fynnsu @shanjiaz @rahul-tuli
+/CODE_OF_CONDUCT.md @dsikka @fynnsu @shanjiaz @rahul-tuli
+/Makefile @dsikka @fynnsu @shanjiaz @rahul-tuli
+/mkdocs.yml @dsikka @fynnsu @shanjiaz @rahul-tuli
+/pyproject.toml @dsikka @fynnsu @shanjiaz @rahul-tuli
+/setup.py @dsikka @fynnsu @shanjiaz @rahul-tuli
+/.coderabbit.yaml @dsikka @fynnsu @shanjiaz @rahul-tuli
+
@@ -79,13 +79,28 @@ pull_request_rules:
           - quality-failed
 
 merge_protections:
+  - name: Require approval from approved reviewers list
+    description: >-
+      All pull requests must have at least one approving review from a
+      member of the approved reviewers list before merging.
+    if: []
+    success_conditions:
+      - &approved_reviewer_check
+        or:
+          - approved-reviews-by = fynnsu
+          - approved-reviews-by = shanjiaz
+          - approved-reviews-by = dsikka
+          - approved-reviews-by = rahul-tuli
+          - approved-reviews-by = orestis-z
+
   - name: Require two reviews
     description: >-
-      PRs labelled "two-reviews" must have at least two approving reviews
-      before merging.
+      PRs labelled "two-reviews" must have at least two approving reviews,
+      with at least one from the approved reviewers list, before merging.
     if:
       - label = two-reviews
     success_conditions:
       - "#approved-reviews-by >= 2"
+      - *approved_reviewer_check
 merge_protections_settings:
   reporting_method: check-runs
diff --git a/.gitignore b/.gitignore
@@ -173,4 +173,7 @@ cython_debug/
 .DS_Store
 
 # Visual Studio Code
-.vscode/
+.vscode/
+
+output
+output/*
@@ -41,11 +41,10 @@ Big updates have landed in Speculators! To get a more in-depth look, check out t
 
 Some of the exciting new features include:
 
+- **DSpark Training Algorithm**: Added support for the DSpark training algorithm, which extends DFlash's anchored-block drafting with a Markov head that conditions each draft position on the previous token within the block, plus a confidence head that predicts per-position acceptance probability. DSpark checkpoints can warm-start from existing DFlash checkpoints.
 - **P-EAGLE Training Support**: Added support for the [P-EAGLE training algorithm](https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/peagle), which extends EAGLE-3's architecture with parallel multi-token prediction via Conditional-On-Distribution (COD) sampling. Rather than generating draft tokens sequentially, P-EAGLE predicts multiple tokens in a single forward pass, reducing drafting latency. The Red Hat team published a [P-EAGLE speculator for Qwen3-8B](https://huggingface.co/RedHatAI/Qwen3-8B-speculator.peagle).
 - **MTP Finetuning Support**: Added support for finetuning the native Multi-Token Prediction (MTP) heads of models like Qwen3-Next on domain-specific data, following the [FastMTP](https://arxiv.org/abs/2509.18362) approach. Because the MTP head is small (~100M–400M params), it can be trained on pre-extracted hidden states without loading the full verifier
 - **Sliding Window Attention for DFlash and DSpark**: DFlash and DSpark speculators use sliding window attention on all draft layers by default. Use `--sliding-window` to set the window size and `--full-attention-indices` to opt specific layers into full attention. Sliding window attention reduces KV cache allocation for long-context sequences and can improve per-position acceptance rates compared to full attention.
-- **Qwen3-8B DFlash Speculator**: The RedHat team published a [DFlash speculator for Qwen3-8B](https://huggingface.co/RedHatAI/Qwen3-8B-speculator.dflash), achieving average speculative token acceptance lengths of up to 3.74 on `math_reasoning`.
-- **Gemma 4 Speculators**: The RedHat team published speculators for Gemma 4 31B-it, including both [DFlash](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dflash) and [EAGLE-3](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.eagle3) checkpoints, enabling production-grade speculative decoding for Gemma 4 models.
 - **DFlash Training Algorithm**: Added support for the DFlash training algorithm with anchored-block drafting, using auxiliary hidden states from multiple verifier layers. Includes CLI options for block size and max anchors, plus DFlash metrics, utilities, and draft model. DFlash models trained through Speculators can now run seamlessly in vLLM as of [vLLM PR #38300](https://github.com/vllm-project/vllm/pull/38300).
 - **Online Training Support**: Added support for online training using the new [vLLM hidden extraction system](https://github.com/vllm-project/vllm/pull/33736), enabling real-time hidden state generation during training without requiring separate offline data generation steps.
 
@@ -118,11 +117,16 @@ The following table summarizes the models that have been trained end-to-end by o
 <td>✅</td>
 </tr>
 <tr>
-  <td rowspan="3">Qwen3 MoE</td>
+  <td rowspan="4">Qwen3 MoE</td>
   <td>30B-Instruct</td>
   <td><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3">
       EAGLE-3
-    </a> ✅</td>
+    </a> ✅<br/><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash">DFlash</a> ✅</td>
+  <td>✅</td>
+</tr>
+<tr>
+  <td>30B</td>
+  <td><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-speculator.dflash">DFlash</a> ✅</td>
   <td>✅</td>
 </tr>
 <tr>
@@ -149,10 +153,10 @@ The following table summarizes the models that have been trained end-to-end by o
 <td>✅</td>
 </tr>
 <tr>
-<td>Mistral 3 Large</td>
-<td>675B-Instruct</td>
-<td>EAGLE-3 ⏳</td>
-<td>⏳</td>
+<td>Mistral Small 4</td>
+<td>119B</td>
+<td><a href="https://huggingface.co/RedHatAI/Mistral-Small-4-119B-2603.dflash">DFlash</a> ✅</td>
+<td>✅</td>
 </tr>
 <tr>
 <td>Gemma 4</td>
@@ -166,6 +170,18 @@ The following table summarizes the models that have been trained end-to-end by o
 <td><a href="https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3">EAGLE-3</a> ✅</td>
 <td>✅</td>
 </tr>
+<tr>
+<td>NVIDIA Nemotron 3 Ultra</td>
+<td>550B-A55B</td>
+<td><a href="https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-speculator.dflash">DFlash</a> ✅</td>
+<td>✅</td>
+</tr>
+<tr>
+<td>NVIDIA Nemotron 3 Super</td>
+<td>120B-A12B</td>
+<td><a href="https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-speculator.dflash">DFlash</a> ✅</td>
+<td>✅</td>
+</tr>
 </tbody>
 </table>
 

diff --git a/docs/cli/launch_vllm.md b/docs/cli/launch_vllm.md
@@ -20,9 +20,9 @@ python scripts/launch_vllm.py meta-llama/Llama-3.1-8B-Instruct
 
 - **`--target-layer-ids`** (int list, default: auto-select) Space-separated list of integer layer IDs from which to capture hidden states. Note: if `--include-last-layer` is enabled (default), the model's last layer will be appended to this list. Default: `[2, num_layers//2, num_layers-3]`
 
-  **Important:** If set, you must also pass the same layer ids to the training script using `--target-layer-ids`.
+  **Important:** If set, you must also pass the same layer ids to the training script using `--target-layer-ids`, **excluding** the final layer — training takes the auxiliary layers only. For the [full example](#full-example) below, that is `--target-layer-ids 5 20 40`.
 
-- **`--include-last-layer` / `--no-include-last-layer`** (flag, default: `True`) For DFlash models, append the last layer (`num_hidden_layers`) to `target_layer_ids` for verifier hidden states extraction.
+- **`--include-last-layer` / `--no-include-last-layer`** (flag, default: `True`) Append the last layer (`num_hidden_layers`) to `target_layer_ids` for verifier hidden states extraction.
 
 - **`--dry-run`** (flag) Print the command that would be executed without running it.
 

diff --git a/docs/cli/response_regeneration.md b/docs/cli/response_regeneration.md
@@ -1,6 +1,6 @@
 # response_regeneration
 
-Regenerates assistant responses in existing datasets using a vLLM-served model. Given a dataset containing user prompts (e.g., Magpie, UltraChat), this pipeline extracts the prompts, sends them to a vLLM server, and produces a new dataset with the original prompts paired with freshly generated responses from the target model. This is useful for creating training data where you want a specific model's outputs in place of the original assistant responses.
+Regenerates assistant responses in existing datasets using a vLLM-served model. Given a dataset containing conversations (e.g., Magpie, UltraChat, GSM8K), this pipeline extracts conversation turns, regenerates each assistant response turn-by-turn against the model's own prior outputs, and produces pre-tokenized training samples. For multi-turn conversations, each turn conditions on the regenerated history, producing on-policy training data.
 
 The pipeline consists of two scripts:
 
@@ -33,6 +33,10 @@ Orchestrates the entire pipeline: starts a vLLM server (with optional data/tenso
 
 - **`--tp-size`** (int) Tensor parallel size per replica (maps to vLLM's `--tensor-parallel-size`).
 
+- **`--max-model-len`** (int) Maximum model context length (passed to `vllm serve --max-model-len`).
+
+- **`--reasoning-parser`** (str) Reasoning parser for the vLLM server (passed to `vllm serve --reasoning-parser`).
+
 - **`--keep-server`** (flag) Don't stop the vLLM server after processing completes.
 
 - **`--tool-call-parser`** (str) vLLM tool-call parser (e.g. `hermes`, `llama3_json`). Adds `--enable-auto-tool-choice --tool-call-parser` to the server; required for tool-call regeneration, otherwise tool calls arrive as raw text and are not regenerated as tools.
@@ -53,13 +57,15 @@ All other arguments are passed through to `script.py`.
 
 ## script.py
 
-Extracts user prompts from a dataset, sends them to a vLLM chat completion endpoint, and writes out new prompt-response pairs with the target model's generated responses.
+Extracts conversation turns from a dataset, regenerates each assistant response turn-by-turn via a vLLM chat completion endpoint, and writes out pre-tokenized training samples with generation boundaries marked in the loss mask.
 
 ### Features
 
+- **Multi-turn support** — detects `messages`/`conversations` fields and regenerates each assistant turn against the model's own prior responses
 - **Auto-detects model** from vLLM server (no need to specify `--model`)
-- **Resume capability** to skip already-processed rows
+- **Resume capability** to skip already-processed conversations
 - **Async processing** with configurable concurrency
+- **Automatic retries** with exponential backoff on transient failures
 
 ### Basic Usage
 
@@ -95,6 +101,8 @@ python scripts/response_regeneration/script.py --dataset magpie
 
 - **`--sampling-params`** (str, default: `None`) JSON object merged into each chat-completion request, e.g. `'{"temperature": 0.6, "top_p": 0.95, "seed": 0}'`. Unset keys use the server defaults.
 
+- **`--max-retries`** (int, default: `3`) Max retry attempts per request on transient HTTP failures (408, 409, 425, 429, 5xx) with exponential backoff. Permanent errors (e.g., 400, 404) fail immediately.
+
 #### Output Arguments
 
 - **`--outfile`** (str, default: auto-generated) Output JSONL path. If not specified, auto-generated as `{dataset}_{model}.jsonl`.

diff --git a/docs/cli/train.md b/docs/cli/train.md
@@ -100,7 +100,7 @@ torchrun --standalone --nproc_per_node=4 scripts/train.py \
 
 - **`--mask-token-id`** (int, default: auto-detect) Token ID to use as mask token (for DFlash). Auto-detected if not provided.
 
-- **`--target-layer-ids`** (int list, default: auto-select) Space-separated list of layer IDs used for hidden states. Default: `[2, num_layers//2, num_layers-3]` **Must match the values used when launching vLLM if custom layers were specified.**
+- **`--target-layer-ids`** (int list, default: auto-select) Space-separated list of layer IDs for the auxiliary hidden states. Default: `[2, num_layers//2, num_layers-3]` **If custom layers were specified when launching vLLM, pass the same ids here, excluding the final layer `launch_vllm.py` appends** — that one reaches training separately as the verifier's last hidden states.
 
 ### Distributed Training Arguments
 

diff --git a/docs/index.md b/docs/index.md
@@ -81,7 +81,7 @@ The following table summarizes the models that have been trained end-to-end by o
 <tr>
 <td rowspan="3">Qwen3</td>
 <td>8B</td>
-<td><a href="https://huggingface.co/RedHatAI/Qwen3-8B-speculator.eagle3">Eagle-3</a> ✅</td>
+<td><a href="https://huggingface.co/RedHatAI/Qwen3-8B-speculator.eagle3">Eagle-3</a> ✅<br/><a href="https://huggingface.co/RedHatAI/Qwen3-8B-speculator.dflash">DFlash</a> ✅<br/><a href="https://huggingface.co/RedHatAI/Qwen3-8B-speculator.peagle">P-EAGLE</a> ✅</td>
 <td>✅</td>
 </tr>
 <tr>
@@ -108,11 +108,16 @@ The following table summarizes the models that have been trained end-to-end by o
 <td>✅</td>
 </tr>
 <tr>
-  <td rowspan="3">Qwen3 MoE</td>
+  <td rowspan="4">Qwen3 MoE</td>
   <td>30B-Instruct</td>
   <td><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3">
       Eagle-3
-    </a> ✅</td>
+    </a> ✅<br/><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash">DFlash</a> ✅</td>
+  <td>✅</td>
+</tr>
+<tr>
+  <td>30B</td>
+  <td><a href="https://huggingface.co/RedHatAI/Qwen3-30B-A3B-speculator.dflash">DFlash</a> ✅</td>
   <td>✅</td>
 </tr>
 <tr>
@@ -139,15 +144,9 @@ The following table summarizes the models that have been trained end-to-end by o
 <td>✅</td>
 </tr>
 <tr>
-<td>Mistral 3 Large</td>
-<td>675B-Instruct</td>
-<td>Eagle-3 ⏳</td>
-<td>⏳</td>
-</tr>
-<tr>
-<td>Qwen3.5</td>
-<td>9B</td>
-<td>MTP ✅</td>
+<td>Mistral Small 4</td>
+<td>119B</td>
+<td><a href="https://huggingface.co/RedHatAI/Mistral-Small-4-119B-2603.dflash">DFlash</a> ✅</td>
 <td>✅</td>
 </tr>
 <tr>
@@ -162,6 +161,18 @@ The following table summarizes the models that have been trained end-to-end by o
 <td><a href="https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3">Eagle-3</a> ✅</td>
 <td>✅</td>
 </tr>
+<tr>
+<td>NVIDIA Nemotron 3 Ultra</td>
+<td>550B-A55B</td>
+<td><a href="https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-speculator.dflash">DFlash</a> ✅</td>
+<td>✅</td>
+</tr>
+<tr>
+<td>NVIDIA Nemotron 3 Super</td>
+<td>120B-A12B</td>
+<td><a href="https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-speculator.dflash">DFlash</a> ✅</td>
+<td>✅</td>
+</tr>
 </tbody>
 </table>
 

diff --git a/docs/user_guide/tutorials/response_regeneration.md b/docs/user_guide/tutorials/response_regeneration.md
@@ -1,6 +1,6 @@
 # Response Regeneration
 
-This tutorial walks you through regenerating assistant responses in an existing dataset using a target model served by vLLM. The resulting dataset pairs the original user prompts with freshly generated responses (on-policy data), and is the recommended starting point for speculator training: the drafter learns to predict what the target model actually generates, not what the dataset's original authors wrote. Training directly on the dataset's original responses (off-policy) is a cheaper fallback, since it skips a full target-model pass over the data, but costs acceptance length at inference time.
+This tutorial walks you through regenerating assistant responses in an existing dataset using a target model served by vLLM. The resulting dataset pairs the original user prompts with freshly generated responses (on-policy data), and is the recommended starting point for speculator training: the drafter learns to predict what the target model actually generates, not what the dataset's original authors wrote. For multi-turn conversations, each assistant turn is regenerated sequentially against the model's own prior responses, keeping the entire history on-policy. Training directly on the dataset's original responses (off-policy) is a cheaper fallback, since it skips a full target-model pass over the data, but costs acceptance length at inference time.
 
 ## Overview
 
@@ -26,8 +26,8 @@ The simplest way to regenerate responses is using the `run_all.sh` script, which
 This will:
 
 1. Start a vLLM server with the specified model
-2. Extract prompts from the dataset and generate new responses
-3. Save results to a JSONL file (e.g., `magpie_Llama-3.3-70B-Instruct.jsonl`)
+2. Extract conversation turns from the dataset and regenerate assistant responses turn-by-turn
+3. Save pre-tokenized results to a JSONL file (e.g., `magpie_Llama-3.3-70B-Instruct.jsonl`)
 4. Stop the server
 
 ### Multi-GPU Configurations
@@ -48,6 +48,34 @@ For larger models, use data parallelism and/or tensor parallelism:
   --dataset magpie
 ```
 
+### Tool-Call Regeneration
+
+For tool-calling datasets (e.g. `hermes-fc`), pass the model's `--tool-call-parser` (and `--reasoning-parser`, for thinking models) so the server returns structured `tool_calls` and separated reasoning instead of plain text. Both are model-specific, so look them up in the model's [vLLM recipe](https://recipes.vllm.ai/):
+
+```bash
+# Qwen3
+./scripts/response_regeneration/run_all.sh \
+  --model "Qwen/Qwen3-8B" \
+  --tool-call-parser hermes --reasoning-parser qwen3 \
+  --dataset hermes-fc
+
+# Gemma 4
+./scripts/response_regeneration/run_all.sh \
+  --model "google/gemma-4-E2B-it" \
+  --tool-call-parser gemma4 --reasoning-parser gemma4 \
+  --dataset hermes-fc
+
+# gpt-oss (reasoning is parsed automatically)
+./scripts/response_regeneration/run_all.sh \
+  --model "openai/gpt-oss-20b" \
+  --tool-call-parser openai \
+  --dataset hermes-fc
+```
+
+This is **semi-on-policy** tool-call regeneration: the target regenerates the tool-call tokens on-policy, but tools are not executed. The *i*-th cached tool result from the source data is spliced positionally after the target's *i*-th regenerated call.
+
+**Limitation:** parallel tool calls are under development; the turn is currently truncated to the first call.
+
 ## Step 2: Verify the Output
 
 The output is a JSONL file with one pre-tokenized row per target generation. `loss_mask` is `0` over the prompt the target conditioned on and `1` over the tokens it generated, so training needs no further masking:
@@ -74,6 +102,30 @@ The output is a JSONL file with one pre-tokenized row per target generation. `lo
 
 Each assistant turn produces at least one row — and more when the target calls a tool, since every call is its own generation — so expect more lines than input conversations. `conversations` is a review-only twin of `input_ids`; training drops it.
 
+For multi-turn datasets, later turns include the regenerated history as context. For example, the second turn of the same conversation would be:
+
+```json
+{
+  "id": "conv-abc_gen1",
+  "primary_id": "conv-abc",
+  "input_ids": [151644, 872, ...],
+  "loss_mask": [0, 0, ..., 1, 1],
+  "conversations": [
+    {"role": "user", "content": "What is the capital of France?"},
+    {"role": "assistant", "content": "The capital of France is Paris."},
+    {"role": "user", "content": "What about Germany?"},
+    {"role": "assistant", "content": "The capital of Germany is Berlin."}
+  ],
+  "metadata": {
+    "idx": 0,
+    "finish_reason": "stop",
+    "is_tool_call": false,
+    "usage": {...},
+    "endpoint": "http://127.0.0.1:8000/v1/chat/completions"
+  }
+}
+```
+
 Check that the output looks correct:
 
 ```bash