Q3 Roadmap

[dsikka]

Multi-node Training Support

Status: In Progress

Support multi-node training using the Mooncake backend for hidden states. This initiative also involves updating the project structure to create a dedicated folder / package for various KV Cache connectors (such as the Mooncake connector) and other modifications that could easily be substituted in during training.

Relevant PRs:

1. Refactor hidden states transfer w/ abstract backend #735
2. Add Mooncake hidden states extraction backend #836

DFlash Extension Research and Support: DSpark, Domino, and DPace

Status: In Progress

Several improvements have been proposed to extend the vanilla DFlash algorithm, most notably the DSpark algorithm (supported as of: ff71b1e) and the DPace loss function: (supported as of: #736). Q3 will include active work to support these extensions and compare their performances.

Relevant PRs / RFCs:

1. feat(dflash): add Domino causal correction head (projector_type=domino) #773
2. RFC: Ablation Study — Domino vs DSpark Performance Comparison #757
3. Add D-PACE loss implementation for D-Flash training #736
4. fix(dspark): pos0 decay from weight=0 #798
5. [Models] Add DSpark speculator (DFlash + Markov + confidence head) #677

Training Extensions and Support

Status: In Progress

1. Long Context Training Support through SP Ulysses: SP Ulysses implementation #640
2. Sliding Window Attention for Eagle3 and P-Eagle: Added sliding window attention for p-eagle and eagle3 models #684

Improved Data Generation Support

Status: In Progress

Relevant PRs / RFCs:

1. [RFC]: Use vLLM Render Endpoint in preprocessing for Tokenization Alignment #652
2. Add --render-endpoint to prepare_data.py for vLLM-delegated tokenization (#652) #665

Tool Call Improvements

1. [Dataset] [Tool call] Add Hermes function-calling dataset #771
2. [Dataset] [Tool call] Add When2Call single-turn tool-use dataset #769
3. [Regen][tool call] Support tool calls in on-policy response regeneration #750
4. refactor(regen): one dataset registry, every preset usable both on&off-policy  #777

Refactoring and Tech Debt Clean-up

Status: In Progress

With the rapid growth of the project, targeted refactoring and cleanup efforts are needed to simplify and improve areas of the codebase that have become more complex over time. Focusing on improving code organization, removing outdated or redundant implementations, and standardizing existing patterns will improve the overall quality, maintainability, and readability of the codebase.

Areas of improvements include:

• Training Args handling (currently at over 70 args)
  • feat(train): config-file-first CLI for scripts/train.py #841
• Legacy code removal (Eagle), potentially remove older converters?
• Speculators CLI support - currently only exposed through a very large training script but need an API that can be used by RHOAI / KFT within the src code
  • feat(cli): unified CLI surface #927

General Training Performance Improvements

Status: Not Yet Started

This is a general initiative to improve the overall training performance end-to-end using Speculators and vLLM, starting with benchmarking existing performance to identify bottlenecks.

• An existing bottleneck is overall support for large hidden state sizes, which can result in timeout issues during training. Potential areas to explore:
  • Reducing sizes in vLLM (projections from the larger hidden sizes should help reduce with the draft models)
  • Quantized hidden states support (e.g. FP8 connector). This will depend on the initiative to have a dedicated folder / package for various KV Cache connectors, one of which would be specific to FP8.

New Model Training and support

DFlash

1. Mistral 4: Add mistral training support #719: https://huggingface.co/RedHatAI/Mistral-Small-4-119B-2603.dflash
2. Nemotron 3 Ultra: https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-speculator.dflash
3. Nemotron 3 Super: https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-speculator.dflash

DSpark

1. Gemma 4 31B: https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dspark
2. GLM 5.2: https://huggingface.co/RedHatAI/GLM-5.2-speculator.dspark
3. Mistral 4: https://huggingface.co/RedHatAI/Mistral-Small-4-119B-2603-speculator.dspark
4. Qwen 3.6 DSpark: https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-speculator.dspark