[Roadmap] On Policy Training

[WindChimeRan]

Motivation

The driver is exposure bias: off-policy multi-turn data conditions each turn on the dataset's responses rather than the target's, so every turn past the first is misaligned. Regenerating turn-by-turn against the on-policy prefix fixes this — and yields longer, more realistic context as a bonus.

Topic

Ensure no cross-instance attention leakage

• test(mtp): pin packed-batch attention to document-local (#621) #627

You Only Tokenize Once

• [RFC]: Use vLLM Render Endpoint in preprocessing for Tokenization Alignment #652 Add --render-endpoint to prepare_data.py for vLLM-delegated tokenization (#652) #665
• vllm renderer endpoint [Render][Speculator] Add return_loss_mask to render endpoint for training data generation vllm#46846
• concurrent preprocessing

Response Regeneration as default

• [Data] Response regeneration should support multi-turn [On Policy Training] Multi-Turn Response Regeneration #693.
• [Doc] Make on-policy the documented default for training. Keep off-policy as a cheaper fallback (it skips a full target-model pass over the data) with its acceptance penalty called out, so the tradeoff is explicit.
• Compatible with multi-modal
• Compatible with tool use
• Support customized sampling config
• Concurrent preprocessing