feat(trainer): symmetric DPPO-Binary TV default loss (no KL, no advantage conditioning)

[samsja]

Summary

Companion to #2401 (IcePop). Same structural change — drop the Kimi-K2.5 KL term and the advantage-conditioned mask from the default loss, mask tokens symmetrically — but the masking criterion uses the probability difference $\pi_{\text{train}} - \pi_{\text{infer}}$ (DPPO-Binary TV, arxiv) instead of the importance ratio $\pi_{\text{train}} / \pi_{\text{infer}}$.

$$J(\theta) = \mathbb{E}\left[ \frac{1}{\sum_i |y_i|} \sum_{i,t} \mathbb{1}!\left[ \pi_{\text{train}}(y_{i,t}) - \pi_{\text{infer}}(y_{i,t}) \in [-\alpha, \beta] \right] \cdot \frac{\pi_{\text{train}}(y_{i,t})}{\pi_{\text{infer}}(y_{i,t})} \cdot \widehat{A}_{i,t} \right]$$

Tokens whose train/infer probability difference falls outside $[-\alpha, \beta]$ get zero policy-gradient weight — they are dropped, not clipped. The mask is symmetric (no longer conditioned on advantage sign). There is no separate KL penalty — the double-sided difference mask is what keeps the update inside the trust region.

Breaking config changes (trainer.loss)

Removed	Added	Default
dppo_mask_low	dppo_diff_low (α)	0.2
dppo_mask_high	dppo_diff_high (β)	0.2
kl_tau	—	(term removed)

adv_tau and teacher_tau are unchanged. mismatch_kl / is_masked* metrics are retained for observability.

Notes

• Token-level mask is detached from autograd: gradient flows through importance_ratio only for in-range tokens.
• Same overall change shape as feat(trainer): replace DPPO+KL default loss with IcePop #2401 — only the masking criterion differs (diff vs. ratio).

🤖 Generated with Claude Code