Thinking Reward

[suqizhang820-del]

Dear authors, we're working on reproducing the Extended RL stage from your BusterX++ paper and have run into some implementation questions regarding the Thinking Reward. Would it be possible to share a few more details?

1.Thinking Reward scoring:

• The paper mentions evaluating "coherence and completeness" – how exactly is this quantified?
• Could you share the prompt used with SophiaVL-R1-Thinking-Reward-Model-3B?
• Do you evaluate only the <think>...</think> portion or the entire response?

2.Reward combination in Extended RL:

• Is it simply: total = format + accuracy + thinking ?
• Are these three rewards summed directly, or are there weighting coefficients?

3.Edge cases:

• How do you handle cases where the model doesn't generate <think> tags or the reasoning is very short?
• Is a zero reward given, or is there some fallback scoring?

Thanks for your time and the great work!

[xiaren-qincai]

good questions~

---

1. Thinking Reward

a) This is handled directly by the reward model itself.
Reward Model is trained to assess whether the reasoning is coherent and whether it sufficiently supports the final answer.

b) For the exact prompt design, please refer:
https://huggingface.co/bunny127/SophiaVL-R1-Thinking-Reward-Model-3B

c) Reward model evaluates the entire response.

One crucial motivation is to prevent reward hacking. We observed this inconsistency can occur during RL stage 1:

<think> ..... ok, the answer is **B** </think>
<answer> A </think>

2. Yes -- just so simple

+= Soft Overlong

3. Edge Cases

a) The format reward is strict. We enforce the following pattern:

pattern = r"^<think>.*?</think>\s*<answer>.*?</answer>(?![\s\S])"

b) This is a great question.

In our experiments during RL stage 1, we observed:

• As training progresses, the model’s reasoning length first increases, then decreases again.
• This happens even without any explicit length reward or thinking reward.

After introducing RL stage 2, the model stabilizes at around 500 tokens, as Reward Model mildly prefers more detailed (but not excessively long) reasoning.

c) All DAPO features are fully enabled, including dynamic sampling, which helps stabilize training.