Require effective GRPO updates

docs/opencode-grpo.md

@@ -103,6 +103,12 @@ TRL GRPOTrainer
     -> GRPO policy update
 ```
 
+Production uses eight OpenCode rollouts per task group, while low-cost smoke
+recipes may use two. `training_diagnostics.json` records the exact TRL recipe,
+reward variance for every group, LoRA-B update statistics, and trainer log
+history. The production recipe rejects an all-zero-variance run instead of
+publishing a no-op adapter.
+
 The BenchFlow LiteLLM proxy is invoked with
 `BENCHFLOW_CAPTURE_TOKEN_LOGPROBS=1`. It requests sampled-token logprobs from
 the chat-completions endpoint and preserves them in

docs/training-pipeline.md

@@ -247,8 +247,11 @@ evaluation health artifacts. Use a new run name when changing a recipe or task
 list.
 Incomplete strict teacher collection reuses finished `attempt-*` directories
 and continues with only tasks that still lack an eligible rollout. Resume may
-increase `[teacher].max_attempts` or `[runtime].max_completion_length`; all
-dataset, model, reward, and optimizer semantics remain immutable.
+increase `[teacher].max_attempts`, `[runtime].max_completion_length`,
+`[runtime].num_generations`, and `[grpo].generation_batch_size`, or tighten
+`[grpo].require_reward_variance`. Changing the GRPO recipe invalidates GRPO and
+downstream evaluation artifacts while preserving the verified SFT checkpoint;
+all dataset, model, reward, and optimizer semantics remain immutable.
 
 The snapshot boundary also rejects byte-equivalent task packages under
 different task IDs after normalizing the package's declared task name. This is
@@ -274,6 +277,10 @@ stall before the first tool call. OpenCode title/summary helpers are not seeded.
 GRPO rollout requests opt into sampled-token logprobs and do not receive a
 forced seed, preserving within-group reward variance without making pass-rate
 comparisons depend on uncontrolled random decoding.
+The production Qwen3.5 recipe uses eight generations per task, matching TRL's
+official default instead of the two-generation smoke setting. It records
+per-group reward ranges and LoRA-B update statistics, and fails before
+publishing a GRPO checkpoint when every complete group has zero reward variance.
 
 The evaluator itself has a real SkillsBench + Daytona canary with score `1.0`,
 complete provider telemetry, and healthy `results.jsonl` and
@@ -350,10 +357,11 @@ The Qwen3.5 recipe requires separate physical devices for the trainer and TRL
 vLLM worker. Two H100 80 GB GPUs are the initial canary topology. Exact memory
 and runtime depend on completion length, generation count, sandbox latency,
 and task trajectory length. Run the 1x1 canary before the full 2,238-task run.
-The checked-in GRPO recipe trains one same-prompt pair at a time: two
-generations, generation batch two, and no gradient accumulation. This keeps
-long OpenCode trajectories within the 80 GB trainer GPU without reducing the
-one-epoch task coverage. The bootstrap also enables PyTorch expandable CUDA
+The checked-in production GRPO recipe trains one same-prompt group at a time:
+eight generations, generation batch eight, and no gradient accumulation. The
+trainer still recomputes policy logprobs with a per-device microbatch of one to
+fit long OpenCode trajectories on the 80 GB trainer GPU. The bootstrap also
+enables PyTorch expandable CUDA
 segments to reduce allocator fragmentation, and the GRPO trainer clears cached
 CUDA allocations between steps.
 

pipelines/benchflow-task-posttrain/README.md

@@ -156,8 +156,10 @@ can be reused.
 If strict teacher coverage is incomplete, resume reuses completed attempts and
 continues only missing tasks. The retry budget may be increased without
 changing the rest of the persisted run plan.
-An interrupted GRPO stage also permits increasing only the aggregate completion
-budget, then restarts cleanly from the saved SFT checkpoint.
+An interrupted GRPO stage also permits increasing the aggregate completion
+budget, generation count, and generation batch, or enabling strict reward
+variance checks. It then restarts GRPO and downstream evaluation cleanly from
+the saved SFT checkpoint.
 
 The public OpenCode endpoint is `posttrainarena-train model-bridge`, which
 forwards to the TRL server at `TRL_VLLM_SERVER_BASE_URL`. The pipeline
@@ -182,7 +184,8 @@ runs/<run-name>/reports/score.json
 Important fields include `baseline_score`, `sft_score`, `grpo_gate_score`,
 `score_after_posttrain`, `delta_score`, `grpo_planned`, `grpo_ran`, exact task
 IDs, dataset revisions, BenchFlow commit, `grpo_run_policy`, and the recorded
-stage commands. The report also records the SFT and GRPO adapter and merged
+stage commands. The report also records `grpo_effective_update`, the compact
+reward-variance/update summary, and the SFT and GRPO adapter and merged
 checkpoint paths. A dry-run may set `grpo_planned` while leaving `grpo_ran`
 false.
 
@@ -191,6 +194,10 @@ false.
 The Qwen3.5 full recipe sets `grpo.run_policy = "always"` because GRPO is part
 of the fixed organizer recipe. The engine still supports `on_reward` for
 low-cost experiments that should skip a constant-zero reward distribution.
+That full recipe samples eight generations per task group, matching TRL's
+official default, and requires at least one group with nonzero verifier-reward
+variance before it publishes a GRPO adapter. Two-generation settings remain in
+the canary and smoke recipes for cost control.
 
 Do not use held-out eval tasks to tune this gate. Production recipes should use
 separate training, gate/development, and final evaluation lists.

pipelines/benchflow-task-posttrain/configs/qwen3.5-9b-data-agent-full.toml

@@ -18,7 +18,7 @@ task_list = "../task-lists/data-agent-eval-all.txt"
 sandbox = "docker"
 sandbox_user = "agent"
 max_completion_length = 40960
-num_generations = 2
+num_generations = 8
 
 [harness]
 agent = "opencode"
@@ -75,8 +75,9 @@ lora_r = 16
 lora_alpha = 32
 lora_dropout = 0.05
 log_completions = false
-generation_batch_size = 2
+generation_batch_size = 8
 rollout_attempts = 2
+require_reward_variance = true
 vllm_server_base_url_env = "TRL_VLLM_SERVER_BASE_URL"
 
 [tracking]

pipelines/benchflow-task-posttrain/src/posttrainarena/benchflow_pipeline/config.py

@@ -68,7 +68,7 @@ class RuntimeConfig:
     sandbox: str | None = None
     sandbox_user: str | None = "agent"
     max_completion_length: int = 2048
-    num_generations: int = 2
+    num_generations: int = 8
 
 
 @dataclass(frozen=True)
@@ -140,6 +140,7 @@ class GrpoConfig:
     log_completions: bool = False
     generation_batch_size: int | None = None
     rollout_attempts: int = 2
+    require_reward_variance: bool = False
     vllm_server_base_url_env: str = "TRL_VLLM_SERVER_BASE_URL"
 
 
@@ -266,6 +267,8 @@ def validate(self) -> None:
             )
         if not _is_positive_int(self.grpo.rollout_attempts):
             errors.append("grpo.rollout_attempts must be positive")
+        if not isinstance(self.grpo.require_reward_variance, bool):
+            errors.append("grpo.require_reward_variance must be boolean")
         if (
             not isinstance(self.grpo.vllm_server_base_url_env, str)
             or not self.grpo.vllm_server_base_url_env.strip()