diff --git a/docs/qwen35-data-agent-e2e-canary.md b/docs/qwen35-data-agent-e2e-canary.md index 32ef35b..a33b68b 100644 --- a/docs/qwen35-data-agent-e2e-canary.md +++ b/docs/qwen35-data-agent-e2e-canary.md @@ -1,6 +1,47 @@ -# Qwen3.5 Data Agent SFT-to-GRPO canary +# Qwen3.5 Data Agent SFT-to-GRPO validation -On July 14, 2026, the Qwen3.5 organizer pipeline completed a real +This document retains the historical July 14 soccer canary and the clean +July 15, 2026 lift run that supersedes its GRPO claim boundary. + +## Clean exact-ID lift run + +The run `qwen35-9b-redwine-full-v3-main-69e37ed7` used 16 red-wine training +tasks and 14 disjoint held-out tasks on two H100 80 GB GPUs. OpenCode was the +agent harness for teacher collection, baseline/SFT/final evaluation, and every +GRPO rollout. + +The path completed with: + +- strict `16/16` verifier-approved Qwen3.5-397B-A17B teacher coverage +- 63 validated tool-calling TRL SFT rows +- one bf16 LoRA SFT epoch, loss `0.142529` +- 128 OpenCode GRPO rollouts: 16 tasks × 8 generations +- four mixed-reward groups and 30 nonzero-gradient optimizer steps +- finite GRPO loss `-0.000946` +- all 248 LoRA-B tensors updated, with no non-finite tensors +- 14/14 healthy paired baseline/final evaluation artifacts + +| Stage | Held-out pass rate | +| --- | ---: | +| Baseline | `8/14` (`57.1%`) | +| After SFT | `8/14` (`57.1%`) | +| After SFT + GRPO | `11/14` (`78.6%`) | +| Paired delta | `+3/14` (`+21.4` percentage points) | + +The final pass set was a strict superset of the baseline pass set: tasks +`0047_164_47164651_qa_1`, `0060_546_60546361_qa_2`, and +`0095_395_95395894_qa_2` improved, with zero regressions. The paired bootstrap +95% interval for pass-rate lift was `[0.0%, 42.9%]`; this is real single-run +evidence on the 14-task canary, not a competition-scale statistical claim. + +PostTrainArena commit +`cf824b214e5ae08d6fc21becbcba7aae55e5109e` produced the run. The runtime used +BenchFlow commit `6d6d2ee0965bdc7fe1e38555d1f7c4c21ee8a840`, whose OpenCode +`1.17.20` pin is tracked in BenchFlow PR #931. + +## Historical soccer canary + +On July 14, 2026, the earlier Qwen3.5 organizer pipeline completed a real eight-training-task/three-held-out-task run on two H100 80 GB GPUs with Docker and OpenCode. @@ -91,8 +132,9 @@ and adapter change therefore do not establish a valid policy update. conversational history. - Evaluation and GRPO materialization retain healthy scored retries instead of rejecting the whole task because an earlier attempt failed. -- GRPO uses one two-generation prompt pair at a time, expandable CUDA segments, - and per-step cache clearing so long trajectories fit on the 80 GB trainer GPU. +- The historical canary used two-generation prompt pairs. The production recipe + now uses eight generations per task, expandable CUDA segments, and per-step + cache clearing so long trajectories fit on the 80 GB trainer GPU. - The final Qwen3.5 recipes synchronize the pinned base checkpoint before baseline evaluation so a reused vLLM server cannot contaminate the reference. - Deterministic policy attestation compares the direct TRL server with the @@ -102,15 +144,12 @@ and adapter change therefore do not establish a valid policy update. ## Claim boundary -This run proves the Qwen3.5 model, teacher, OpenCode harness, native BenchFlow -tasks, LoRA SFT, checkpoint transport, retry handling, and final pass-rate -reporting components execute together. Because the teacher stage crossed a -recipe change during resume and the old GRPO path used mismatched reconstructed -prompt IDs, it is orchestration evidence rather than a valid reproduction of -the final training contract. - -It does **not** demonstrate model-quality lift. Held-out score remained -`1/3 -> 1/3`, and the single-run training-task check decreased from `4/8` after -SFT to `3/8` after GRPO. The next quality experiment must use a denser -domain-matched train/eval slice with more verified teacher coverage and the -corrected exact-ID rollout path. +The historical soccer run proves orchestration only. Its teacher stage crossed +a recipe change during resume and its old rollout parser reconstructed prompt +IDs incorrectly, so its `1/3 -> 1/3` result is not valid GRPO-learning +evidence. + +The clean red-wine run above closes that gap: exact served prompt IDs, +provider-sampled logprobs, OpenCode-only rollouts, finite nonzero LoRA updates, +and healthy held-out evaluation jointly establish a valid `8/14 -> 11/14` +post-training lift on the canary slice. diff --git a/docs/training-pipeline.md b/docs/training-pipeline.md index 6d0b092..04b299e 100644 --- a/docs/training-pipeline.md +++ b/docs/training-pipeline.md @@ -370,6 +370,21 @@ GPU hosts after artifacts and checkpoints are backed up. ## Historical validation evidence and limits +The clean Qwen3.5-9B run +`qwen35-9b-redwine-full-v3-main-69e37ed7` is the current end-to-end quality +proof: + +- 16/16 verified Qwen3.5-397B-A17B teacher tasks +- 63 tool-calling SFT rows and one LoRA SFT epoch +- 128 OpenCode GRPO rollouts and one LoRA GRPO epoch +- four mixed-reward groups and 30 nonzero-gradient steps +- all 248 LoRA-B tensors updated with finite values +- held-out pass rate `8/14 -> 11/14`, with zero task regressions + +This is a valid paired lift on a small canary slice. The bootstrap 95% interval +includes zero, so competition-scale claims still require the full private eval +set and repeated or larger-sample evidence. + The retained Qwen3-4B recipe mirrors a completed H100 smoke with: - 15 training tasks and two held-out eval tasks diff --git a/pipelines/benchflow-task-posttrain/README.md b/pipelines/benchflow-task-posttrain/README.md index 3a08eb3..3a52d14 100644 --- a/pipelines/benchflow-task-posttrain/README.md +++ b/pipelines/benchflow-task-posttrain/README.md @@ -202,22 +202,20 @@ the canary and smoke recipes for cost control. Do not use held-out eval tasks to tune this gate. Production recipes should use separate training, gate/development, and final evaluation lists. -## Qwen3.5 Data Agent Canary Result - -The July 14, 2026 soccer-domain canary completed the current Docker + OpenCode -path on two H100 80 GB GPUs: - -- four verifier-approved teacher trajectories from eight attempted tasks -- 30 validated TRL SFT rows and one bf16 LoRA SFT epoch -- held-out baseline and post-SFT pass rate `1/3` -- post-SFT training-task gate `4/8` -- 16 OpenCode GRPO rollouts and one LoRA GRPO epoch -- final held-out pass rate `1/3`, for delta `0.0` - -The historical adapter file changed during GRPO, but a post-run audit found -that its reconstructed prompt IDs did not match the serving endpoint. Treat it -as orchestration evidence, not valid GRPO learning evidence; the corrected -exact-ID path is awaiting a clean live rerun. +## Qwen3.5 Data Agent Lift Result + +The clean July 15, 2026 Docker + OpenCode run on two H100 80 GB GPUs completed: + +- strict `16/16` verifier-approved Qwen3.5-397B-A17B teacher coverage +- 63 validated tool-calling TRL SFT rows and one bf16 LoRA SFT epoch +- 128 OpenCode GRPO rollouts, using eight generations for each of 16 tasks +- four mixed-reward groups and 30 nonzero-gradient optimizer steps +- all 248 LoRA-B tensors updated with finite values +- held-out baseline/SFT/final pass rates `8/14`, `8/14`, and `11/14` +- paired lift `+3/14` (`+21.4` percentage points), with zero regressions + +The earlier July 14 soccer canary remains historical orchestration evidence +because it predated exact served-prompt-ID reconstruction. See [`docs/qwen35-data-agent-e2e-canary.md`](../../docs/qwen35-data-agent-e2e-canary.md) for the evidence and claim boundary.