[Non-record] Long-Train Artifact Scaling: post-TTT BPB = 1.0399, artifact size constant across 10–60 min

[Christopher-Lee-McClendon]

[Non-record] Long-Train Artifact Scaling: Final post-TTT BPB = 1.0399, no compressibility gain

Summary

Non-record experiment studying whether longer training makes the PR #1950 recipe (INT6 GPTQ + per-group lrzip) produce smaller compressed artifacts that could free byte budget for a larger record-track model.

Result: Artifact size is essentially constant (±9 KB = 0.06%) across 10–60 minutes of training. BPB improves substantially (post-TTT: 1.06 → 1.04) but the compression pipeline has already reached its entropy floor by 10 minutes.

No ML changes from PR #1950 — identical architecture, hyperparameters, tokenizer, and scoring pipeline. Only the wallclock cap was extended.

Motivation

If a model trained for 60 minutes compresses to a significantly smaller artifact than one trained for 10 minutes, the freed bytes could fund a larger model variant (more layers, wider dim, or higher LQER rank) on the record track. This experiment tests that hypothesis.

Results

Artifact Size vs Training Duration

Minute	Steps	Artifact (bytes)	Δ vs 10 min	Notes
10	6,348	15,953,292	baseline	In-loop export
20	7,193	15,952,677	−615	In-loop export
30	7,899	15,956,638	+3,346	In-loop export
45	12,135	15,955,847	+2,555	In-loop export
final (~60 min cap)	16,001	15,944,203	−9,089	Post-stop export at 3598s¹

¹ Training stopped at 3598.25s due to GPTQ_RESERVE_SECONDS=5.5. The in-loop 60-min trigger
never fired. The final row is the standard end-of-training serialize (identical path to PR #1950).

Final Model Quality (post-stop at 3598s)

Eval	val_bpb
Pre-quantization (EMA)	1.03969
Post-INT6-GPTQ	1.04944
Post-TTT (phased score-first)	1.03988

Derived Metrics (final model only)

• Quantization tax: 0.00975 BPB
• TTT gain: 0.00956 BPB
• Artifact shrink (final vs same-run 10-min): −9,089 bytes (0.06%)
• PR Record: Compliant PR #1934 Reproduction (GPTQ_RESERVE=5.5) — val_bpb 1.06003 (3-seed) #1950 3-seed mean post-TTT: 1.06003; this run final post-TTT: 1.03988

Conclusion

The INT6 GPTQ + per-group lrzip compression reaches its entropy floor within the first 10 minutes of training. Longer training improves model quality (BPB) but does not reduce the compressed artifact size enough to justify a larger model variant. The −9 KB shrink is far below our pre-registered 300 KB threshold for recommending architectural changes.

Method

Training uses the exact PR #1950 recipe with one addition: NON_RECORD_LONGTRAIN=1 mode which:

1. Extends training to 60 min instead of 10 min
2. At milestones (10/20/30/45/60 min), pauses training and runs full GPTQ+lrzip serialize
3. Records artifact size and step count per milestone
4. After reaching the wallclock cap, runs final serialize + phased TTT eval

All checkpoint exports use proper distributed synchronization (dist.broadcast + barriers) to prevent NCCL rank desync during the 130s serialize pause.

Hardware

• 8×H100 80GB SXM (RunPod COMMUNITY, $21.52/hr)
• Seed 42, ~101 min total runtime (incl. data download + checkpoints + eval)

Compliance

• ⚠️ NOT record-track compliant — training time 3598s >> 600s budget
• ✅ Eval scoring unchanged from PR Record: Compliant PR #1934 Reproduction (GPTQ_RESERVE=5.5) — val_bpb 1.06003 (3-seed) #1950
• ✅ No PPM-D, n-gram cache, or eval-time scoring changes
• ✅ Score-first TTT (no validation tokens accessed before scoring)
• ✅ No external network calls during train/eval
• ✅ Artifact fits within 16 MB cap (15,944,203 < 16,000,000 bytes)
• ✅ No ML changes from base recipe

Related PRs

• PR Record: Compliant PR #1934 Reproduction (GPTQ_RESERVE=5.5) — val_bpb 1.06003 (3-seed) #1950: Base recipe (compliance-audited reproduction of PR Record: SP8192 CaseOps + TTT + GPTQ + LRZIP — val_bpb 1.05993 (3-seed mean) #1934)
• PR Record: SP8192 CaseOps + TTT + GPTQ + LRZIP — val_bpb 1.05993 (3-seed mean) #1934: Original record (clips=12.0, EMBED_WD=0.06, COMPRESSOR=pergroup)

Files

records/track_non_record_16mb/2026-04-30_PR1950_LongTrainArtifactScaling/
├── README.md                  # Full documentation
├── submission.json            # Experiment metadata
├── train_gpt.py               # Modified PR #1950 script (NON_RECORD_LONGTRAIN mode)
├── train.log                  # Rank-0 training log (seed 42, 8×H100)
├── pgolf_stdout.txt           # Combined stdout (launcher + training + eval)
├── notes/
│   └── IMPLEMENTATION_NOTES.md
├── scripts/
│   ├── run_longtrain_scaling.sh
│   ├── analyze_scaling.py
│   └── make_larger_variant_plan.py
└── results/
    ├── checkpoint_10min.json
    ├── checkpoint_20min.json
    ├── checkpoint_30min.json
    ├── checkpoint_45min.json
    ├── checkpoint_60min.json   # Manually created from final export metrics
    ├── scaling_results.csv
    └── experiment_summary.json