Non-record: Post-Quantization LoRA Distillation (LCQ) on PR #1855 stack, val_bpb=1.06767

[okezue]

Non-record submission

val_bpb = 1.06767 (seed 42, single-seed) | artifact 15,912,974 bytes | 8xH100 SXM | strict 600s train + eval

This is a non-record submission per README §"Non-record Submissions". It does not beat current SOTA, and is offered as documentation of a novel technique with detailed analysis of a negative result.

What is novel

Post-Quantization LoRA Distillation (LCQ): after GPTQ produces quantized weights, a small rank=4 LoRA is trained on the post-GPTQ dequantized model via KL divergence against the pre-quantization BF16 teacher logits, on TRAIN data only (FineWeb_train shards), entirely within the 10-minute training cap (GPTQ_RESERVE_SECONDS=80 cuts main training to 520s, leaving 80s for GPTQ + LCQ). The trained LoRA is held in memory across the train-to-eval boundary in the same Python process and applied at eval via the model's existing forward_ttt path with a new cu_seqlens-aware variable-length attention dispatch (the same BOS-aware masking the legal sliding window uses).

Code-level: BatchedLinearLoRA.forward extended to broadcast a bsz=1 LoRA against multi-batch eval inputs; forward_ttt, _block_with_lora, _parallel_block_with_lora extended to accept and propagate cu_seqlens, max_seqlen and dispatch to flash_attn_varlen_func accordingly; forward_ttt(..., return_logits=True) returns logits for distillation; new postquant_lora_distill(...) runs the 60s KL distillation training loop; serialize calls it after GPTQ; eval_val_sliding(..., lora=lora) accepts the trained LoRA and uses forward_ttt instead of forward_logits when set.

Why it's a negative result

stage	val_bpb
post-EMA BF16 (cut 520s training)	1.06870
quantized	1.07702
quantized + sliding window + LCQ LoRA	1.06767
(reference) sliding-only at full 600s training	1.06286

LCQ recovers about -0.0094 from quantized, which roughly matches plain sliding window (-0.0099). The actual LoRA contribution on top of plain sliding window is only about -0.0003 BPB. The 80 seconds LCQ steals from main training costs about +0.005 BPB on the BF16 model, more than LoRA recovers. Net negative versus plain sliding window at full training.

KL distillation training loss converged very low (~0.02) but the BPB delta is small, suggesting the residual quant error lives in the long tail of the next-token distribution where rank=4 LoRA has too little capacity to fix.

Possible follow-ups (not attempted here): higher-rank LoRA (16-32) with careful artifact-size accounting, temperature-scaled distillation to up-weight tail tokens, or shipping LoRA in the artifact and running LCQ inside the eval budget on val tokens after they are graded (legal score-first on val).

Compliance

• Train budget: main + LCQ within the 10-minute cap (520s + ~75s).
• Eval budget: post-quant + sliding window with LoRA, under 10 minutes.
• Artifact: 15,912,974 bytes < 16,000,000 byte cap.
• C3 score-first on val: LCQ trains exclusively on TRAIN data shards. The reported quantized_sliding_window val_bpb is single-pass causal scoring with the trained LoRA already loaded; no parameter updates driven by val tokens before scoring them.
• C1 causality: cu_seqlens-aware variable-length attention masks doc boundaries during sliding window eval (same as the legal eval_val pattern).
• No SLOT, no n-gram cache, no logit bias, no ETLB.
• 8xH100 80GB SXM.

Files

• README.md (full technique writeup)
• submission.json (structured metadata)
• train_gpt.py (LCQ implementation)
• lossless_caps.py (CaseOps tokenizer utility)
• train_seed42.log (full run log including LCQ training trace)

Credits

• PR Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 / @codemath3000 — base stack
• PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 / @bigbag, PR Record: SP8192 + QK-Gain 5 + Legal Score-First TTT — val_bpb 1.08279 (3-seed mean) #1413 / @dexhunter — legal sliding-window pattern
• PR Record: Per-Layer Adaptive GPTQ Clip + int7 Embeddings + MLR 0.026 — val_bpb 1.07493 (3-seed mean) #1586 / PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667 / PR Record: CaseOps Tokenizer + Tapered WD - val_bpb 1.0678 (3-seed mean) #1729 — pergroup lrzip compression
• PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 / @clarkkev — SP8192 + GPTQ + Muon base
• PR Non-record: Blueprint Stack + ProgSeq + Multi-scale RoPE + ByteEmbed — val_bpb 1.5568 (1xRTX 3080) #1411 line — BatchedTTTLoRA infrastructure
• LQER (Yao et al., 2024)
• GPTQ (Frantar et al., 2023)

cc @cocohearts @valerio-oai for visibility (non-record review).

[Christopher-Lee-McClendon]

Nice idea! Regarding "The trained LoRA is held in memory across the train-to-eval boundary", is there room for this LoRA in your artifact size? The model/weights learned from the training set need to fit within 16,000,000 bytes.