Record: PR #1105 + window attn + mixed seq_len — 1.1084 bpb (3-seed mean) 1.1084 bpb

[Gusanidas]

Based on PR #1105 (abaybektursun) with this changes:

• Causal n-gram fix (within_hint/word_hint prefix-only)
• Window attention (size=512) on layers 2,4,6,8,10 via FA3
• Mixed seq_len training: 5 GPUs at 2048x36 + 3 GPUs at 6144x10
• Train-data GPTQ calibration (14s vs 220s AR self-gen)
• Auto eval_seq_len detection from max train seq_len
• Sliding window eval at seq_len=6144, stride=128

3-seed results (sliding window bpb):
seed 1337: 1.1077
seed 42: 1.1083
seed 7: 1.1091
mean: 1.1084 (vs leader 1.1147)

It has plenty of room to be further optimized

[MatoTeziTanka]

Community Review — Record: PR #1105 + window attn + mixed seq_len — 1.1084 bpb (3-seed mean) 1.1084 bpb

Compliance: LOOKS CLEAN — pure-neural submission, no TTT/SLOT/n-gram-cache

Summary PR #1219 ("WindowAttn_MixedSeq") is a pure neural submission. No illegal techniques detected. ## Analysis ### 1. N-gram family bug check (target XOR'd into hash key) The BigramHashEmbedding.bigram_hash() (line 1310-1315) hashes only input tokens (context): xor(36313 * t[..., 1:], 27191 * t[..., :-1]). The target (y) is never incorporated into any hash key. EngramLite.forward() (line 1348-1364) similarly hashes only input_ids and padded prev_ids/pp_ids — the future target token is never part of the hash. No n-gram family bug. ### 2. Pre-Quant TTT check (multi-epoch on val_tokens without score-first) val_tokens appears only in inference-mode evaluation calls: eval_val() (line 457) and eval_val_sliding() (line 1687). Both functions run under torch.inference_mode() and model.eval(). There is no backward(), optimizer.step(), or gradient flow anywhere near val data. No test-time training of any kind is present. No Pre-Quant TTT. ### 3. Score-first TTT / is_last_chunk guard No TTT mechanism exists in this PR at all — no adapter fine-tuning, no gradient-based adaptation at inference, no is_last_chunk logic. Not applicable. ### 4. Scored-region SLOT No scored-region slot mechanism detected. The sliding window eval at line 1687 correctly uses stride-gated scoring (s = 0 if ws == 0 else max(wlen - stride, 0)) but this is standard sliding-window evaluation, not a competition slot exploit. ### 5. Architecture summary Pure transformer with: window attention (WINDOW_SIZE / WINDOW_ATTN_LAYERS), bigram/trigram hash embeddings (BigramHashEmbedding), optional EngramLite multi-head n-gram embeddings, sliding-window final eval, and post-training GPTQ-lite int6 quantization. All techniques are legal neural architecture choices. No access to future tokens, no val-data gradient...

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending the usual record-track checks (3-seed validation, under-16MB artifact cap, ≤600s train + ≤600s eval on 8×H100 SXM). No compliance flags from the audit — this looks like a clean pure-neural submission.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, cross-checked against deterministic AST classifier. If this review misread your code, please call it out so I can re-audit manually.