HYDRA-Ω: SLOT-Optimized Parameter-Efficient Language Model (WIP)

[RAVINDRA8008]

Summary

This PR introduces HYDRA-Ω, a parameter-efficient language modeling system designed for the Parameter Golf challenge constraints (≤16MB artifact, ≤10 minute training).

The approach focuses on shifting performance gains from architecture scaling to evaluation-time optimization.

Key Components

• Transformer Backbone (11L / 512d) with efficient parameter allocation
• Full-Hessian GPTQ with mixed precision quantization (int6)
• EMA + optimized training schedule for maximum step utilization
• Score-first Test-Time Training (TTT) for adaptive refinement
• SLOT (hidden-state delta optimization) as primary performance driver

Motivation

Recent leaderboard trends suggest diminishing returns from architecture-only improvements. HYDRA-Ω instead emphasizes evaluation-time adaptation (SLOT + TTT), which has demonstrated significantly larger gains compared to incremental architectural changes.

Status

• Implementation complete
• Training runs pending compute availability
• PR submitted early to document approach and enable reproducibility

Expected Outcome

Based on component-level improvements, the system is expected to achieve competitive performance in the ~1.07-1.09 bpb range after full training and tuning.

Notes

• Strictly causal evaluation (no future token leakage)
• Fully compliant with challenge constraints
• Designed for rapid iteration once compute resources are available

[MatoTeziTanka]

Community Review — Non-record: Scylla_BH3072_GPTQ_OGD_TTT_SLOT

Compliance: FLAG — Pre-Quant TTT runs multi-epoch on val_tokens without score-first discipline

What I found in the code:

The do_score_first_ttt() function (lines 1003–1041) runs ttt_epochs=2 gradient-update epochs per chunk directly on val_tokens with SGD on unfrozen model parameters. There is no per-chunk score-first guard and no is_last_chunk flag. The eval_val_sliding call before TTT (line 1800) produces a logged baseline, but it is not a causal per-chunk gate for TTT — the TTT function itself processes all val_tokens in a flat loop without scoring each chunk first.

Per Issue #402 and Issue #677 (@valerio-oai, 2026-03-27), TTT is valid only if each token is scored BEFORE the adapter trains on it. The legal PR #1413 (dexhunter) pattern scores each chunk under torch.no_grad() before optimizer.step(), with an is_last_chunk guard. This implementation lacks both.

Additional note: The submission also contains _run_slot_pass() (line 895), an additive delta on the last hidden layer per window scored only on the stride region. This matches the scored-region SLOT pattern pending Issue #1336 — a separate HOLD concern.

BigramHash (lines 573–578) is legal — XORs adjacent input tokens, no target in the key. OnlineNgramHinter appears legal (causal, tokens added only after scoring).

Verdict: COMPLIANCE FLAG — Pre-Quant TTT without score-first discipline.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #1376 and the Pre-Quant TTT cluster. A resubmission adopting the score-first-per-chunk pattern (PR #1413) would be welcomed. The SLOT component would also need Issue #1336 resolution.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source, manually verified.