Non-record: Neuromodulatory Depth-Recurrent Transformer with FiLM-only TTT (WIP, val_bpb=1.3151)

[nirmathur]

Work in progress. Depth-recurrent transformer (3 shared blocks x 4 loops) with FiLM conditioning vectors inspired by cortical neuromodulation. Sliding window val_bpb = 1.3151 on 1xH100. Artifact 12.87MB (3MB headroom). FiLM-only TTT implemented but crashed on a tensor comparison bug before credits ran out -- fix identified, rerun pending. Full write-up and ablations to follow.

Summary

• Architecture: 9 physical blocks, 11 virtual layers via partial weight sharing (blocks 3-4 and 9-10 share weights)
• FiLM conditioning: Per-loop scale/shift vectors disambiguate shared block iterations (inspired by cortical neuromodulation)
• Parameters: 22.2M (vs 26.9M baseline) -- 17% fewer with better BPB
• Built on: PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 SOTA stack (LeakyReLU², Parallel Muon, XSA, BigramHash, EMA)

Results

Experiment	Config	Params	val_bpb
Baseline (PR #549)	11 unique blocks	26.9M	1.7075
Recurrence + FiLM	9 phys / 11 virtual	22.2M	1.6864
FiLM only (ablation)	11 blocks + FiLM	26.9M	1.7446

Full training (4000 iters, 1xH100): sliding window val_bpb = 1.3151, artifact = 12.87 MB

Test plan

• Rerun with TTT bug fix (id(p) instead of p not in)
• Verify on 8xH100 SXM for leaderboard-comparable timing
• Try alternative recurrence patterns (3 shared blocks, MLP-only sharing)

🤖 Generated with Claude Code

[MatoTeziTanka]

[RETRACTED 2026-04-11] — This IMPORT_FAIL was a false positive. Root cause: sibling module exists in same records/ folder; runner sys.path bug. Your code is not broken. See correction below: #1383 (comment)

---

Community Review — Non-record: Neuromodulatory Depth-Recurrent Transformer with FiLM-only TTT (WIP, val_bpb=1.3151)

Compliance: NEEDS AUTHOR ACTION — train_gpt.py fails to import on CT2038 (Python 3.10 / torch 2.10.0+cpu)

What I found: The CPU smoke test on CT2038 (proteus-engine, 128 GB RAM, Triton 3.6.0, flash_attn stub, cutlass_evt_fusion stub) failed at the import step with:

ModuleNotFoundError: No module named 'flash_attn'

A few of the common patterns I've seen for this class of error in the 2026-04-11 sweep:

• PEP 701 f-string nesting — e.g. log(f" {cat}: {", ".join(...)}") is valid Python 3.12+ but invalid Python 3.10 because the inner ", " re-enters the outer double-quote context. One-character fix: ', ' instead of ", ". See PR Record: SP8192 + Improved Parallel Residuals + Muon 0.97 + LR 0.03 + Legal TTT — val_bpb 1.07785 (3-seed mean) #1541 / Record: SP8192 + Triple Recurrence + Banking + Fused MLP + Muon 0.97 — val_bpb 1.0778 (3-seed mean) #1523 for reference.
• Missing flash_attn variants — e.g. from flash_attn_interface import flash_attn_varlen_func when the wrapper script only stubs flash_attn_func. Not a PR defect on H100s, but the eval image / CPU preflight path needs a guarded import.
• Local compiled extension — e.g. import cutlass_evt_fusion from a records/*/cutlass_evt_fusion/ subfolder that isn't on the import path at smoke time. Usually an import-order issue inside the script.
• Actual syntax error — typo, missing bracket, etc.

Recommendation: Could you run python3 -c "import py_compile; py_compile.compile('train_gpt.py')" on your records-folder train_gpt.py under Python 3.10 specifically? The eval image is Python 3.10 per Issue #17 / the README, so any parse error on 3.10 blocks the submission at import time before any of the scored-eval logic runs.

Once the parse/import issue is fixed, I'll re-run the compliance audit through the normal pipeline. No other flags identified yet because the audit halts at the import step.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): IMPORT_FAIL — ModuleNotFoundError: No module named 'flash_attn'. Classification via classify_prs.py AST-based classifier; full compliance audit deferred until the import issue is resolved. Auto-drafted from a template and spot-checked before posting.

[MatoTeziTanka]

Retraction — this IMPORT_FAIL was a flash_attn stub gap in my runner

Sorry @nirmathur, this one's on me. My CPU smoke runner already ships a stub for flash_attn_interface (so imports like from flash_attn_interface import causal_attention resolve), but it does not stub the bare flash_attn top-level package. Your records/track_non_record_16mb/2026-04-04_NeuromodulatoryRecurrence/train_gpt.py imports flash_attn directly, hit the missing stub, and the runner reported ModuleNotFoundError: No module named 'flash_attn'.

On the real eval image (8×H100 SXM Python 3.10), flash_attn is present and your import resolves correctly. The error was a CPU-preflight path gap, not a submission defect.

Your PR is not broken. I'm retracting the IMPORT_FAIL classification and adding a flash_attn stub to the runner so this doesn't hit other PRs. I'll re-queue the full compliance audit and post findings separately.

Again — sorry for the noise.

[MatoTeziTanka]

Community Review — Non-record: Neuromodulatory Depth-Recurrent Transformer with FiLM-only TTT (WIP, val_bpb=1.3151)

BPB: 1.3151 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 6633e80bcc27, file records/track_non_record_16mb/2026-04-04_NeuromodulatoryRecurrence/train_gpt.py):

The TTT path at line 1169 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=97109 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=11, vocab=1024, code=97109 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

[nirmathur]

Closing this PR. The 1xH100 SP1024 setup it was built on is no longer competitive with the current SP8192 + CaseOps + 3-layer recurrence stack, and the FiLM-only TTT idea is being superseded by a cleaner technique.

Pivoting to: porting Bae et al. 2024 (Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA, arXiv 2410.20672) onto the current SOTA stack (forking #1787). Layer-wise LoRA addresses the same disambiguation problem FiLM was attempting, but with stronger published evidence (+13.5 accuracy points vs pure weight sharing on Gemma-2B → recursive-1B ablations) and direct compatibility with the existing CaseOps + GatedAttn + Score-First TTT pipeline.

New PR will land before April 30. Apologies for the noise.