Non-Record: TTSM — Typical Ternary State-Space Model, 2.0032 bpb

[dd-dent]

TTSM: Typical Ternary State-Space Model

val_bpb: 2.0032 (seed 42)
Track: non-record, 10min/16MB
Artifact: 12,039,626 bytes
Params: 11M (7.8M ternary at 1.6 bits/param, 3.3M fp16/fp32 dynamics)
Hardware: 8×H100 SXM, 154 ms/step, 3889 steps in 600s

First ternary SSM submission. Mamba-1 selective SSM with B/C projections quantized to {-1,0,+1} via STE. Hidden state remains fp16 — protected from quantization errors at both write gate (B) and readout selector (C).

Key findings

• State is protected: ternary boundary at B/C gates, not in hidden state. DeltaNet k_t is the harder problem. mradassaad (PR Non-record: Mamba-3 Hybrid + Multi-Epoch TTT + Dynamics-Protected Quant — 1.1456 bpb (3-seed mean) #1890) independently reached the same dynamics-protection boundary.
• Reversed-scan backward: backward recurrence = forward scan reversed in time. Reuse forward Triton kernel. 15 lines, 31s→1.2s/step (26×). Generalizes to any reversible recurrence.
• NS=5 > NS=10: less-orthogonal Muon step acts as diversity regularizer under ternary STE. 52× worse orthogonality, better val_bpb.
• Overtraining degrades quality: phase transition at step ~3000, 600s budget is coincidentally near-optimal.
• Frozen conv > trained conv: +0.07 bpb. Discovered after submission run; GPU availability precluded rerun.

Compliance

• All seeds train in ≤600s
• All artifacts ≤16,000,000 bytes
• No SLOT, no pre-quant TTT, no n-gram cache

Attributions

• @thwu1 (fork base)
• @CiprianFlorin PR Record Submission: 1.1570 BPB - 73.7M Ternary U-Net + NeoMuon + 4x relu²MLP + Factored Tied Emb + Poly5 Softcap + YaRN2048 + 8192BPE + FP8QAT + Bitmask-LZMA + Stride-16 Sliding #640 (ternary QAT + Z-loss)
• Albert Gu & Tri Dao (Mamba)
• fla-org HGRN (Triton kernel, MIT)
• @raahil Shah (BigramHash)
• @clarkkev PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 (MuonEq-R)
• @mradassaad PR Non-record: Mamba-3 Hybrid SSM + SP8192 + Legal TTT — 1.1473 bpb #1644 (SSM baseline) + PR Non-record: Mamba-3 Hybrid + Multi-Epoch TTT + Dynamics-Protected Quant — 1.1456 bpb (3-seed mean) #1890 (dynamics protection convergence)
• Kosson et al. (Muon)
• @newjordan (steps > depth)
• DeepSeek-V4 (CSA m=4 convergence, NS calibration)

See records/track_non_record_16mb/2026-04-30_TTSM_TernarySSM/README.md for full writeup.