diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/README.md b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/README.md new file mode 100644 index 0000000000..4e7eb8e87c --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/README.md @@ -0,0 +1,183 @@ +# Newton-Muon × Document-Packing — Negative Result + +**Result: technique strongly REGRESSES (+0.0378 nat) under PR #1874's document-packed loader.** Filed as a non-record submission with full diagnostic logs so other competitors don't burn compute on the same dead end. + +> **What this submission is, in one paragraph.** +> An honest negative result. We tried to graft hook-based Newton–Schulz residual orthogonalization ("Newton-Muon") onto PR #1874's full stack as part of our weekend's exploration. It regressed val_bpb by +0.0378 nat in a controlled same-seed A/B. Rather than bury the result, we are filing it here with the full diagnostic logs and a root-cause analysis. The same-seed reproduction baseline — produced by the same `train_gpt.py` source with `NEWTON_MUON_ENABLED=0` — is included for direct comparison and is the same configuration as the reproduction in our companion record submission. + +--- + +## TL;DR + +| Config (seed=42, identical data and hyperparameters otherwise) | quantized + TTT-phased val_bpb | +|--------|-------------------------------:| +| PR #1874 baseline | **1.06928** | +| PR #1874 + `NEWTON_MUON_ENABLED=1` | **1.10705** | +| **Δ** | **+0.0378 nat (worse)** | + +Both runs use identical seed, dataset, batch size, step count, and hardware. The only delta is the `NEWTON_MUON_ENABLED=1` env var. + +--- + +## Root Cause + +The Newton-Muon implementation we tested uses a **forward-pre-hook on every Linear module** to accumulate a per-module integer counter `_nm_K_count` and trigger Newton–Schulz preconditioning every K-th forward pass. This is incompatible with PR #1874's training pipeline because: + +1. **Document packing produces variable `cu_seqlens` per step.** PR #1874's loader concatenates documents into mixed-length sequences and passes per-step `cu_seqlens` into FlashAttention 3. Each unique `cu_seqlens` shape is already one source of dynamo specialization. + +2. **`_nm_K_count` is a Python int attribute.** `torch._dynamo` treats integer attributes on `nn.Module` as **static** — it specializes on their value. Every step the hook does `module._nm_K_count += n`, which dynamo sees as a new value, triggering a new graph specialization. + +3. **The recompile limit is hit within ~10 steps.** On 8 ranks, each transformer block hits `config.recompile_limit=16` almost immediately. From [`train_nm_default.log`](train_nm_default.log): + + ``` + torch._dynamo hit config.recompile_limit (16) + function: 'forward' + last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn'] + ._modules['attn_gate_proj']._nm_K_count == 1474560 + ``` + +4. **Cascade failure.** Dynamo falls back to eager for the affected blocks → `fullgraph=True` is silently violated → FlashAttention 3 fused kernels stop emitting cleanly → step time inflates ~2.4× → fewer steps fit in the 600s budget → final val_bpb regresses by +0.0378 nat. + +The PyTorch hint in the warning is on point: + +> HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using `torch._dynamo.config.allow_unspec_int_on_nn_module = True`, or convert this integer into a tensor. + +We tried `allow_unspec_int_on_nn_module = True` early in development. It suppressed the recompile warnings but the underlying graph fragmentation persisted because dynamo still has to reconcile the variable-length `cu_seqlens` against a now-dynamic counter, and FA3's specialized paths get bypassed regardless. + +--- + +## Why This Is Worth Filing + +PR #1874's stack is currently the best public score on the SP8192 track in absolute terms (1.06766 single-seed; we reproduced it independently to 1.06907 single-seed and 1.06996 3-seed mean). Newton-Muon-style optimizers are a frequently-suggested next step in leaderboard discussions and in PR #1900's threads. Anyone porting Newton-Muon onto PR #1874 — or onto any base that uses a document-packing loader with variable `cu_seqlens` — will hit this same wall. + +The fix is non-trivial. It probably requires moving the K-counter and the preconditioning trigger out of the compiled region entirely (e.g. into the `optimizer.step()` boundary, not a forward hook), which is enough of a redesign that we did not attempt it within our compute budget. + +--- + +## What's Compatible + +- **Static-shape forward** (single fixed seq_len, no document packing): the hook approach likely works there because `cu_seqlens` doesn't change and `_nm_K_count` becomes the only specialization source. We did not test this, but `allow_unspec_int_on_nn_module = True` should cover it. +- **Optimizer-step-based preconditioning** (not hook-based): triggered from `optimizer.step()`, runs outside any compiled region. The suggested fix above. We did not implement it. + +--- + +## Reproduction + +Identical to the record submission's reproduction steps; the only delta is the env var: + +```bash +# Negative-result run (Newton-Muon enabled) +NEWTON_MUON_ENABLED=1 NEWTON_MUON_CAPTURE_EVERY=4 \ + SEED=42 \ + torchrun --standalone --nproc_per_node=8 train_gpt.py 2>&1 | tee train_nm_default.log + +# Baseline (PR #1874 stack, NM disabled — identical seed, identical everything else) +SEED=42 torchrun --standalone --nproc_per_node=8 train_gpt.py 2>&1 | tee train_baseline_seed42.log +``` + +The diff between the two runs is exactly one env var. + +--- + +## Files + +- `README.md` (this file) +- `submission.json` — machine-readable metadata, including the diagnostic log lines +- `train_gpt.py` — PR #1874 source with the Newton-Muon graft (134 KB unwrapped, kept readable for the root-cause analysis) +- `train_nm_default.log` — full Newton-Muon run, val_bpb 1.10705 +- `train_nm_smoke.log` — short capture run that surfaces the dynamo recompile diagnostics in detail +- `train_baseline_seed42.log` — PR #1874 baseline (NM disabled), val_bpb 1.06928. Identical seed and data — direct A/B comparison. +- `models/` — pre-trained `.int6.ptz` artifacts so a reviewer can eval-only without retraining (see "Eval-only verification" below). + +### `models/` directory + +| File | What it is | Size | Reported val_bpb | +|------|-----------|-----:|-----------------:| +| `models/nm_default.int6.ptz` | Newton-Muon enabled, full 600s training, seed=42 | 15,928,150 B | 1.10705 | +| `models/nm_smoke.int6.ptz` | Newton-Muon enabled, short 180s smoke run | 15,943,987 B | (smoke; not the headline number) | +| `models/baseline_pr1874_seed42.int6.ptz` | PR #1874 baseline, NM disabled, seed=42 (the A/B comparison artifact) | 15,921,161 B | 1.06928 | + +These are checked into the submission so any reviewer can inspect the trained artifacts directly without having to retrain. Including model artifacts is not standard practice on this leaderboard; we're including them here because the value of a negative-result submission is "anyone can verify the failure mode," and shipping the artifacts gives the reviewer two independent ways to do that (re-run the script, or eval the shipped weights). + +### How to use the shipped artifacts + +The included `train_gpt.py` does **not** ship with an explicit `EVAL_ONLY` flag — its pipeline is `train → quantize → eval` end-to-end (the int6 artifact is written to `final_model.int6.ptz` near the bottom of training, then loaded and eval'd by the same process). To eval one of the shipped `.int6.ptz` artifacts without retraining, point the script at it via the `final_model.int6.ptz` filename it expects, and use the existing `deserialize(h, device)` helper at `train_gpt.py:2139`. A minimal harness looks like: + +```python +# eval_shipped_artifact.py — sketch, not shipped +import shutil, train_gpt as TG +shutil.copy("models/nm_default.int6.ptz", "final_model.int6.ptz") +h = TG.Hyperparameters() # same env-var-driven config as training +TG.set_logging_hparams(h) +device = TG.setup_distributed() +eval_model = TG.deserialize(h, device) +TG.run_sliding_eval(eval_model, h, device) # sliding-window eval +TG.run_phased_ttt_eval(eval_model, h, device) # phased TTT eval +``` + +(Function names follow PR #1874's structure; exact entrypoints may need adjusting against the source.) For most reviewers, **re-running `train_gpt.py` from scratch on a fresh seed=42 is the simpler verification path**, since the script is already wired end-to-end and the regression is large (+0.0378 nat) and stable. The artifacts in `models/` are primarily archival evidence of the runs that produced the reported logs. + +### Direct inspection without GPUs (verified) + +The `.int6.ptz` files are produced by PR #1874's `serialize()` (in `train_gpt.py`): a torch-saved `{"w": , "m": }` dict, byte-shuffled with stride 2, then brotli-compressed. To read on CPU you need to undo those two steps in reverse: + +```python +# verified on 2026-04-28 against models/nm_default.int6.ptz +import brotli, io, torch, numpy as np + +_BSHF_MAGIC = b"BSHF" + +def _byte_unshuffle(data): # mirrors train_gpt.py:_byte_unshuffle + if len(data) < 5 or data[:4] != _BSHF_MAGIC: + return data + stride = data[4] + if stride < 2: + return data[5:] + payload = np.frombuffer(data, dtype=np.uint8, offset=5) + n = len(payload) + out = np.empty(n, dtype=np.uint8) + src_off = 0 + for pos in range(stride): + chunk_len = (n - pos + stride - 1) // stride + out[pos::stride] = payload[src_off:src_off + chunk_len] + src_off += chunk_len + return out.tobytes() + +with open("models/nm_default.int6.ptz", "rb") as f: + raw = brotli.decompress(f.read()) +state = torch.load(io.BytesIO(_byte_unshuffle(raw)), map_location="cpu", weights_only=False) + +print(list(state.keys())) # ['w', 'm'] +print(len(state["w"]), "quantized tensor entries") # 207 +print(list(state["w"].keys())[:4]) +# ['blocks.0.attn.c_q.weight.q', +# 'blocks.0.attn.c_q.weight.scale', +# 'blocks.0.attn.proj.weight.q', +# 'blocks.0.attn.proj.weight.scale'] +print(list(state["m"].items())[:1]) +# [('blocks.0.attn.c_q.weight', 'gptq (int6)')] +``` + +This is enough to confirm the artifacts are well-formed int6 GPTQ-quantized models with the expected layer structure on a laptop, no GPU required. The byte-shuffle step (`_byte_shuffle`/`_byte_unshuffle` at `train_gpt.py:1976-2002`) is part of PR #1874's compression pipeline, not something we added. + +--- + +## Compute Cost of This Negative Result + +~$12 of 8×H100 SXM time on RunPod (one full 600 s training run + one short 180 s smoke run + diagnostic capture). Posted publicly so the next person doesn't repeat it. + +--- + +## Hardware + +8 × H100 80 GB SXM (RunPod), PyTorch 2.9.1 + CUDA 12.8, FlashAttention 3 (`cu128_torch291` wheel). + +--- + +## Acknowledgements + +Thanks to **@AjAnubolu** for [PR #1874](https://github.com/openai/parameter-golf/pull/1874) (the base stack we grafted onto). Newton-Muon idea credit goes to the broader Newton–Schulz orthogonalization literature in the leaderboard discussion. + +Submitted by: +- **Saicharan Ramineni** ([@GodlyDonuts](https://github.com/GodlyDonuts)) +- csramineni@gmail.com diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/baseline_pr1874_seed42.int6.ptz b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/baseline_pr1874_seed42.int6.ptz new file mode 100644 index 0000000000..52095742e2 Binary files /dev/null and b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/baseline_pr1874_seed42.int6.ptz differ diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_default.int6.ptz b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_default.int6.ptz new file mode 100644 index 0000000000..1885059962 Binary files /dev/null and b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_default.int6.ptz differ diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_smoke.int6.ptz b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_smoke.int6.ptz new file mode 100644 index 0000000000..7c778391b2 Binary files /dev/null and b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/models/nm_smoke.int6.ptz differ diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/requirements.txt b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/requirements.txt new file mode 100644 index 0000000000..e96b4f8dd7 --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/requirements.txt @@ -0,0 +1,12 @@ +numpy +tqdm +torch +huggingface-hub +kernels +setuptools +typing-extensions==4.15.0 +datasets +tiktoken +sentencepiece +brotli +zstandard diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/submission.json b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/submission.json new file mode 100644 index 0000000000..d78f7bc4bb --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/submission.json @@ -0,0 +1,42 @@ +{ + "author": "Saicharan Ramineni", + "github_id": "GodlyDonuts", + "name": "Newton-Muon × Document-Packing — Negative Result", + "blurb": "Grafting hook-based Newton–Schulz residual orthogonalization (Newton-Muon) onto PR #1874's full stack regresses val_bpb by +0.0378 nat in a controlled same-seed A/B. Root cause is dynamo recompile fragmentation: the per-module integer counter `_nm_K_count` is mutated inside a forward-pre-hook, dynamo treats it as a static attribute, every transformer block hits config.recompile_limit=16 within ~10 steps, fullgraph compilation silently breaks, FA3 fused kernels stop emitting cleanly, and step time inflates ~2.4×. Filed as a non-record submission with full diagnostic logs so other competitors don't repeat the dead end.", + "date": "2026-04-28", + "track": "non_record_16mb", + "result": "negative", + "result_summary": { + "newton_muon_enabled_val_bpb": 1.10705440, + "pr_1874_baseline_same_seed_val_bpb": 1.06927777, + "regression_nats": 0.03777663, + "regression_direction": "Newton-Muon REGRESSES vs baseline by +0.0378 nat", + "controlled_for": "identical seed=42, identical SP8192 dataset, identical batch size, identical step count, identical hardware. Only delta is NEWTON_MUON_ENABLED=1." + }, + "root_cause_summary": "Per-module Python int `_nm_K_count` updated inside a forward-pre-hook is treated as a static attribute by torch._dynamo. It mutates every step → every block recompiles → hits config.recompile_limit=16 within ~10 steps → dynamo falls back to eager → fullgraph=True is silently violated → FA3 fused attention kernels stop emitting cleanly → ~2.4× wall-clock slowdown → fewer training steps fit in the 600s budget → final val_bpb regresses by +0.0378 nat.", + "diagnostic_evidence": { + "log_lines": [ + "torch._dynamo hit config.recompile_limit (16)", + "function: 'forward'", + "last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560" + ], + "pytorch_hint": "torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.", + "we_tried_the_hint": "Setting allow_unspec_int_on_nn_module = True suppressed the recompile warnings but the underlying graph fragmentation persisted, because dynamo still has to reconcile the variable-length cu_seqlens (from PR #1874's document-packed loader) against a now-dynamic counter, and FA3's specialized paths get bypassed regardless." + }, + "compatibility": { + "compatible_with_static_seq_len_loaders": "likely yes (untested in this submission)", + "compatible_with_pr_1874_document_packing_loader": false, + "suggested_fixes": [ + "Convert _nm_K_count to a torch.Tensor scalar.", + "Move K-accumulation outside the compiled region (e.g. trigger preconditioning from optimizer.step() rather than from a forward hook).", + "Apply preconditioning post-reduce-scatter on a non-compiled path." + ] + }, + "hardware": "8 × H100 80GB SXM (RunPod)", + "pytorch_version": "2.9.1+cu128", + "compute_spent_usd": 12, + "attribution": { + "newton_schulz_optimizer_family": "broad community / leaderboard discussion", + "base_stack_we_grafted_onto": "@AjAnubolu (PR #1874)" + } +} diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_baseline_seed42.log b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_baseline_seed42.log new file mode 100644 index 0000000000..214867f35a --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_baseline_seed42.log @@ -0,0 +1,760 @@ +W0428 09:39:54.906000 552616 torch/distributed/run.py:803] +W0428 09:39:54.906000 552616 torch/distributed/run.py:803] ***************************************** +W0428 09:39:54.906000 552616 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0428 09:39:54.906000 552616 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + beta1: 0.9 + beta2: 0.95 + compressor: brotli + data_dir: ./data/ + datasets_dir: ./data/datasets/fineweb10B_sp8192 + distributed: True + ema_decay: 0.9965 + embed_bits: 7 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.35 + eval_seq_len: 2048 + eval_stride: 64 + gate_attn_out: True + gate_attn_width: 24 + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 4.0 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 20000 + ln_scale: True + local_rank: 0 + logfile: logs/champion_3seed_42.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 3 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 600.0 + min_lr: 0.0 + mlp_clip_sigmas: 12.0 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + newton_muon_all_reduce_k: True + newton_muon_beta: 0.95 + newton_muon_capture_every: 4 + newton_muon_enabled: False + newton_muon_gamma: 0.2 + newton_muon_k_refresh: 32 + newton_muon_warmup: 100 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 8 + phased_ttt_enabled: True + phased_ttt_num_phases: 3 + phased_ttt_prefix_docs: 2000 + polar_express_ns: True + qk_gain_init: 5.25 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: champion_3seed_42 + scalar_lr: 0.02 + seed: 42 + skip_gates_enabled: True + sliding_window_enabled: False + smear_gate_enabled: True + smear_gate_width: 12 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./data/tokenizers/fineweb_8192_bpe.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: True + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 192 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192/fineweb_val_*.bin + val_loss_every: 4000 + vocab_size: 8192 + warmdown_frac: 0.75 + warmup_steps: 20 + world_size: 8 + xsa_last_n: 11 +train_shards: 80 +val_tokens: 40540160 +model_params:35946727 +gptq:reserving 4s, effective=596000ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +warmup_step: 1/20 +warmup_step: 2/20 +warmup_step: 3/20 +warmup_step: 4/20 +warmup_step: 5/20 +warmup_step: 6/20 +warmup_step: 10/20 +warmup_step: 20/20 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/20 +loop_warmup_step: 2/20 +loop_warmup_step: 3/20 +loop_warmup_step: 4/20 +loop_warmup_step: 5/20 +loop_warmup_step: 6/20 +loop_warmup_step: 10/20 +loop_warmup_step: 20/20 +0/20000 val_loss: 9.0076 val_bpb: 3.4870 +1/20000 train_loss: 9.0080 train_time: 0.0m tok/s: 12149258 +2/20000 train_loss: 12.2467 train_time: 0.0m tok/s: 11427590 +3/20000 train_loss: 11.2837 train_time: 0.0m tok/s: 10114722 +4/20000 train_loss: 9.6321 train_time: 0.0m tok/s: 9634568 +5/20000 train_loss: 8.2293 train_time: 0.0m tok/s: 9344944 +500/20000 train_loss: 3.2487 train_time: 0.8m tok/s: 8071161 +1000/20000 train_loss: 3.0106 train_time: 1.6m tok/s: 8026480 +1500/20000 train_loss: 3.0169 train_time: 2.5m tok/s: 8019256 +2000/20000 train_loss: 2.9687 train_time: 3.3m tok/s: 8019331 +layer_loop:enabled step:2127 frac:0.350 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +2500/20000 train_loss: 3.0582 train_time: 4.4m tok/s: 7475855 +3000/20000 train_loss: 2.9000 train_time: 5.6m tok/s: 7034521 +3500/20000 train_loss: 2.9632 train_time: 6.8m tok/s: 6750196 +4000/20000 train_loss: 2.8926 train_time: 8.0m tok/s: 6566613 +4000/20000 val_loss: 2.8696 val_bpb: 1.1109 +4500/20000 train_loss: 2.8410 train_time: 9.2m tok/s: 6433042 +4824/20000 val_loss: 2.7668 val_bpb: 1.0711 +stopping_early: wallclock_cap train_time: 596086ms step: 4824/20000 +peak memory allocated: 40141 MiB reserved: 44206 MiB +ema:applying EMA weights +diagnostic pre-quantization post-ema val_loss:2.76583281 val_bpb:1.07070556 eval_time:6819ms +Serialized model: 135422397 bytes +Code size (uncompressed): 134706 bytes +Code size (compressed): 33710 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians in 3.5s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + gptq (int7)+lqer_asym: tok_emb.weight + passthrough (float16): blocks.attn.attn_gate_proj.weight, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15921161 bytes +Total submission size quantized+brotli: 15954871 bytes +diagnostic quantized val_loss:2.79116183 val_bpb:1.08051090 eval_time:10643ms +ttt_lora:warming up compile (random tokens, no val data) +ttt_lora:compile warmup done (104.3s) + +beginning TTT eval timer +ttt_phased: total_docs:50000 prefix_docs:2000 suffix_docs:48000 num_phases:3 boundaries:[666, 1333, 2000] +ttp: b777/782 bl:2.7181 bb:1.0872 rl:2.7181 rb:1.0872 dl:7190-7938 gd:0 +ttp: b773/782 bl:2.6440 bb:1.0729 rl:2.6874 rb:1.0813 dl:5203-5550 gd:0 +ttp: b769/782 bl:2.7599 bb:1.0922 rl:2.7058 rb:1.0841 dl:4307-4479 gd:0 +ttpp: phase:1/3 pd:1104 gd:666 t:202.0s +tttg: c1/95 lr:0.001000 t:0.3s +tttg: c2/95 lr:0.001000 t:0.4s +tttg: c3/95 lr:0.000999 t:0.5s +tttg: c4/95 lr:0.000997 t:0.6s +tttg: c5/95 lr:0.000996 t:0.7s +tttg: c6/95 lr:0.000993 t:0.8s +tttg: c7/95 lr:0.000990 t:0.8s +tttg: c8/95 lr:0.000986 t:0.9s +tttg: c9/95 lr:0.000982 t:1.0s +tttg: c10/95 lr:0.000978 t:1.1s +tttg: c11/95 lr:0.000972 t:1.1s +tttg: c12/95 lr:0.000967 t:1.2s +tttg: c13/95 lr:0.000960 t:1.3s +tttg: c14/95 lr:0.000954 t:1.4s +tttg: c15/95 lr:0.000946 t:1.4s +tttg: c16/95 lr:0.000938 t:1.5s +tttg: c17/95 lr:0.000930 t:1.6s +tttg: c18/95 lr:0.000921 t:1.7s +tttg: c19/95 lr:0.000912 t:1.7s +tttg: c20/95 lr:0.000903 t:1.8s +tttg: c21/95 lr:0.000892 t:1.9s +tttg: c22/95 lr:0.000882 t:2.0s +tttg: c23/95 lr:0.000871 t:2.0s +tttg: c24/95 lr:0.000859 t:2.1s +tttg: c25/95 lr:0.000848 t:2.2s +tttg: c26/95 lr:0.000835 t:2.3s +tttg: c27/95 lr:0.000823 t:2.4s +tttg: c28/95 lr:0.000810 t:2.4s +tttg: c29/95 lr:0.000797 t:2.5s +tttg: c30/95 lr:0.000783 t:2.6s +tttg: c31/95 lr:0.000769 t:2.7s +tttg: c32/95 lr:0.000755 t:2.8s +tttg: c33/95 lr:0.000740 t:2.8s +tttg: c34/95 lr:0.000726 t:2.9s +tttg: c35/95 lr:0.000710 t:3.0s +tttg: c36/95 lr:0.000695 t:3.0s +tttg: c37/95 lr:0.000680 t:3.1s +tttg: c38/95 lr:0.000664 t:3.2s +tttg: c39/95 lr:0.000648 t:3.3s +tttg: c40/95 lr:0.000632 t:3.4s +tttg: c41/95 lr:0.000616 t:3.4s +tttg: c42/95 lr:0.000600 t:3.5s +tttg: c43/95 lr:0.000583 t:3.6s +tttg: c44/95 lr:0.000567 t:3.7s +tttg: c45/95 lr:0.000550 t:3.7s +tttg: c46/95 lr:0.000533 t:3.8s +tttg: c47/95 lr:0.000517 t:3.9s +tttg: c48/95 lr:0.000500 t:4.0s +tttg: c49/95 lr:0.000483 t:4.0s +tttg: c50/95 lr:0.000467 t:4.1s +tttg: c51/95 lr:0.000450 t:4.2s +tttg: c52/95 lr:0.000433 t:4.3s +tttg: c53/95 lr:0.000417 t:4.3s +tttg: c54/95 lr:0.000400 t:4.4s +tttg: c55/95 lr:0.000384 t:4.5s +tttg: c56/95 lr:0.000368 t:4.6s +tttg: c57/95 lr:0.000352 t:4.7s +tttg: c58/95 lr:0.000336 t:4.7s +tttg: c59/95 lr:0.000320 t:4.8s +tttg: c60/95 lr:0.000305 t:4.9s +tttg: c61/95 lr:0.000290 t:5.0s +tttg: c62/95 lr:0.000274 t:5.0s +tttg: c63/95 lr:0.000260 t:5.1s +tttg: c64/95 lr:0.000245 t:5.2s +tttg: c65/95 lr:0.000231 t:5.3s +tttg: c66/95 lr:0.000217 t:5.3s +tttg: c67/95 lr:0.000203 t:5.4s +tttg: c68/95 lr:0.000190 t:5.5s +tttg: c69/95 lr:0.000177 t:5.6s +tttg: c70/95 lr:0.000165 t:5.6s +tttg: c71/95 lr:0.000152 t:5.7s +tttg: c72/95 lr:0.000141 t:5.8s +tttg: c73/95 lr:0.000129 t:5.9s +tttg: c74/95 lr:0.000118 t:5.9s +tttg: c75/95 lr:0.000108 t:6.0s +tttg: c76/95 lr:0.000097 t:6.1s +tttg: c77/95 lr:0.000088 t:6.2s +tttg: c78/95 lr:0.000079 t:6.2s +tttg: c79/95 lr:0.000070 t:6.3s +tttg: c80/95 lr:0.000062 t:6.4s +tttg: c81/95 lr:0.000054 t:6.5s +tttg: c82/95 lr:0.000046 t:6.5s +tttg: c83/95 lr:0.000040 t:6.6s +tttg: c84/95 lr:0.000033 t:6.7s +tttg: c85/95 lr:0.000028 t:6.8s +tttg: c86/95 lr:0.000022 t:6.9s +tttg: c87/95 lr:0.000018 t:6.9s +tttg: c88/95 lr:0.000014 t:7.0s +tttg: c89/95 lr:0.000010 t:7.1s +tttg: c90/95 lr:0.000007 t:7.2s +tttg: c91/95 lr:0.000004 t:7.2s +tttg: c92/95 lr:0.000003 t:7.3s +tttg: c93/95 lr:0.000001 t:7.4s +tttg: c94/95 lr:0.000000 t:7.5s +ttpr: phase:1/3 t:213.7s +ttp: b760/782 bl:2.8350 bb:1.1134 rl:2.7265 rb:1.0889 dl:3255-3334 gd:0 +ttp: b754/782 bl:2.6758 bb:1.0499 rl:2.7203 rb:1.0840 dl:2839-2899 gd:0 +ttpp: phase:2/3 pd:1808 gd:1333 t:284.2s +tttg: c1/158 lr:0.001000 t:0.1s +tttg: c2/158 lr:0.001000 t:0.2s +tttg: c3/158 lr:0.001000 t:0.2s +tttg: c4/158 lr:0.000999 t:0.3s +tttg: c5/158 lr:0.000998 t:0.4s +tttg: c6/158 lr:0.000997 t:0.5s +tttg: c7/158 lr:0.000996 t:0.5s +tttg: c8/158 lr:0.000995 t:0.6s +tttg: c9/158 lr:0.000994 t:0.7s +tttg: c10/158 lr:0.000992 t:0.8s +tttg: c11/158 lr:0.000990 t:0.9s +tttg: c12/158 lr:0.000988 t:0.9s +tttg: c13/158 lr:0.000986 t:1.0s +tttg: c14/158 lr:0.000983 t:1.1s +tttg: c15/158 lr:0.000981 t:1.2s +tttg: c16/158 lr:0.000978 t:1.3s +tttg: c17/158 lr:0.000975 t:1.3s +tttg: c18/158 lr:0.000971 t:1.4s +tttg: c19/158 lr:0.000968 t:1.5s +tttg: c20/158 lr:0.000964 t:1.6s +tttg: c21/158 lr:0.000960 t:1.6s +tttg: c22/158 lr:0.000957 t:1.7s +tttg: c23/158 lr:0.000952 t:1.8s +tttg: c24/158 lr:0.000948 t:1.9s +tttg: c25/158 lr:0.000943 t:1.9s +tttg: c26/158 lr:0.000939 t:2.0s +tttg: c27/158 lr:0.000934 t:2.1s +tttg: c28/158 lr:0.000929 t:2.2s +tttg: c29/158 lr:0.000924 t:2.2s +tttg: c30/158 lr:0.000918 t:2.3s +tttg: c31/158 lr:0.000913 t:2.4s +tttg: c32/158 lr:0.000907 t:2.5s +tttg: c33/158 lr:0.000901 t:2.5s +tttg: c34/158 lr:0.000895 t:2.6s +tttg: c35/158 lr:0.000889 t:2.7s +tttg: c36/158 lr:0.000882 t:2.8s +tttg: c37/158 lr:0.000876 t:2.8s +tttg: c38/158 lr:0.000869 t:2.9s +tttg: c39/158 lr:0.000862 t:3.0s +tttg: c40/158 lr:0.000855 t:3.1s +tttg: c41/158 lr:0.000848 t:3.1s +tttg: c42/158 lr:0.000841 t:3.2s +tttg: c43/158 lr:0.000834 t:3.3s +tttg: c44/158 lr:0.000826 t:3.4s +tttg: c45/158 lr:0.000818 t:3.4s +tttg: c46/158 lr:0.000811 t:3.5s +tttg: c47/158 lr:0.000803 t:3.6s +tttg: c48/158 lr:0.000795 t:3.7s +tttg: c49/158 lr:0.000787 t:3.8s +tttg: c50/158 lr:0.000778 t:3.8s +tttg: c51/158 lr:0.000770 t:3.9s +tttg: c52/158 lr:0.000761 t:4.0s +tttg: c53/158 lr:0.000753 t:4.1s +tttg: c54/158 lr:0.000744 t:4.1s +tttg: c55/158 lr:0.000735 t:4.2s +tttg: c56/158 lr:0.000727 t:4.3s +tttg: c57/158 lr:0.000718 t:4.4s +tttg: c58/158 lr:0.000709 t:4.5s +tttg: c59/158 lr:0.000699 t:4.5s +tttg: c60/158 lr:0.000690 t:4.6s +tttg: c61/158 lr:0.000681 t:4.7s +tttg: c62/158 lr:0.000672 t:4.8s +tttg: c63/158 lr:0.000662 t:4.8s +tttg: c64/158 lr:0.000653 t:4.9s +tttg: c65/158 lr:0.000643 t:5.0s +tttg: c66/158 lr:0.000633 t:5.1s +tttg: c67/158 lr:0.000624 t:5.1s +tttg: c68/158 lr:0.000614 t:5.2s +tttg: c69/158 lr:0.000604 t:5.3s +tttg: c70/158 lr:0.000594 t:5.4s +tttg: c71/158 lr:0.000585 t:5.5s +tttg: c72/158 lr:0.000575 t:5.5s +tttg: c73/158 lr:0.000565 t:5.6s +tttg: c74/158 lr:0.000555 t:5.7s +tttg: c75/158 lr:0.000545 t:5.8s +tttg: c76/158 lr:0.000535 t:5.8s +tttg: c77/158 lr:0.000525 t:5.9s +tttg: c78/158 lr:0.000515 t:6.0s +tttg: c79/158 lr:0.000505 t:6.1s +tttg: c80/158 lr:0.000495 t:6.2s +tttg: c81/158 lr:0.000485 t:6.2s +tttg: c82/158 lr:0.000475 t:6.3s +tttg: c83/158 lr:0.000465 t:6.4s +tttg: c84/158 lr:0.000455 t:6.5s +tttg: c85/158 lr:0.000445 t:6.5s +tttg: c86/158 lr:0.000435 t:6.6s +tttg: c87/158 lr:0.000425 t:6.7s +tttg: c88/158 lr:0.000415 t:6.8s +tttg: c89/158 lr:0.000406 t:6.8s +tttg: c90/158 lr:0.000396 t:6.9s +tttg: c91/158 lr:0.000386 t:7.0s +tttg: c92/158 lr:0.000376 t:7.1s +tttg: c93/158 lr:0.000367 t:7.1s +tttg: c94/158 lr:0.000357 t:7.2s +tttg: c95/158 lr:0.000347 t:7.3s +tttg: c96/158 lr:0.000338 t:7.4s +tttg: c97/158 lr:0.000328 t:7.5s +tttg: c98/158 lr:0.000319 t:7.5s +tttg: c99/158 lr:0.000310 t:7.6s +tttg: c100/158 lr:0.000301 t:7.7s +tttg: c101/158 lr:0.000291 t:7.8s +tttg: c102/158 lr:0.000282 t:7.9s +tttg: c103/158 lr:0.000273 t:7.9s +tttg: c104/158 lr:0.000265 t:8.0s +tttg: c105/158 lr:0.000256 t:8.1s +tttg: c106/158 lr:0.000247 t:8.2s +tttg: c107/158 lr:0.000239 t:8.2s +tttg: c108/158 lr:0.000230 t:8.3s +tttg: c109/158 lr:0.000222 t:8.4s +tttg: c110/158 lr:0.000213 t:8.5s +tttg: c111/158 lr:0.000205 t:8.5s +tttg: c112/158 lr:0.000197 t:8.6s +tttg: c113/158 lr:0.000189 t:8.7s +tttg: c114/158 lr:0.000182 t:8.8s +tttg: c115/158 lr:0.000174 t:8.8s +tttg: c116/158 lr:0.000166 t:8.9s +tttg: c117/158 lr:0.000159 t:9.0s +tttg: c118/158 lr:0.000152 t:9.1s +tttg: c119/158 lr:0.000145 t:9.1s +tttg: c120/158 lr:0.000138 t:9.2s +tttg: c121/158 lr:0.000131 t:9.3s +tttg: c122/158 lr:0.000124 t:9.4s +tttg: c123/158 lr:0.000118 t:9.4s +tttg: c124/158 lr:0.000111 t:9.5s +tttg: c125/158 lr:0.000105 t:9.6s +tttg: c126/158 lr:0.000099 t:9.7s +tttg: c127/158 lr:0.000093 t:9.8s +tttg: c128/158 lr:0.000087 t:9.8s +tttg: c129/158 lr:0.000082 t:9.9s +tttg: c130/158 lr:0.000076 t:10.0s +tttg: c131/158 lr:0.000071 t:10.1s +tttg: c132/158 lr:0.000066 t:10.1s +tttg: c133/158 lr:0.000061 t:10.2s +tttg: c134/158 lr:0.000057 t:10.3s +tttg: c135/158 lr:0.000052 t:10.4s +tttg: c136/158 lr:0.000048 t:10.5s +tttg: c137/158 lr:0.000043 t:10.6s +tttg: c138/158 lr:0.000040 t:10.6s +tttg: c139/158 lr:0.000036 t:10.7s +tttg: c140/158 lr:0.000032 t:10.8s +tttg: c141/158 lr:0.000029 t:10.9s +tttg: c142/158 lr:0.000025 t:10.9s +tttg: c143/158 lr:0.000022 t:11.0s +tttg: c144/158 lr:0.000019 t:11.1s +tttg: c145/158 lr:0.000017 t:11.2s +tttg: c146/158 lr:0.000014 t:11.2s +tttg: c147/158 lr:0.000012 t:11.3s +tttg: c148/158 lr:0.000010 t:11.4s +tttg: c149/158 lr:0.000008 t:11.5s +tttg: c150/158 lr:0.000006 t:11.5s +tttg: c151/158 lr:0.000005 t:11.6s +tttg: c152/158 lr:0.000004 t:11.7s +tttg: c153/158 lr:0.000003 t:11.8s +tttg: c154/158 lr:0.000002 t:11.8s +tttg: c155/158 lr:0.000001 t:11.9s +tttg: c156/158 lr:0.000000 t:12.0s +tttg: c157/158 lr:0.000000 t:12.1s +ttpr: phase:2/3 t:300.5s +ttp: b751/782 bl:2.7826 bb:1.0682 rl:2.7268 rb:1.0823 dl:2689-2740 gd:0 +ttpp: phase:3/3 pd:2448 gd:2000 t:315.7s +tttg: c1/213 lr:0.001000 t:0.1s +tttg: c2/213 lr:0.001000 t:0.2s +tttg: c3/213 lr:0.001000 t:0.2s +tttg: c4/213 lr:0.001000 t:0.3s +tttg: c5/213 lr:0.000999 t:0.4s +tttg: c6/213 lr:0.000999 t:0.5s +tttg: c7/213 lr:0.000998 t:0.5s +tttg: c8/213 lr:0.000997 t:0.6s +tttg: c9/213 lr:0.000996 t:0.7s +tttg: c10/213 lr:0.000996 t:0.8s +tttg: c11/213 lr:0.000995 t:0.8s +tttg: c12/213 lr:0.000993 t:0.9s +tttg: c13/213 lr:0.000992 t:1.0s +tttg: c14/213 lr:0.000991 t:1.1s +tttg: c15/213 lr:0.000989 t:1.2s +tttg: c16/213 lr:0.000988 t:1.2s +tttg: c17/213 lr:0.000986 t:1.3s +tttg: c18/213 lr:0.000984 t:1.4s +tttg: c19/213 lr:0.000982 t:1.5s +tttg: c20/213 lr:0.000980 t:1.5s +tttg: c21/213 lr:0.000978 t:1.6s +tttg: c22/213 lr:0.000976 t:1.7s +tttg: c23/213 lr:0.000974 t:1.8s +tttg: c24/213 lr:0.000971 t:1.8s +tttg: c25/213 lr:0.000969 t:1.9s +tttg: c26/213 lr:0.000966 t:2.0s +tttg: c27/213 lr:0.000963 t:2.1s +tttg: c28/213 lr:0.000961 t:2.1s +tttg: c29/213 lr:0.000958 t:2.2s +tttg: c30/213 lr:0.000955 t:2.3s +tttg: c31/213 lr:0.000951 t:2.4s +tttg: c32/213 lr:0.000948 t:2.5s +tttg: c33/213 lr:0.000945 t:2.5s +tttg: c34/213 lr:0.000941 t:2.6s +tttg: c35/213 lr:0.000938 t:2.7s +tttg: c36/213 lr:0.000934 t:2.7s +tttg: c37/213 lr:0.000931 t:2.8s +tttg: c38/213 lr:0.000927 t:2.9s +tttg: c39/213 lr:0.000923 t:3.0s +tttg: c40/213 lr:0.000919 t:3.1s +tttg: c41/213 lr:0.000915 t:3.1s +tttg: c42/213 lr:0.000911 t:3.2s +tttg: c43/213 lr:0.000906 t:3.3s +tttg: c44/213 lr:0.000902 t:3.4s +tttg: c45/213 lr:0.000897 t:3.4s +tttg: c46/213 lr:0.000893 t:3.5s +tttg: c47/213 lr:0.000888 t:3.6s +tttg: c48/213 lr:0.000884 t:3.7s +tttg: c49/213 lr:0.000879 t:3.7s +tttg: c50/213 lr:0.000874 t:3.8s +tttg: c51/213 lr:0.000869 t:3.9s +tttg: c52/213 lr:0.000864 t:4.0s +tttg: c53/213 lr:0.000859 t:4.0s +tttg: c54/213 lr:0.000854 t:4.1s +tttg: c55/213 lr:0.000848 t:4.2s +tttg: c56/213 lr:0.000843 t:4.3s +tttg: c57/213 lr:0.000837 t:4.3s +tttg: c58/213 lr:0.000832 t:4.4s +tttg: c59/213 lr:0.000826 t:4.5s +tttg: c60/213 lr:0.000821 t:4.6s +tttg: c61/213 lr:0.000815 t:4.7s +tttg: c62/213 lr:0.000809 t:4.7s +tttg: c63/213 lr:0.000803 t:4.8s +tttg: c64/213 lr:0.000797 t:4.9s +tttg: c65/213 lr:0.000791 t:5.0s +tttg: c66/213 lr:0.000785 t:5.0s +tttg: c67/213 lr:0.000779 t:5.1s +tttg: c68/213 lr:0.000773 t:5.2s +tttg: c69/213 lr:0.000767 t:5.3s +tttg: c70/213 lr:0.000761 t:5.4s +tttg: c71/213 lr:0.000754 t:5.4s +tttg: c72/213 lr:0.000748 t:5.5s +tttg: c73/213 lr:0.000741 t:5.6s +tttg: c74/213 lr:0.000735 t:5.7s +tttg: c75/213 lr:0.000728 t:5.7s +tttg: c76/213 lr:0.000722 t:5.8s +tttg: c77/213 lr:0.000715 t:5.9s +tttg: c78/213 lr:0.000708 t:6.0s +tttg: c79/213 lr:0.000702 t:6.0s +tttg: c80/213 lr:0.000695 t:6.1s +tttg: c81/213 lr:0.000688 t:6.2s +tttg: c82/213 lr:0.000681 t:6.3s +tttg: c83/213 lr:0.000674 t:6.3s +tttg: c84/213 lr:0.000667 t:6.4s +tttg: c85/213 lr:0.000660 t:6.5s +tttg: c86/213 lr:0.000653 t:6.6s +tttg: c87/213 lr:0.000646 t:6.7s +tttg: c88/213 lr:0.000639 t:6.7s +tttg: c89/213 lr:0.000632 t:6.8s +tttg: c90/213 lr:0.000625 t:6.9s +tttg: c91/213 lr:0.000617 t:7.0s +tttg: c92/213 lr:0.000610 t:7.0s +tttg: c93/213 lr:0.000603 t:7.1s +tttg: c94/213 lr:0.000596 t:7.2s +tttg: c95/213 lr:0.000588 t:7.3s +tttg: c96/213 lr:0.000581 t:7.3s +tttg: c97/213 lr:0.000574 t:7.4s +tttg: c98/213 lr:0.000566 t:7.5s +tttg: c99/213 lr:0.000559 t:7.6s +tttg: c100/213 lr:0.000552 t:7.7s +tttg: c101/213 lr:0.000544 t:7.7s +tttg: c102/213 lr:0.000537 t:7.8s +tttg: c103/213 lr:0.000530 t:7.9s +tttg: c104/213 lr:0.000522 t:8.0s +tttg: c105/213 lr:0.000515 t:8.0s +tttg: c106/213 lr:0.000507 t:8.1s +tttg: c107/213 lr:0.000500 t:8.2s +tttg: c108/213 lr:0.000493 t:8.3s +tttg: c109/213 lr:0.000485 t:8.4s +tttg: c110/213 lr:0.000478 t:8.4s +tttg: c111/213 lr:0.000470 t:8.5s +tttg: c112/213 lr:0.000463 t:8.6s +tttg: c113/213 lr:0.000456 t:8.7s +tttg: c114/213 lr:0.000448 t:8.7s +tttg: c115/213 lr:0.000441 t:8.8s +tttg: c116/213 lr:0.000434 t:8.9s +tttg: c117/213 lr:0.000426 t:9.0s +tttg: c118/213 lr:0.000419 t:9.0s +tttg: c119/213 lr:0.000412 t:9.1s +tttg: c120/213 lr:0.000404 t:9.2s +tttg: c121/213 lr:0.000397 t:9.3s +tttg: c122/213 lr:0.000390 t:9.3s +tttg: c123/213 lr:0.000383 t:9.4s +tttg: c124/213 lr:0.000375 t:9.5s +tttg: c125/213 lr:0.000368 t:9.6s +tttg: c126/213 lr:0.000361 t:9.6s +tttg: c127/213 lr:0.000354 t:9.7s +tttg: c128/213 lr:0.000347 t:9.8s +tttg: c129/213 lr:0.000340 t:9.9s +tttg: c130/213 lr:0.000333 t:10.0s +tttg: c131/213 lr:0.000326 t:10.0s +tttg: c132/213 lr:0.000319 t:10.1s +tttg: c133/213 lr:0.000312 t:10.2s +tttg: c134/213 lr:0.000305 t:10.3s +tttg: c135/213 lr:0.000298 t:10.3s +tttg: c136/213 lr:0.000292 t:10.4s +tttg: c137/213 lr:0.000285 t:10.5s +tttg: c138/213 lr:0.000278 t:10.6s +tttg: c139/213 lr:0.000272 t:10.6s +tttg: c140/213 lr:0.000265 t:10.7s +tttg: c141/213 lr:0.000259 t:10.8s +tttg: c142/213 lr:0.000252 t:10.9s +tttg: c143/213 lr:0.000246 t:10.9s +tttg: c144/213 lr:0.000239 t:11.0s +tttg: c145/213 lr:0.000233 t:11.1s +tttg: c146/213 lr:0.000227 t:11.2s +tttg: c147/213 lr:0.000221 t:11.3s +tttg: c148/213 lr:0.000215 t:11.3s +tttg: c149/213 lr:0.000209 t:11.4s +tttg: c150/213 lr:0.000203 t:11.5s +tttg: c151/213 lr:0.000197 t:11.6s +tttg: c152/213 lr:0.000191 t:11.7s +tttg: c153/213 lr:0.000185 t:11.7s +tttg: c154/213 lr:0.000179 t:11.8s +tttg: c155/213 lr:0.000174 t:11.9s +tttg: c156/213 lr:0.000168 t:12.0s +tttg: c157/213 lr:0.000163 t:12.0s +tttg: c158/213 lr:0.000157 t:12.1s +tttg: c159/213 lr:0.000152 t:12.2s +tttg: c160/213 lr:0.000146 t:12.3s +tttg: c161/213 lr:0.000141 t:12.3s +tttg: c162/213 lr:0.000136 t:12.4s +tttg: c163/213 lr:0.000131 t:12.5s +tttg: c164/213 lr:0.000126 t:12.6s +tttg: c165/213 lr:0.000121 t:12.6s +tttg: c166/213 lr:0.000116 t:12.7s +tttg: c167/213 lr:0.000112 t:12.8s +tttg: c168/213 lr:0.000107 t:12.9s +tttg: c169/213 lr:0.000103 t:12.9s +tttg: c170/213 lr:0.000098 t:13.0s +tttg: c171/213 lr:0.000094 t:13.1s +tttg: c172/213 lr:0.000089 t:13.2s +tttg: c173/213 lr:0.000085 t:13.3s +tttg: c174/213 lr:0.000081 t:13.4s +tttg: c175/213 lr:0.000077 t:13.4s +tttg: c176/213 lr:0.000073 t:13.5s +tttg: c177/213 lr:0.000069 t:13.6s +tttg: c178/213 lr:0.000066 t:13.7s +tttg: c179/213 lr:0.000062 t:13.7s +tttg: c180/213 lr:0.000059 t:13.8s +tttg: c181/213 lr:0.000055 t:13.9s +tttg: c182/213 lr:0.000052 t:14.0s +tttg: c183/213 lr:0.000049 t:14.0s +tttg: c184/213 lr:0.000045 t:14.1s +tttg: c185/213 lr:0.000042 t:14.2s +tttg: c186/213 lr:0.000039 t:14.3s +tttg: c187/213 lr:0.000037 t:14.4s +tttg: c188/213 lr:0.000034 t:14.4s +tttg: c189/213 lr:0.000031 t:14.5s +tttg: c190/213 lr:0.000029 t:14.6s +tttg: c191/213 lr:0.000026 t:14.7s +tttg: c192/213 lr:0.000024 t:14.7s +tttg: c193/213 lr:0.000022 t:14.8s +tttg: c194/213 lr:0.000020 t:14.9s +tttg: c195/213 lr:0.000018 t:15.0s +tttg: c196/213 lr:0.000016 t:15.0s +tttg: c197/213 lr:0.000014 t:15.1s +tttg: c198/213 lr:0.000012 t:15.2s +tttg: c199/213 lr:0.000011 t:15.3s +tttg: c200/213 lr:0.000009 t:15.4s +tttg: c201/213 lr:0.000008 t:15.4s +tttg: c202/213 lr:0.000007 t:15.5s +tttg: c203/213 lr:0.000005 t:15.6s +tttg: c204/213 lr:0.000004 t:15.7s +tttg: c205/213 lr:0.000004 t:15.7s +tttg: c206/213 lr:0.000003 t:15.8s +tttg: c207/213 lr:0.000002 t:15.9s +tttg: c208/213 lr:0.000001 t:16.0s +tttg: c209/213 lr:0.000001 t:16.0s +tttg: c210/213 lr:0.000000 t:16.1s +tttg: c211/213 lr:0.000000 t:16.2s +tttg: c212/213 lr:0.000000 t:16.3s +ttpr: phase:3/3 t:336.1s +ttp: b736/782 bl:2.6672 bb:1.0396 rl:2.7223 rb:1.0790 dl:2140-2165 gd:1 +ttp: b734/782 bl:2.7618 bb:1.0532 rl:2.7250 rb:1.0772 dl:2091-2115 gd:1 +ttp: b721/782 bl:2.7378 bb:1.0219 rl:2.7257 rb:1.0738 dl:1832-1846 gd:1 +ttp: b718/782 bl:2.7655 bb:1.0660 rl:2.7278 rb:1.0734 dl:1773-1792 gd:1 +ttp: b705/782 bl:2.7714 bb:1.0675 rl:2.7298 rb:1.0732 dl:1606-1617 gd:1 +ttp: b702/782 bl:2.7917 bb:1.0619 rl:2.7324 rb:1.0727 dl:1572-1581 gd:1 +ttp: b695/782 bl:2.7735 bb:1.0753 rl:2.7340 rb:1.0728 dl:1504-1513 gd:1 +ttp: b684/782 bl:2.7837 bb:1.0702 rl:2.7357 rb:1.0727 dl:1407-1414 gd:1 +ttp: b675/782 bl:2.8286 bb:1.0619 rl:2.7388 rb:1.0723 dl:1341-1347 gd:1 +ttp: b665/782 bl:2.7304 bb:1.0289 rl:2.7385 rb:1.0710 dl:1275-1282 gd:1 +ttp: b661/782 bl:2.7118 bb:1.0167 rl:2.7377 rb:1.0694 dl:1251-1258 gd:1 +ttp: b650/782 bl:2.7803 bb:1.0705 rl:2.7389 rb:1.0694 dl:1188-1193 gd:1 +ttp: b646/782 bl:2.7613 bb:1.0692 rl:2.7394 rb:1.0694 dl:1166-1171 gd:1 +ttp: b636/782 bl:2.7486 bb:1.0662 rl:2.7396 rb:1.0693 dl:1116-1120 gd:1 +ttp: b627/782 bl:2.7237 bb:1.0310 rl:2.7393 rb:1.0684 dl:1073-1077 gd:1 +ttp: b620/782 bl:2.7688 bb:1.0377 rl:2.7399 rb:1.0678 dl:1041-1046 gd:1 +ttp: b615/782 bl:2.8267 bb:1.0614 rl:2.7417 rb:1.0676 dl:1020-1023 gd:1 +ttp: b604/782 bl:2.7173 bb:1.0330 rl:2.7412 rb:1.0670 dl:974-978 gd:1 +ttp: b596/782 bl:2.7667 bb:1.0596 rl:2.7417 rb:1.0668 dl:943-947 gd:1 +ttp: b589/782 bl:2.7406 bb:1.0491 rl:2.7416 rb:1.0665 dl:921-924 gd:1 +ttp: b582/782 bl:2.8517 bb:1.0877 rl:2.7435 rb:1.0669 dl:897-901 gd:1 +ttp: b574/782 bl:2.7689 bb:1.0343 rl:2.7439 rb:1.0663 dl:871-874 gd:1 +ttp: b564/782 bl:2.8583 bb:1.1059 rl:2.7456 rb:1.0669 dl:840-843 gd:1 +ttp: b555/782 bl:2.7517 bb:1.0501 rl:2.7457 rb:1.0667 dl:812-815 gd:1 +ttp: b552/782 bl:2.7868 bb:1.0386 rl:2.7462 rb:1.0663 dl:804-806 gd:1 +ttp: b544/782 bl:2.7418 bb:1.0383 rl:2.7462 rb:1.0659 dl:782-785 gd:1 +ttp: b535/782 bl:2.7800 bb:1.0540 rl:2.7466 rb:1.0657 dl:759-762 gd:1 +ttp: b527/782 bl:2.7252 bb:1.0356 rl:2.7463 rb:1.0654 dl:739-742 gd:1 +ttp: b519/782 bl:2.7256 bb:1.0336 rl:2.7461 rb:1.0650 dl:720-723 gd:1 +ttp: b506/782 bl:2.7990 bb:1.0722 rl:2.7467 rb:1.0651 dl:688-690 gd:1 +ttp: b498/782 bl:2.6658 bb:1.0320 rl:2.7458 rb:1.0647 dl:671-673 gd:1 +ttp: b496/782 bl:2.8226 bb:1.0461 rl:2.7466 rb:1.0645 dl:666-668 gd:1 +ttp: b488/782 bl:2.8097 bb:1.0471 rl:2.7473 rb:1.0643 dl:649-651 gd:1 +ttp: b480/782 bl:2.7803 bb:1.0496 rl:2.7476 rb:1.0642 dl:632-635 gd:1 +ttp: b472/782 bl:2.7905 bb:1.0667 rl:2.7480 rb:1.0642 dl:616-618 gd:1 +ttp: b465/782 bl:2.7999 bb:1.0562 rl:2.7485 rb:1.0641 dl:602-604 gd:1 +ttp: b458/782 bl:2.8041 bb:1.0629 rl:2.7490 rb:1.0641 dl:589-591 gd:1 +ttp: b450/782 bl:2.7490 bb:1.0260 rl:2.7490 rb:1.0638 dl:575-576 gd:1 +ttp: b442/782 bl:2.7922 bb:1.0487 rl:2.7494 rb:1.0636 dl:560-562 gd:1 +ttp: b434/782 bl:2.7183 bb:1.0387 rl:2.7491 rb:1.0634 dl:545-547 gd:1 +ttp: b426/782 bl:2.7054 bb:1.0587 rl:2.7488 rb:1.0634 dl:532-533 gd:1 +ttp: b414/782 bl:2.8084 bb:1.0826 rl:2.7492 rb:1.0635 dl:511-513 gd:1 +ttp: b407/782 bl:2.7650 bb:1.0526 rl:2.7493 rb:1.0634 dl:500-501 gd:1 +ttp: b398/782 bl:2.8740 bb:1.0916 rl:2.7502 rb:1.0636 dl:486-487 gd:1 +ttp: b390/782 bl:2.8020 bb:1.0868 rl:2.7505 rb:1.0638 dl:473-475 gd:1 +ttp: b383/782 bl:2.8200 bb:1.0800 rl:2.7510 rb:1.0639 dl:463-464 gd:1 +ttp: b375/782 bl:2.7990 bb:1.1030 rl:2.7513 rb:1.0642 dl:452-453 gd:1 +ttp: b368/782 bl:2.8367 bb:1.0823 rl:2.7518 rb:1.0643 dl:441-443 gd:1 +ttp: b360/782 bl:2.8265 bb:1.0779 rl:2.7523 rb:1.0644 dl:430-432 gd:1 +ttp: b353/782 bl:2.7879 bb:1.0925 rl:2.7525 rb:1.0645 dl:420-422 gd:1 +ttp: b345/782 bl:2.8509 bb:1.1056 rl:2.7530 rb:1.0647 dl:410-412 gd:1 +ttp: b337/782 bl:2.8210 bb:1.0741 rl:2.7534 rb:1.0648 dl:399-400 gd:1 +ttp: b329/782 bl:2.8198 bb:1.0999 rl:2.7538 rb:1.0650 dl:389-390 gd:1 +ttp: b320/782 bl:2.7532 bb:1.0740 rl:2.7538 rb:1.0650 dl:377-378 gd:1 +ttp: b313/782 bl:2.8172 bb:1.0850 rl:2.7541 rb:1.0651 dl:368-369 gd:1 +ttp: b307/782 bl:2.8894 bb:1.1045 rl:2.7547 rb:1.0653 dl:361-362 gd:1 +ttp: b301/782 bl:2.7829 bb:1.0822 rl:2.7549 rb:1.0654 dl:353-354 gd:1 +ttp: b293/782 bl:2.7542 bb:1.0640 rl:2.7549 rb:1.0654 dl:343-345 gd:1 +ttp: b286/782 bl:2.8757 bb:1.0924 rl:2.7554 rb:1.0655 dl:335-336 gd:1 +ttp: b279/782 bl:2.8394 bb:1.0852 rl:2.7558 rb:1.0656 dl:327-329 gd:1 +ttp: b272/782 bl:2.8573 bb:1.1084 rl:2.7562 rb:1.0658 dl:320-321 gd:1 +ttp: b264/782 bl:2.8850 bb:1.1419 rl:2.7567 rb:1.0661 dl:311-312 gd:1 +ttp: b255/782 bl:2.8663 bb:1.1311 rl:2.7571 rb:1.0663 dl:300-301 gd:1 +ttp: b247/782 bl:2.7867 bb:1.0767 rl:2.7572 rb:1.0664 dl:292-293 gd:1 +ttp: b239/782 bl:2.8831 bb:1.1307 rl:2.7577 rb:1.0666 dl:284-285 gd:1 +ttp: b198/782 bl:2.9541 bb:1.1425 rl:2.7583 rb:1.0668 dl:245-246 gd:1 +ttp: b190/782 bl:2.8603 bb:1.0873 rl:2.7586 rb:1.0669 dl:237-238 gd:1 +ttp: b182/782 bl:2.8380 bb:1.1290 rl:2.7589 rb:1.0671 dl:230-231 gd:1 +ttp: b173/782 bl:2.9579 bb:1.1499 rl:2.7594 rb:1.0673 dl:223-224 gd:1 +ttp: b167/782 bl:2.9464 bb:1.1778 rl:2.7600 rb:1.0676 dl:218-218 gd:1 +ttp: b158/782 bl:2.8747 bb:1.1380 rl:2.7603 rb:1.0678 dl:210-211 gd:1 +ttp: b149/782 bl:2.9599 bb:1.1669 rl:2.7608 rb:1.0680 dl:203-204 gd:1 +ttp: b142/782 bl:2.9537 bb:1.1580 rl:2.7613 rb:1.0683 dl:197-198 gd:1 +ttp: b134/782 bl:3.0194 bb:1.2076 rl:2.7619 rb:1.0686 dl:190-191 gd:1 +ttp: b128/782 bl:2.8411 bb:1.0907 rl:2.7621 rb:1.0686 dl:186-187 gd:1 +ttp: b120/782 bl:2.9612 bb:1.1635 rl:2.7625 rb:1.0688 dl:180-181 gd:1 +ttp: b114/782 bl:2.9907 bb:1.1840 rl:2.7630 rb:1.0691 dl:176-176 gd:1 +ttp: b105/782 bl:3.0612 bb:1.2263 rl:2.7636 rb:1.0694 dl:169-170 gd:1 +ttp: b98/782 bl:2.9635 bb:1.1762 rl:2.7640 rb:1.0696 dl:164-164 gd:1 +ttp: b90/782 bl:3.0098 bb:1.1872 rl:2.7645 rb:1.0699 dl:158-158 gd:1 +ttp: b78/782 bl:2.8959 bb:1.1236 rl:2.7648 rb:1.0699 dl:148-149 gd:1 +ttp: b70/782 bl:3.0556 bb:1.1611 rl:2.7653 rb:1.0701 dl:142-143 gd:1 +ttp: b64/782 bl:2.9938 bb:1.2409 rl:2.7657 rb:1.0704 dl:138-139 gd:1 +ttp: b59/782 bl:3.0410 bb:1.1878 rl:2.7661 rb:1.0706 dl:134-134 gd:1 +ttp: b51/782 bl:3.0265 bb:1.2097 rl:2.7665 rb:1.0708 dl:127-128 gd:1 +ttp: b44/782 bl:3.1242 bb:1.2174 rl:2.7671 rb:1.0710 dl:122-122 gd:1 +ttp: b35/782 bl:3.0135 bb:1.1967 rl:2.7674 rb:1.0712 dl:115-115 gd:1 +ttp: b25/782 bl:3.2810 bb:1.3003 rl:2.7681 rb:1.0715 dl:106-107 gd:1 +ttp: b18/782 bl:3.1400 bb:1.2720 rl:2.7685 rb:1.0717 dl:99-100 gd:1 +ttp: b9/782 bl:3.1890 bb:1.2637 rl:2.7690 rb:1.0719 dl:87-89 gd:1 +ttp: b2/782 bl:3.1434 bb:1.1659 rl:2.7693 rb:1.0720 dl:70-75 gd:1 +quantized_ttt_phased val_loss:2.76205552 val_bpb:1.06927777 eval_time:438339ms +total_eval_time:438.3s diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_gpt.py b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_gpt.py new file mode 100644 index 0000000000..206c596bbc --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_gpt.py @@ -0,0 +1,3247 @@ +import base64, collections, copy, fcntl, glob, io, json, lzma, math, os +from pathlib import Path +import random, re, subprocess, sys, time, uuid, numpy as np, sentencepiece as spm, torch, torch.distributed as dist, torch.nn.functional as F +from torch import nn +from flash_attn_interface import ( + flash_attn_func as flash_attn_3_func, + flash_attn_varlen_func, +) +from concurrent.futures import ThreadPoolExecutor +import triton +import triton.language as tl +from triton.tools.tensor_descriptor import TensorDescriptor + + +class Hyperparameters: + data_dir = os.environ.get("DATA_DIR", "./data/") + seed = int(os.environ.get("SEED", 1337)) + run_id = os.environ.get("RUN_ID", str(uuid.uuid4())) + iterations = int(os.environ.get("ITERATIONS", 20000)) + warmdown_frac = float(os.environ.get("WARMDOWN_FRAC", 0.75)) + warmup_steps = int(os.environ.get("WARMUP_STEPS", 20)) + train_batch_tokens = int(os.environ.get("TRAIN_BATCH_TOKENS", 786432)) + train_seq_len = int(os.environ.get("TRAIN_SEQ_LEN", 2048)) + train_log_every = int(os.environ.get("TRAIN_LOG_EVERY", 500)) + max_wallclock_seconds = float(os.environ.get("MAX_WALLCLOCK_SECONDS", 6e2)) + val_batch_tokens = int(os.environ.get("VAL_BATCH_TOKENS", 524288)) + eval_seq_len = int(os.environ.get("EVAL_SEQ_LEN", 2048)) + val_loss_every = int(os.environ.get("VAL_LOSS_EVERY", 4000)) + sliding_window_enabled = bool(int(os.environ.get("SLIDING_WINDOW_ENABLED", "0"))) + vocab_size = int(os.environ.get("VOCAB_SIZE", 8192)) + num_layers = int(os.environ.get("NUM_LAYERS", 11)) + xsa_last_n = int(os.environ.get("XSA_LAST_N", 11)) + model_dim = int(os.environ.get("MODEL_DIM", 512)) + num_kv_heads = int(os.environ.get("NUM_KV_HEADS", 4)) + num_heads = int(os.environ.get("NUM_HEADS", 8)) + mlp_mult = float(os.environ.get("MLP_MULT", 4.0)) + skip_gates_enabled = bool(int(os.environ.get("SKIP_GATES_ENABLED", "1"))) + tie_embeddings = bool(int(os.environ.get("TIE_EMBEDDINGS", "1"))) + logit_softcap = float(os.environ.get("LOGIT_SOFTCAP", 3e1)) + rope_base = float(os.environ.get("ROPE_BASE", 1e4)) + rope_dims = int(os.environ.get("ROPE_DIMS", 16)) + rope_train_seq_len = int(os.environ.get("ROPE_TRAIN_SEQ_LEN", 2048)) + rope_yarn = bool(int(os.environ.get("ROPE_YARN", "0"))) + ln_scale = bool(int(os.environ.get("LN_SCALE", "1"))) + qk_gain_init = float(os.environ.get("QK_GAIN_INIT", 5.0)) + num_loops = int(os.environ.get("NUM_LOOPS", 2)) + loop_start = int(os.environ.get("LOOP_START", 3)) + loop_end = int(os.environ.get("LOOP_END", 5)) + enable_looping_at = float(os.environ.get("ENABLE_LOOPING_AT", 0.35)) + parallel_start_layer = int(os.environ.get("PARALLEL_START_LAYER", 8)) + parallel_final_lane = os.environ.get("PARALLEL_FINAL_LANE", "mean") + min_lr = float(os.environ.get("MIN_LR", 0.0)) + embed_lr = float(os.environ.get("EMBED_LR", 0.6)) + tied_embed_lr = float(os.environ.get("TIED_EMBED_LR", 0.03)) + tied_embed_init_std = float(os.environ.get("TIED_EMBED_INIT_STD", 0.005)) + matrix_lr = float(os.environ.get("MATRIX_LR", 0.026)) + scalar_lr = float(os.environ.get("SCALAR_LR", 0.02)) + muon_momentum = float(os.environ.get("MUON_MOMENTUM", 0.97)) + muon_backend_steps = int(os.environ.get("MUON_BACKEND_STEPS", 5)) + muon_momentum_warmup_start = float( + os.environ.get("MUON_MOMENTUM_WARMUP_START", 0.92) + ) + muon_momentum_warmup_steps = int(os.environ.get("MUON_MOMENTUM_WARMUP_STEPS", 1500)) + muon_row_normalize = bool(int(os.environ.get("MUON_ROW_NORMALIZE", "1"))) + beta1 = float(os.environ.get("BETA1", 0.9)) + beta2 = float(os.environ.get("BETA2", 0.95)) + adam_eps = float(os.environ.get("ADAM_EPS", 1e-08)) + grad_clip_norm = float(os.environ.get("GRAD_CLIP_NORM", 0.3)) + eval_stride = int(os.environ.get("EVAL_STRIDE", 64)) + adam_wd = float(os.environ.get("ADAM_WD", 0.02)) + muon_wd = float(os.environ.get("MUON_WD", 0.095)) + embed_wd = float(os.environ.get("EMBED_WD", 0.085)) + ema_decay = float(os.environ.get("EMA_DECAY", 0.9965)) + ttt_enabled = bool(int(os.environ.get("TTT_ENABLED", "1"))) + ttt_lora_rank = int(os.environ.get("TTT_LORA_RANK", 128)) + ttt_lora_lr = float(os.environ.get("TTT_LORA_LR", 0.0001)) + ttt_chunk_size = int(os.environ.get("TTT_CHUNK_SIZE", 48)) + ttt_eval_seq_len = int(os.environ.get("TTT_EVAL_SEQ_LEN", 2048)) + ttt_batch_size = int(os.environ.get("TTT_BATCH_SIZE", 64)) + ttt_grad_steps = int(os.environ.get("TTT_GRAD_STEPS", 1)) + ttt_weight_decay = float(os.environ.get("TTT_WEIGHT_DECAY", 1.0)) + ttt_beta1 = float(os.environ.get("TTT_BETA1", 0)) + ttt_beta2 = float(os.environ.get("TTT_BETA2", 0.999)) + ttt_k_lora = bool(int(os.environ.get("TTT_K_LORA", "1"))) + ttt_mlp_lora = bool(int(os.environ.get("TTT_MLP_LORA", "1"))) + ttt_o_lora = bool(int(os.environ.get("TTT_O_LORA", "1"))) + ttt_optimizer = os.environ.get("TTT_OPTIMIZER", "adam") + ttt_eval_batches = os.environ.get("TTT_EVAL_BATCHES", "") + val_doc_fraction = float(os.environ.get("VAL_DOC_FRACTION", 1.0)) + compressor = os.environ.get("COMPRESSOR", "brotli") + gptq_calibration_batches = int(os.environ.get("GPTQ_CALIBRATION_BATCHES", 16)) + gptq_reserve_seconds = float(os.environ.get("GPTQ_RESERVE_SECONDS", 4.0)) + phased_ttt_enabled = bool(int(os.environ.get("PHASED_TTT_ENABLED", "1"))) + phased_ttt_prefix_docs = int(os.environ.get("PHASED_TTT_PREFIX_DOCS", 2000)) + phased_ttt_num_phases = int(os.environ.get("PHASED_TTT_NUM_PHASES", 3)) + smear_gate_enabled = bool(int(os.environ.get("SMEAR_GATE", "1"))) + smear_gate_width = int(os.environ.get("SMEAR_GATE_WIDTH", 12)) + gate_attn_out = bool(int(os.environ.get("GATE_ATTN_OUT", "1"))) + gate_attn_width = int(os.environ.get("GATE_ATTN_WIDTH", 12)) + global_ttt_lr = float(os.environ.get("GLOBAL_TTT_LR", 0.001)) + global_ttt_momentum = float(os.environ.get("GLOBAL_TTT_MOMENTUM", 0.9)) + global_ttt_epochs = int(os.environ.get("GLOBAL_TTT_EPOCHS", 1)) + global_ttt_chunk_tokens = int(os.environ.get("GLOBAL_TTT_CHUNK_TOKENS", 32768)) + global_ttt_batch_seqs = int(os.environ.get("GLOBAL_TTT_BATCH_SEQS", 32)) + global_ttt_warmup_start_lr = float(os.environ.get("GLOBAL_TTT_WARMUP_START_LR", 0.0)) + global_ttt_warmup_chunks = int(os.environ.get("GLOBAL_TTT_WARMUP_CHUNKS", 0)) + global_ttt_grad_clip = float(os.environ.get("GLOBAL_TTT_GRAD_CLIP", 1.0)) + global_ttt_respect_doc_boundaries = bool(int(os.environ.get("GLOBAL_TTT_RESPECT_DOC_BOUNDARIES", "1"))) + matrix_bits = int(os.environ.get("MATRIX_BITS", 6)) + embed_bits = int(os.environ.get("EMBED_BITS", 7)) + matrix_clip_sigmas = float(os.environ.get("MATRIX_CLIP_SIGMAS", 12.85)) + embed_clip_sigmas = float(os.environ.get("EMBED_CLIP_SIGMAS", 15.0)) + mlp_clip_sigmas = float(os.environ.get("MLP_CLIP_SIGMAS", 12.0)) + attn_clip_sigmas = float(os.environ.get("ATTN_CLIP_SIGMAS", 13.0)) + # Polar Express per-iteration minimax NS coefficients (PR #1344). Default on. + polar_express_ns = bool(int(os.environ.get("POLAR_EXPRESS_NS", "1"))) + # LQER asymmetric rank-k correction on top-K quant-error tensors (PR #1797). + lqer_enabled = bool(int(os.environ.get("LQER_ENABLED", "1"))) + lqer_rank = int(os.environ.get("LQER_RANK", 4)) + lqer_top_k = int(os.environ.get("LQER_TOP_K", 3)) + lqer_factor_bits = int(os.environ.get("LQER_FACTOR_BITS", 4)) + lqer_asym_enabled = bool(int(os.environ.get("LQER_ASYM_ENABLED", "1"))) + lqer_asym_group = int(os.environ.get("LQER_ASYM_GROUP", "64")) + newton_muon_enabled = bool(int(os.environ.get("NEWTON_MUON_ENABLED", "0"))) + newton_muon_beta = float(os.environ.get("NEWTON_MUON_BETA", "0.95")) + newton_muon_gamma = float(os.environ.get("NEWTON_MUON_GAMMA", "0.2")) + newton_muon_k_refresh = int(os.environ.get("NEWTON_MUON_K_REFRESH", "32")) + newton_muon_warmup = int(os.environ.get("NEWTON_MUON_WARMUP", "100")) + newton_muon_capture_every = int(os.environ.get("NEWTON_MUON_CAPTURE_EVERY", "4")) + newton_muon_all_reduce_k = bool(int(os.environ.get("NEWTON_MUON_ALL_REDUCE_K", "1"))) + distributed = "RANK" in os.environ and "WORLD_SIZE" in os.environ + rank = int(os.environ.get("RANK", "0")) + world_size = int(os.environ.get("WORLD_SIZE", "1")) + local_rank = int(os.environ.get("LOCAL_RANK", "0")) + is_main_process = rank == 0 + grad_accum_steps = 8 // world_size + datasets_dir = os.path.join(data_dir, "datasets", f"fineweb10B_sp{vocab_size}") + train_files = os.path.join(datasets_dir, "fineweb_train_*.bin") + val_files = os.path.join(datasets_dir, "fineweb_val_*.bin") + tokenizer_path = os.path.join( + data_dir, "tokenizers", f"fineweb_{vocab_size}_bpe.model" + ) + artifact_dir = os.environ.get("ARTIFACT_DIR", "") + logfile = ( + os.path.join(artifact_dir, f"{run_id}.txt") + if artifact_dir + else f"logs/{run_id}.txt" + ) + model_path = ( + os.path.join(artifact_dir, "final_model.pt") + if artifact_dir + else "final_model.pt" + ) + quantized_model_path = ( + os.path.join(artifact_dir, "final_model.int6.ptz") + if artifact_dir + else "final_model.int6.ptz" + ) + + +_logger_hparams = None + + +def set_logging_hparams(h): + global _logger_hparams + _logger_hparams = h + + +def log(msg, console=True): + if _logger_hparams is None: + print(msg) + return + if _logger_hparams.is_main_process: + if console: + print(msg) + if _logger_hparams.logfile is not None: + with open(_logger_hparams.logfile, "a", encoding="utf-8") as f: + print(msg, file=f) + + +class ValidationData: + def __init__(self, h, device): + self.sp = spm.SentencePieceProcessor(model_file=h.tokenizer_path) + if int(self.sp.vocab_size()) != h.vocab_size: + raise ValueError( + f"VOCAB_SIZE={h.vocab_size} does not match tokenizer vocab_size={int(self.sp.vocab_size())}" + ) + self.val_tokens = load_validation_tokens(h.val_files, h.eval_seq_len) + ( + self.base_bytes_lut, + self.has_leading_space_lut, + self.is_boundary_token_lut, + ) = build_sentencepiece_luts(self.sp, h.vocab_size, device) + + +def build_sentencepiece_luts(sp, vocab_size, device): + sp_vocab_size = int(sp.vocab_size()) + assert ( + sp.piece_to_id("▁") != sp.unk_id() + ), "Tokenizer must have '▁' (space) as its own token for correct BPB byte counting" + table_size = max(sp_vocab_size, vocab_size) + base_bytes_np = np.zeros((table_size,), dtype=np.int16) + has_leading_space_np = np.zeros((table_size,), dtype=np.bool_) + is_boundary_token_np = np.ones((table_size,), dtype=np.bool_) + for token_id in range(sp_vocab_size): + if sp.is_control(token_id) or sp.is_unknown(token_id) or sp.is_unused(token_id): + continue + is_boundary_token_np[token_id] = False + if sp.is_byte(token_id): + base_bytes_np[token_id] = 1 + continue + piece = sp.id_to_piece(token_id) + if piece.startswith("▁"): + has_leading_space_np[token_id] = True + piece = piece[1:] + base_bytes_np[token_id] = len(piece.encode("utf-8")) + return ( + torch.tensor(base_bytes_np, dtype=torch.int16, device=device), + torch.tensor(has_leading_space_np, dtype=torch.bool, device=device), + torch.tensor(is_boundary_token_np, dtype=torch.bool, device=device), + ) + + +def load_validation_tokens(pattern, seq_len): + files = [Path(p) for p in sorted(glob.glob(pattern))] + if not files: + raise FileNotFoundError(f"No files found for pattern: {pattern}") + tokens = torch.cat([load_data_shard(file) for file in files]).contiguous() + usable = (tokens.numel() - 1) // seq_len * seq_len + if usable <= 0: + raise ValueError(f"Validation split is too short for TRAIN_SEQ_LEN={seq_len}") + return tokens[: usable + 1] + + +def load_data_shard(file): + header_bytes = 256 * np.dtype(" 0: + pos = start + while pos < end: + seg_starts.append(pos) + pos += max_doc_len + else: + seg_starts.append(start) + boundaries = seg_starts + [total_len] + padded_len = get_next_multiple_of_n(len(boundaries), bucket_size) + cu = torch.full((padded_len,), total_len, dtype=torch.int32, device=device) + cu[: len(boundaries)] = torch.tensor(boundaries, dtype=torch.int32, device=device) + seg_ends = seg_starts[1:] + [total_len] + max_seqlen = max(end - start for start, end in zip(seg_starts, seg_ends)) + return cu, max_seqlen + +class DocumentPackingLoader: + _shard_pool = ThreadPoolExecutor(1) + + def __init__(self, h, device, cu_bucket_size=64): + self.rank = h.rank + self.world_size = h.world_size + self.device = device + self.cu_bucket_size = cu_bucket_size + self.max_seq_len = h.train_seq_len + all_files = [Path(p) for p in sorted(glob.glob(h.train_files))] + if not all_files: + raise FileNotFoundError(f"No files found for pattern: {h.train_files}") + self.files = all_files + self.file_iter = iter(self.files) + self._init_shard(load_data_shard(next(self.file_iter))) + self._next_shard = self._submit_next_shard() + self._batch_pool = ThreadPoolExecutor(1) + self._next_batch = None + + def _init_shard(self, tokens): + global BOS_ID + self.tokens = tokens + self.shard_size = tokens.numel() + if BOS_ID is None: + BOS_ID = 1 + self.bos_idx = ( + (tokens == BOS_ID).nonzero(as_tuple=True)[0].to(torch.int64).cpu().numpy() + ) + if self.bos_idx.size == 0: + self.bos_idx = np.array([0], dtype=np.int64) + self.cursor = int(self.bos_idx[0]) + + def _submit_next_shard(self): + try: + path = next(self.file_iter) + return self._shard_pool.submit(load_data_shard, path) + except StopIteration: + return None + + def _advance_shard(self): + if self._next_shard is None: + self.file_iter = iter(self.files) + self._next_shard = self._shard_pool.submit( + load_data_shard, next(self.file_iter) + ) + self._init_shard(self._next_shard.result()) + self._next_shard = self._submit_next_shard() + + def _local_doc_starts(self, local_start, total_len): + lo = np.searchsorted(self.bos_idx, local_start, side="left") + hi = np.searchsorted(self.bos_idx, local_start + total_len, side="left") + return (self.bos_idx[lo:hi] - local_start).tolist() + + def _prepare_batch(self, num_tokens_local, max_seq_len): + per_rank_span = num_tokens_local + 1 + global_span = per_rank_span * self.world_size + while self.cursor + global_span > self.shard_size: + self._advance_shard() + local_start = self.cursor + self.rank * per_rank_span + buf = self.tokens[local_start : local_start + per_rank_span] + inputs = buf[:-1].to(dtype=torch.int64).pin_memory() + targets = buf[1:].to(dtype=torch.int64).pin_memory() + starts = self._local_doc_starts(local_start, inputs.numel()) + cu_seqlens, max_seqlen = _build_cu_seqlens( + starts, inputs.numel(), inputs.device, max_seq_len, self.cu_bucket_size + ) + cu_seqlens = cu_seqlens.pin_memory() + self.cursor += global_span + return inputs, targets, cu_seqlens, max_seqlen + + def next_batch(self, global_tokens, grad_accum_steps): + num_tokens_local = global_tokens // (self.world_size * grad_accum_steps) + if self._next_batch is not None: + inputs, targets, cu_seqlens, max_seqlen = self._next_batch.result() + else: + inputs, targets, cu_seqlens, max_seqlen = self._prepare_batch( + num_tokens_local, self.max_seq_len + ) + self._next_batch = self._batch_pool.submit( + self._prepare_batch, num_tokens_local, self.max_seq_len + ) + return ( + inputs[None].to(self.device, non_blocking=True), + targets[None].to(self.device, non_blocking=True), + cu_seqlens.to(self.device, non_blocking=True), + max_seqlen, + ) + + +class ShuffledSequenceLoader: + def __init__(self, h, device): + self.world_size = h.world_size + self.seq_len = h.train_seq_len + self.device = device + all_files = [Path(p) for p in sorted(glob.glob(h.train_files))] + if not all_files: + raise FileNotFoundError(f"No files found for pattern: {h.train_files}") + self.files = all_files[h.rank :: h.world_size] + self.rng = np.random.Generator(np.random.PCG64(h.rank)) + self.num_tokens = [_read_num_tokens(f) for f in self.files] + self.start_inds = [[] for _ in self.files] + for si in range(len(self.files)): + self._reset_shard(si) + + def _reset_shard(self, si): + max_phase = min( + self.seq_len - 1, max(0, self.num_tokens[si] - self.seq_len - 1) + ) + phase = int(self.rng.integers(max_phase + 1)) if max_phase > 0 else 0 + num_sequences = (self.num_tokens[si] - 1 - phase) // self.seq_len + sequence_order = self.rng.permutation(num_sequences) + self.start_inds[si] = (phase + sequence_order * self.seq_len).tolist() + + def next_batch(self, global_tokens, grad_accum_steps): + device_tokens = global_tokens // (self.world_size * grad_accum_steps) + device_batch_size = device_tokens // self.seq_len + remaining = np.array([len(s) for s in self.start_inds], dtype=np.float64) + x = torch.empty((device_batch_size, self.seq_len), dtype=torch.int64) + y = torch.empty((device_batch_size, self.seq_len), dtype=torch.int64) + for bi in range(device_batch_size): + total = remaining.sum() + if total <= 0: + for si in range(len(self.files)): + self._reset_shard(si) + remaining = np.array( + [len(s) for s in self.start_inds], dtype=np.float64 + ) + total = remaining.sum() + probs = remaining / total + si = int(self.rng.choice(len(self.files), p=probs)) + start_ind = self.start_inds[si].pop() + remaining[si] -= 1 + mm = _get_shard_memmap(self.files[si]) + window = torch.as_tensor( + np.array(mm[start_ind : start_ind + self.seq_len + 1], dtype=np.int64) + ) + x[bi] = window[:-1] + y[bi] = window[1:] + return x.to(self.device, non_blocking=True), y.to( + self.device, non_blocking=True + ) + + +class RMSNorm(nn.Module): + def __init__(self, eps=None): + super().__init__() + self.eps = eps + + def forward(self, x): + return F.rms_norm(x, (x.size(-1),), eps=self.eps) + + +class CastedLinear(nn.Linear): + def forward(self, x): + w = self.weight.to(x.dtype) + bias = self.bias.to(x.dtype) if self.bias is not None else None + return F.linear(x, w, bias) + + +@triton.jit +def linear_leaky_relu_square_kernel( + a_desc, + b_desc, + c_desc, + aux_desc, + M, + N, + K, + BLOCK_SIZE_M: tl.constexpr, + BLOCK_SIZE_N: tl.constexpr, + BLOCK_SIZE_K: tl.constexpr, + NUM_SMS: tl.constexpr, + FORWARD: tl.constexpr, +): + dtype = tl.bfloat16 + start_pid = tl.program_id(axis=0) + num_pid_m = tl.cdiv(M, BLOCK_SIZE_M) + num_pid_n = tl.cdiv(N, BLOCK_SIZE_N) + k_tiles = tl.cdiv(K, BLOCK_SIZE_K) + num_tiles = num_pid_m * num_pid_n + tile_id_c = start_pid - NUM_SMS + for tile_id in tl.range(start_pid, num_tiles, NUM_SMS, flatten=True): + pid_m = tile_id // num_pid_n + pid_n = tile_id % num_pid_n + offs_am = pid_m * BLOCK_SIZE_M + offs_bn = pid_n * BLOCK_SIZE_N + accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32) + for ki in range(k_tiles): + offs_k = ki * BLOCK_SIZE_K + a = a_desc.load([offs_am, offs_k]) + b = b_desc.load([offs_bn, offs_k]) + accumulator = tl.dot(a, b.T, accumulator) + tile_id_c += NUM_SMS + offs_am_c = offs_am + offs_bn_c = offs_bn + acc = tl.reshape(accumulator, (BLOCK_SIZE_M, 2, BLOCK_SIZE_N // 2)) + acc = tl.permute(acc, (0, 2, 1)) + acc0, acc1 = tl.split(acc) + c0 = acc0.to(dtype) + c1 = acc1.to(dtype) + if not FORWARD: + pre0 = aux_desc.load([offs_am_c, offs_bn_c]) + pre1 = aux_desc.load([offs_am_c, offs_bn_c + BLOCK_SIZE_N // 2]) + c0 = c0 * tl.where(pre0 > 0, 2.0 * pre0, 0.5 * pre0) + c1 = c1 * tl.where(pre1 > 0, 2.0 * pre1, 0.5 * pre1) + c_desc.store([offs_am_c, offs_bn_c], c0) + c_desc.store([offs_am_c, offs_bn_c + BLOCK_SIZE_N // 2], c1) + if FORWARD: + aux0 = tl.where(c0 > 0, c0, 0.5 * c0) + aux1 = tl.where(c1 > 0, c1, 0.5 * c1) + aux_desc.store([offs_am_c, offs_bn_c], aux0 * aux0) + aux_desc.store([offs_am_c, offs_bn_c + BLOCK_SIZE_N // 2], aux1 * aux1) + + +def linear_leaky_relu_square(a, b, aux=None): + M, K = a.shape + N, K2 = b.shape + assert K == K2 + c = torch.empty((M, N), device=a.device, dtype=a.dtype) + forward = aux is None + if aux is None: + aux = torch.empty((M, N), device=a.device, dtype=a.dtype) + num_sms = torch.cuda.get_device_properties(a.device).multi_processor_count + BLOCK_SIZE_M, BLOCK_SIZE_N, BLOCK_SIZE_K = 128, 256, 64 + num_stages = 4 if forward else 3 + a_desc = TensorDescriptor.from_tensor(a, [BLOCK_SIZE_M, BLOCK_SIZE_K]) + b_desc = TensorDescriptor.from_tensor(b, [BLOCK_SIZE_N, BLOCK_SIZE_K]) + c_desc = TensorDescriptor.from_tensor(c, [BLOCK_SIZE_M, BLOCK_SIZE_N // 2]) + aux_desc = TensorDescriptor.from_tensor(aux, [BLOCK_SIZE_M, BLOCK_SIZE_N // 2]) + grid = lambda _meta: ( + min(num_sms, triton.cdiv(M, BLOCK_SIZE_M) * triton.cdiv(N, BLOCK_SIZE_N)), + ) + linear_leaky_relu_square_kernel[grid]( + a_desc, + b_desc, + c_desc, + aux_desc, + M, + N, + K, + BLOCK_SIZE_M=BLOCK_SIZE_M, + BLOCK_SIZE_N=BLOCK_SIZE_N, + BLOCK_SIZE_K=BLOCK_SIZE_K, + NUM_SMS=num_sms, + FORWARD=forward, + num_stages=num_stages, + num_warps=8, + ) + if forward: + return c, aux + return c + + +class FusedLinearLeakyReLUSquareFunction(torch.autograd.Function): + @staticmethod + def forward(ctx, x, w1, w2): + x_flat = x.reshape(-1, x.shape[-1]) + pre, post = linear_leaky_relu_square(x_flat, w1) + out = F.linear(post, w2) + ctx.save_for_backward(x, w1, w2, pre, post) + return out.view(*x.shape[:-1], out.shape[-1]) + + @staticmethod + def backward(ctx, grad_output): + x, w1, w2, pre, post = ctx.saved_tensors + x_flat = x.reshape(-1, x.shape[-1]) + grad_output_flat = grad_output.reshape(-1, grad_output.shape[-1]) + dw2 = grad_output_flat.T @ post + dpre = linear_leaky_relu_square(grad_output_flat, w2.T.contiguous(), aux=pre) + dw1 = dpre.T @ x_flat + dx = dpre @ w1 + return dx.view_as(x), dw1, dw2 + + +FusedLeakyReLUSquareMLP = FusedLinearLeakyReLUSquareFunction.apply + + +class Rotary(nn.Module): + def __init__(self, dim, base=1e4, train_seq_len=1024, rope_dims=0, yarn=True): + super().__init__() + self.dim = dim + self.base = base + self.train_seq_len = train_seq_len + self.yarn = yarn + self.rope_dims = rope_dims if rope_dims > 0 else dim + inv_freq = 1.0 / base ** ( + torch.arange(0, self.rope_dims, 2, dtype=torch.float32) / self.rope_dims + ) + self.register_buffer("inv_freq", inv_freq, persistent=False) + self._seq_len_cached = 0 + self._cos_cached = None + self._sin_cached = None + + def forward(self, seq_len, device, dtype): + if ( + self._cos_cached is None + or self._sin_cached is None + or self._seq_len_cached < seq_len + or self._cos_cached.device != device + ): + rd = self.rope_dims + if self.yarn and seq_len > self.train_seq_len: + scale = seq_len / self.train_seq_len + new_base = self.base * scale ** (rd / (rd - 2)) + inv_freq = 1.0 / new_base ** ( + torch.arange(0, rd, 2, dtype=torch.float32, device=device) / rd + ) + else: + inv_freq = self.inv_freq.float().to(device) + t = torch.arange(seq_len, device=device, dtype=torch.float32) + freqs = torch.outer(t, inv_freq) + self._cos_cached = freqs.cos()[None, :, None, :] + self._sin_cached = freqs.sin()[None, :, None, :] + self._seq_len_cached = seq_len + return self._cos_cached[:, :seq_len].to(dtype=dtype), self._sin_cached[:, :seq_len].to(dtype=dtype) + + +def apply_rotary_emb(x, cos, sin, rope_dims=0): + if rope_dims > 0 and rope_dims < x.size(-1): + x_rope, x_pass = x[..., :rope_dims], x[..., rope_dims:] + half = rope_dims // 2 + x1, x2 = x_rope[..., :half], x_rope[..., half:] + x_rope = torch.cat((x1 * cos + x2 * sin, x1 * -sin + x2 * cos), dim=-1) + return torch.cat((x_rope, x_pass), dim=-1) + half = x.size(-1) // 2 + x1, x2 = x[..., :half], x[..., half:] + return torch.cat((x1 * cos + x2 * sin, x1 * -sin + x2 * cos), dim=-1) + + +class CausalSelfAttention(nn.Module): + def __init__( + self, dim, num_heads, num_kv_heads, rope_base, qk_gain_init, train_seq_len, yarn=True + ): + super().__init__() + if dim % num_heads != 0: + raise ValueError("model_dim must be divisible by num_heads") + if num_heads % num_kv_heads != 0: + raise ValueError("num_heads must be divisible by num_kv_heads") + self.num_heads = num_heads + self.num_kv_heads = num_kv_heads + self.head_dim = dim // num_heads + if self.head_dim % 2 != 0: + raise ValueError("head_dim must be even for RoPE") + self.q_gain = nn.Parameter( + torch.full((num_heads,), qk_gain_init, dtype=torch.float32) + ) + self.rope_dims = 0 + self.rotary = Rotary(self.head_dim, base=rope_base, train_seq_len=train_seq_len, yarn=yarn) + self.use_xsa = False + self.gate_attn_out = False + self.gate_attn_width = 12 + + def _xsa_efficient(self, y, v): + B, T, H, D = y.shape + Hkv = v.size(-2) + group = H // Hkv + y_g = y.reshape(B, T, Hkv, group, D) + vn = F.normalize(v, dim=-1).unsqueeze(-2) + proj = (y_g * vn).sum(dim=-1, keepdim=True) * vn + return (y_g - proj).reshape(B, T, H, D) + + def forward(self, x, q_w, k_w, v_w, out_w, cu_seqlens=None, max_seqlen=0): + bsz, seqlen, dim = x.shape + q = F.linear(x, q_w.to(x.dtype)).reshape(bsz, seqlen, self.num_heads, self.head_dim) + k = F.linear(x, k_w.to(x.dtype)).reshape(bsz, seqlen, self.num_kv_heads, self.head_dim) + v = F.linear(x, v_w.to(x.dtype)).reshape(bsz, seqlen, self.num_kv_heads, self.head_dim) + q = F.rms_norm(q, (q.size(-1),)) + k = F.rms_norm(k, (k.size(-1),)) + cos, sin = self.rotary(seqlen, x.device, q.dtype) + q = apply_rotary_emb(q, cos, sin, self.rope_dims) + k = apply_rotary_emb(k, cos, sin, self.rope_dims) + q = q * self.q_gain.to(dtype=q.dtype)[None, None, :, None] + if cu_seqlens is not None: + y = flash_attn_varlen_func( + q[0], + k[0], + v[0], + cu_seqlens_q=cu_seqlens, + cu_seqlens_k=cu_seqlens, + max_seqlen_q=max_seqlen, + max_seqlen_k=max_seqlen, + causal=True, + window_size=(-1, -1), + )[None] + else: + y = flash_attn_3_func(q, k, v, causal=True) + if self.use_xsa: + y = self._xsa_efficient(y, v) + y = y.reshape(bsz, seqlen, self.num_heads, self.head_dim) + if self.gate_attn_out: + g = 2.0 * torch.sigmoid(self.attn_gate_proj(x[:, :, :self.gate_attn_width])) + y = y * g.unsqueeze(-1) + y = y.reshape(bsz, seqlen, dim) + self._last_proj_input = y.detach() if getattr(self, "_calib", False) else None + return F.linear(y, out_w.to(x.dtype)) + + +class MLP(nn.Module): + def __init__(self, dim, mlp_mult): + super().__init__() + self.use_fused = True + + def forward(self, x, up_w, down_w): + if self.training and self.use_fused: + return FusedLeakyReLUSquareMLP(x, up_w.to(x.dtype), down_w.to(x.dtype)) + hidden = F.leaky_relu(F.linear(x, up_w.to(x.dtype)), negative_slope=0.5).square() + self._last_down_input = hidden.detach() if getattr(self, "_calib", False) else None + return F.linear(hidden, down_w.to(x.dtype)) + + +class Block(nn.Module): + def __init__( + self, + dim, + num_heads, + num_kv_heads, + mlp_mult, + rope_base, + qk_gain_init, + train_seq_len, + layer_idx=0, + ln_scale=False, + yarn=True, + ): + super().__init__() + self.attn_norm = RMSNorm() + self.mlp_norm = RMSNorm() + self.attn = CausalSelfAttention( + dim, num_heads, num_kv_heads, rope_base, qk_gain_init, train_seq_len, yarn=yarn + ) + self.mlp = MLP(dim, mlp_mult) + self.attn_scale = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + self.mlp_scale = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + self.resid_mix = nn.Parameter( + torch.stack((torch.ones(dim), torch.zeros(dim))).float() + ) + self.ln_scale_factor = 1.0 / math.sqrt(layer_idx + 1) if ln_scale else 1.0 + + def forward(self, x, x0, q_w, k_w, v_w, out_w, up_w, down_w, cu_seqlens=None, max_seqlen=0): + mix = self.resid_mix.to(dtype=x.dtype) + x_in = mix[0][None, None, :] * x + mix[1][None, None, :] * x0 + attn_out = self.attn( + self.attn_norm(x_in) * self.ln_scale_factor, + q_w, k_w, v_w, out_w, + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen, + ) + x_out = x_in + self.attn_scale.to(dtype=x_in.dtype)[None, None, :] * attn_out + x_out = x_out + self.mlp_scale.to(dtype=x_out.dtype)[ + None, None, : + ] * self.mlp(self.mlp_norm(x_out) * self.ln_scale_factor, up_w, down_w) + return x_out + +class GPT(nn.Module): + def __init__(self, h): + super().__init__() + if h.logit_softcap <= 0.0: + raise ValueError(f"logit_softcap must be positive, got {h.logit_softcap}") + self.tie_embeddings = h.tie_embeddings + self.tied_embed_init_std = h.tied_embed_init_std + self.logit_softcap = h.logit_softcap + self.tok_emb = nn.Embedding(h.vocab_size, h.model_dim) + self.num_layers = h.num_layers + head_dim = h.model_dim // h.num_heads + kv_dim = h.num_kv_heads * head_dim + hidden_dim = int(h.mlp_mult * h.model_dim) + self.qo_bank = nn.Parameter(torch.empty(2 * h.num_layers, h.model_dim, h.model_dim)) + self.kv_bank = nn.Parameter(torch.empty(2 * h.num_layers, kv_dim, h.model_dim)) + self.mlp_up_bank = nn.Parameter(torch.empty(h.num_layers, hidden_dim, h.model_dim)) + self.mlp_down_bank = nn.Parameter(torch.empty(h.num_layers, h.model_dim, hidden_dim)) + self.num_encoder_layers = h.num_layers // 2 + self.num_decoder_layers = h.num_layers - self.num_encoder_layers + self.blocks = nn.ModuleList( + [ + Block( + h.model_dim, + h.num_heads, + h.num_kv_heads, + h.mlp_mult, + h.rope_base, + h.qk_gain_init, + h.train_seq_len, + layer_idx=i, + ln_scale=h.ln_scale, + yarn=h.rope_yarn, + ) + for i in range(h.num_layers) + ] + ) + if h.rope_dims > 0: + head_dim = h.model_dim // h.num_heads + for block in self.blocks: + block.attn.rope_dims = h.rope_dims + block.attn.rotary = Rotary( + head_dim, + base=h.rope_base, + train_seq_len=h.train_seq_len, + rope_dims=h.rope_dims, + yarn=h.rope_yarn, + ) + self.final_norm = RMSNorm() + self.lm_head = ( + None + if h.tie_embeddings + else CastedLinear(h.model_dim, h.vocab_size, bias=False) + ) + if self.lm_head is not None: + self.lm_head._zero_init = True + if h.xsa_last_n > 0: + for i in range(max(0, h.num_layers - h.xsa_last_n), h.num_layers): + self.blocks[i].attn.use_xsa = True + if h.gate_attn_out: + for block in self.blocks: + block.attn.gate_attn_out = True + block.attn.gate_attn_width = h.gate_attn_width + block.attn.attn_gate_proj = CastedLinear(h.gate_attn_width, h.num_heads, bias=False) + nn.init.zeros_(block.attn.attn_gate_proj.weight) + self.looping_active = False + if h.num_loops > 0: + loop_seg = list(range(h.loop_start, h.loop_end + 1)) + all_indices = list(range(h.loop_start)) + for _ in range(h.num_loops + 1): + all_indices.extend(loop_seg) + all_indices.extend(range(h.loop_end + 1, h.num_layers)) + num_enc = len(all_indices) // 2 + self.encoder_indices = all_indices[:num_enc] + self.decoder_indices = all_indices[num_enc:] + else: + self.encoder_indices = list(range(self.num_encoder_layers)) + self.decoder_indices = list(range(self.num_encoder_layers, h.num_layers)) + self.num_skip_weights = min( + len(self.encoder_indices), len(self.decoder_indices) + ) + self.skip_weights = nn.Parameter( + torch.ones(self.num_skip_weights, h.model_dim, dtype=torch.float32) + ) + self.skip_gates = ( + nn.Parameter( + torch.zeros(self.num_skip_weights, h.model_dim, dtype=torch.float32) + ) + if h.skip_gates_enabled + else None + ) + self.parallel_start_layer = h.parallel_start_layer + self.parallel_final_lane = h.parallel_final_lane.lower() + self.parallel_post_lambdas = nn.Parameter( + torch.ones(h.num_layers, 2, 2, dtype=torch.float32) + ) + self.parallel_resid_lambdas = nn.Parameter( + torch.full((h.num_layers, 2), 1.1, dtype=torch.float32) + ) + self.smear_gate_enabled = h.smear_gate_enabled + if self.smear_gate_enabled: + self.smear_width = h.smear_gate_width + self.smear_gate = CastedLinear(self.smear_width, 1, bias=False) + nn.init.zeros_(self.smear_gate.weight) + self.smear_lambda = nn.Parameter(torch.zeros(1, dtype=torch.float32)) + self._init_weights() + + def _init_weights(self): + if self.tie_embeddings: + nn.init.normal_(self.tok_emb.weight, mean=0.0, std=self.tied_embed_init_std) + n = self.num_layers + proj_scale = 1.0 / math.sqrt(2 * n) + for i in range(n): + nn.init.orthogonal_(self.qo_bank.data[i], gain=1.0) + nn.init.zeros_(self.qo_bank.data[n + i]) + self.qo_bank.data[n + i].mul_(proj_scale) + nn.init.orthogonal_(self.kv_bank.data[i], gain=1.0) + nn.init.orthogonal_(self.kv_bank.data[n + i], gain=1.0) + for i in range(n): + nn.init.orthogonal_(self.mlp_up_bank.data[i], gain=1.0) + nn.init.zeros_(self.mlp_down_bank.data[i]) + self.mlp_down_bank.data[i].mul_(proj_scale) + for name, module in self.named_modules(): + if isinstance(module, nn.Linear): + if getattr(module, "_zero_init", False): + nn.init.zeros_(module.weight) + elif ( + module.weight.ndim == 2 + and module.weight.shape[0] >= 64 + and module.weight.shape[1] >= 64 + ): + nn.init.orthogonal_(module.weight, gain=1.0) + + def _bank_weights(self, i): + n = self.num_layers + return ( + self.qo_bank[i], + self.kv_bank[i], + self.kv_bank[n + i], + self.qo_bank[n + i], + self.mlp_up_bank[i], + self.mlp_down_bank[i], + ) + + def _parallel_block( + self, block_idx, lane0, lane1, x0, + q_w, k_w, v_w, out_w, up_w, down_w, + cu_seqlens=None, max_seqlen=0, + ): + block = self.blocks[block_idx] + mix = block.resid_mix.to(dtype=lane0.dtype) + attn_read = mix[0][None, None, :] * lane0 + mix[1][None, None, :] * x0 + attn_out = block.attn( + block.attn_norm(attn_read) * block.ln_scale_factor, + q_w, k_w, v_w, out_w, + cu_seqlens=cu_seqlens, max_seqlen=max_seqlen, + ) + attn_out = block.attn_scale.to(dtype=attn_out.dtype)[None, None, :] * attn_out + mlp_read = lane1 + mlp_out = block.mlp_scale.to(dtype=lane1.dtype)[None, None, :] * block.mlp( + block.mlp_norm(mlp_read) * block.ln_scale_factor, up_w, down_w + ) + attn_resid = self.parallel_resid_lambdas[block_idx, 0].to(dtype=lane0.dtype) + attn_post = self.parallel_post_lambdas[block_idx, 0].to(dtype=lane0.dtype) + mlp_resid = self.parallel_resid_lambdas[block_idx, 1].to(dtype=lane0.dtype) + mlp_post = self.parallel_post_lambdas[block_idx, 1].to(dtype=lane0.dtype) + lane0 = attn_resid * lane0 + attn_post[0] * attn_out + mlp_post[0] * mlp_out + lane1 = mlp_resid * lane1 + attn_post[1] * attn_out + mlp_post[1] * mlp_out + return lane0, lane1 + + def _final_parallel_hidden(self, lane0, lane1): + if self.parallel_final_lane == "mlp": + return lane1 + if self.parallel_final_lane == "attn": + return lane0 + return 0.5 * (lane0 + lane1) + + def forward_logits(self, input_ids, cu_seqlens=None, max_seqlen=0): + x = self.tok_emb(input_ids) + if self.smear_gate_enabled and x.dim() == 3 and x.size(1) > 1: + sl = self.smear_lambda.to(dtype=x.dtype) + g = sl * torch.sigmoid(self.smear_gate(x[:, 1:, :self.smear_width])) + x = torch.cat([x[:, :1], x[:, 1:] + g * x[:, :-1]], dim=1) + x = F.rms_norm(x, (x.size(-1),)) + x0 = x + skips = [] + enc_iter = ( + self.encoder_indices + if self.looping_active + else range(self.num_encoder_layers) + ) + dec_iter = ( + self.decoder_indices + if self.looping_active + else range( + self.num_encoder_layers, + self.num_encoder_layers + self.num_decoder_layers, + ) + ) + for i in enc_iter: + q_w, k_w, v_w, out_w, up_w, down_w = self._bank_weights(i) + x = self.blocks[i](x, x0, q_w, k_w, v_w, out_w, up_w, down_w, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen) + skips.append(x) + psl = self.parallel_start_layer + lane0 = None + lane1 = None + for skip_idx, i in enumerate(dec_iter): + q_w, k_w, v_w, out_w, up_w, down_w = self._bank_weights(i) + if i >= psl and psl > 0: + if lane0 is None: + lane0 = x + lane1 = x + if skip_idx < self.num_skip_weights and skips: + skip = skips.pop() + w = self.skip_weights[skip_idx].to(dtype=lane0.dtype)[None, None, :] + if self.skip_gates is not None: + g = torch.sigmoid(self.skip_gates[skip_idx].to(dtype=lane0.dtype))[None, None, :] + lane0 = torch.lerp(w * skip, lane0, g) + else: + lane0 = lane0 + w * skip + lane0, lane1 = self._parallel_block( + i, lane0, lane1, x0, q_w, k_w, v_w, out_w, up_w, down_w, + cu_seqlens=cu_seqlens, max_seqlen=max_seqlen, + ) + else: + if skip_idx < self.num_skip_weights and skips: + scaled_skip = ( + self.skip_weights[skip_idx].to(dtype=x.dtype)[None, None, :] + * skips.pop() + ) + if self.skip_gates is not None: + g = torch.sigmoid(self.skip_gates[skip_idx].to(dtype=x.dtype))[None, None, :] + x = torch.lerp(scaled_skip, x, g) + else: + x = x + scaled_skip + x = self.blocks[i](x, x0, q_w, k_w, v_w, out_w, up_w, down_w, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen) + if lane0 is not None: + x = self._final_parallel_hidden(lane0, lane1) + x = self.final_norm(x) + if self.tie_embeddings: + logits_proj = F.linear(x, self.tok_emb.weight) + else: + logits_proj = self.lm_head(x) + return self.logit_softcap * torch.tanh(logits_proj / self.logit_softcap) + + def forward(self, input_ids, target_ids, cu_seqlens=None, max_seqlen=0): + logits = self.forward_logits( + input_ids, cu_seqlens=cu_seqlens, max_seqlen=max_seqlen + ) + return F.cross_entropy( + logits.reshape(-1, logits.size(-1)).float(), + target_ids.reshape(-1), + reduction="mean", + ) + + def forward_ttt(self, input_ids, target_ids, lora): + x = self.tok_emb(input_ids) + if self.smear_gate_enabled and x.dim() == 3 and x.size(1) > 1: + sl = self.smear_lambda.to(dtype=x.dtype) + g = sl * torch.sigmoid(self.smear_gate(x[:, 1:, :self.smear_width])) + x = torch.cat([x[:, :1], x[:, 1:] + g * x[:, :-1]], dim=1) + x = F.rms_norm(x, (x.size(-1),)) + x0 = x + skips = [] + enc_iter = ( + self.encoder_indices + if self.looping_active + else list(range(self.num_encoder_layers)) + ) + dec_iter = ( + self.decoder_indices + if self.looping_active + else list( + range( + self.num_encoder_layers, + self.num_encoder_layers + self.num_decoder_layers, + ) + ) + ) + slot = 0 + for i in enc_iter: + q_w, k_w, v_w, out_w, up_w, down_w = self._bank_weights(i) + x = self._block_with_lora(self.blocks[i], x, x0, lora, slot, q_w, k_w, v_w, out_w, up_w, down_w) + slot += 1 + skips.append(x) + psl = self.parallel_start_layer + lane0 = None + lane1 = None + for skip_idx, i in enumerate(dec_iter): + q_w, k_w, v_w, out_w, up_w, down_w = self._bank_weights(i) + if i >= psl and psl > 0: + if lane0 is None: + lane0 = x + lane1 = x + if skip_idx < self.num_skip_weights and skips: + skip = skips.pop() + w = self.skip_weights[skip_idx].to(dtype=lane0.dtype)[None, None, :] + if self.skip_gates is not None: + g = torch.sigmoid(self.skip_gates[skip_idx].to(dtype=lane0.dtype))[None, None, :] + lane0 = torch.lerp(w * skip, lane0, g) + else: + lane0 = lane0 + w * skip + lane0, lane1 = self._parallel_block_with_lora( + i, lane0, lane1, x0, lora, slot, + q_w, k_w, v_w, out_w, up_w, down_w, + ) + else: + if skip_idx < self.num_skip_weights and skips: + scaled_skip = ( + self.skip_weights[skip_idx].to(dtype=x.dtype)[None, None, :] + * skips.pop() + ) + if self.skip_gates is not None: + g = torch.sigmoid(self.skip_gates[skip_idx].to(dtype=x.dtype))[None, None, :] + x = torch.lerp(scaled_skip, x, g) + else: + x = x + scaled_skip + x = self._block_with_lora(self.blocks[i], x, x0, lora, slot, q_w, k_w, v_w, out_w, up_w, down_w) + slot += 1 + if lane0 is not None: + x = self._final_parallel_hidden(lane0, lane1) + x = self.final_norm(x) + if self.tie_embeddings: + logits = F.linear(x, self.tok_emb.weight) + else: + logits = self.lm_head(x) + logits = logits + lora.lm_head_lora(x) + logits = self.logit_softcap * torch.tanh(logits / self.logit_softcap) + bsz, sl, V = logits.shape + return F.cross_entropy( + logits.float().reshape(-1, V), target_ids.reshape(-1), reduction="none" + ).reshape(bsz, sl) + + def _block_with_lora(self, block, x, x0, lora, slot, q_w, k_w, v_w, out_w, up_w, down_w): + mix = block.resid_mix.to(dtype=x.dtype) + x_in = mix[0][None, None, :] * x + mix[1][None, None, :] * x0 + n = block.attn_norm(x_in) * block.ln_scale_factor + attn = block.attn + bsz, seqlen, dim = n.shape + q = (F.linear(n, q_w.to(n.dtype)) + lora.q_loras[slot](n)).reshape( + bsz, seqlen, attn.num_heads, attn.head_dim + ) + k = F.linear(n, k_w.to(n.dtype)) + if lora.k_loras is not None: + k = k + lora.k_loras[slot](n) + k = k.reshape(bsz, seqlen, attn.num_kv_heads, attn.head_dim) + v = (F.linear(n, v_w.to(n.dtype)) + lora.v_loras[slot](n)).reshape( + bsz, seqlen, attn.num_kv_heads, attn.head_dim + ) + q = F.rms_norm(q, (q.size(-1),)) + k = F.rms_norm(k, (k.size(-1),)) + cos, sin = attn.rotary(seqlen, n.device, q.dtype) + q = apply_rotary_emb(q, cos, sin, attn.rope_dims) + k = apply_rotary_emb(k, cos, sin, attn.rope_dims) + q = q * attn.q_gain.to(dtype=q.dtype)[None, None, :, None] + y = flash_attn_3_func(q, k, v, causal=True) + if attn.use_xsa: + y = attn._xsa_efficient(y, v) + if attn.gate_attn_out: + g = 2.0 * torch.sigmoid(attn.attn_gate_proj(n[:, :, :attn.gate_attn_width])) + y = y * g.unsqueeze(-1) + y = y.reshape(bsz, seqlen, dim) + attn_out = F.linear(y, out_w.to(n.dtype)) + if lora.o_loras is not None: + attn_out = attn_out + lora.o_loras[slot](n) + x_out = x_in + block.attn_scale.to(dtype=x_in.dtype)[None, None, :] * attn_out + mlp_n = block.mlp_norm(x_out) * block.ln_scale_factor + mlp_out = block.mlp(mlp_n, up_w, down_w) + if lora.mlp_loras is not None: + mlp_out = mlp_out + lora.mlp_loras[slot](mlp_n) + x_out = x_out + block.mlp_scale.to(dtype=x_out.dtype)[None, None, :] * mlp_out + return x_out + + def _parallel_block_with_lora( + self, block_idx, lane0, lane1, x0, lora, slot, + q_w, k_w, v_w, out_w, up_w, down_w, + ): + block = self.blocks[block_idx] + mix = block.resid_mix.to(dtype=lane0.dtype) + attn_read = mix[0][None, None, :] * lane0 + mix[1][None, None, :] * x0 + n = block.attn_norm(attn_read) * block.ln_scale_factor + attn = block.attn + bsz, seqlen, dim = n.shape + q = (F.linear(n, q_w.to(n.dtype)) + lora.q_loras[slot](n)).reshape( + bsz, seqlen, attn.num_heads, attn.head_dim + ) + k = F.linear(n, k_w.to(n.dtype)) + if lora.k_loras is not None: + k = k + lora.k_loras[slot](n) + k = k.reshape(bsz, seqlen, attn.num_kv_heads, attn.head_dim) + v = (F.linear(n, v_w.to(n.dtype)) + lora.v_loras[slot](n)).reshape( + bsz, seqlen, attn.num_kv_heads, attn.head_dim + ) + q = F.rms_norm(q, (q.size(-1),)) + k = F.rms_norm(k, (k.size(-1),)) + cos, sin = attn.rotary(seqlen, n.device, q.dtype) + q = apply_rotary_emb(q, cos, sin, attn.rope_dims) + k = apply_rotary_emb(k, cos, sin, attn.rope_dims) + q = q * attn.q_gain.to(dtype=q.dtype)[None, None, :, None] + y = flash_attn_3_func(q, k, v, causal=True) + if attn.use_xsa: + y = attn._xsa_efficient(y, v) + if attn.gate_attn_out: + g = 2.0 * torch.sigmoid(attn.attn_gate_proj(n[:, :, :attn.gate_attn_width])) + y = y * g.unsqueeze(-1) + y = y.reshape(bsz, seqlen, dim) + attn_out = F.linear(y, out_w.to(n.dtype)) + if lora.o_loras is not None: + attn_out = attn_out + lora.o_loras[slot](n) + attn_out = block.attn_scale.to(dtype=attn_out.dtype)[None, None, :] * attn_out + mlp_read = lane1 + mlp_n = block.mlp_norm(mlp_read) * block.ln_scale_factor + mlp_out = block.mlp(mlp_n, up_w, down_w) + if lora.mlp_loras is not None: + mlp_out = mlp_out + lora.mlp_loras[slot](mlp_n) + mlp_out = block.mlp_scale.to(dtype=lane1.dtype)[None, None, :] * mlp_out + attn_resid = self.parallel_resid_lambdas[block_idx, 0].to(dtype=lane0.dtype) + attn_post = self.parallel_post_lambdas[block_idx, 0].to(dtype=lane0.dtype) + mlp_resid = self.parallel_resid_lambdas[block_idx, 1].to(dtype=lane0.dtype) + mlp_post = self.parallel_post_lambdas[block_idx, 1].to(dtype=lane0.dtype) + lane0 = attn_resid * lane0 + attn_post[0] * attn_out + mlp_post[0] * mlp_out + lane1 = mlp_resid * lane1 + attn_post[1] * attn_out + mlp_post[1] * mlp_out + return lane0, lane1 + + +class BatchedLinearLoRA(nn.Module): + # Novel: rank-scaled output (alpha/rank), like standard LoRA. Decouples + # effective magnitude from rank so changing rank does not change LR scale. + _ALPHA = float(os.environ.get("TTT_LORA_ALPHA", "144")) + + def __init__(self, bsz, in_features, out_features, rank): + super().__init__() + self._bound = 1.0 / math.sqrt(in_features) + self._scale = self._ALPHA / rank + self.A = nn.Parameter( + torch.empty(bsz, rank, in_features).uniform_(-self._bound, self._bound) + ) + self.B = nn.Parameter(torch.zeros(bsz, out_features, rank)) + + _WARM_START_A = bool(int(os.environ.get("TTT_WARM_START_A", "1"))) + + def reset(self): + with torch.no_grad(): + # Novel: optionally keep A warm across batch resets (accumulates feature directions). + if not self._WARM_START_A: + self.A.uniform_(-self._bound, self._bound) + self.B.zero_() + + def forward(self, x): + return ((x @ self.A.transpose(1, 2)) @ self.B.transpose(1, 2)) * self._scale + + +class BatchedTTTLoRA(nn.Module): + def __init__(self, bsz, model, rank, k_lora=True, mlp_lora=True, o_lora=True): + super().__init__() + self.bsz = bsz + dim = model.qo_bank.shape[-1] + vocab = model.tok_emb.num_embeddings + if getattr(model, "looping_active", False): + num_slots = len(model.encoder_indices) + len(model.decoder_indices) + else: + num_slots = len(model.blocks) + kv_dim = model.blocks[0].attn.num_kv_heads * ( + dim // model.blocks[0].attn.num_heads + ) + embed_dim = model.tok_emb.embedding_dim + self.lm_head_lora = BatchedLinearLoRA(bsz, embed_dim, vocab, rank) + self.q_loras = nn.ModuleList( + [BatchedLinearLoRA(bsz, dim, dim, rank) for _ in range(num_slots)] + ) + self.v_loras = nn.ModuleList( + [BatchedLinearLoRA(bsz, dim, kv_dim, rank) for _ in range(num_slots)] + ) + self.k_loras = ( + nn.ModuleList( + [BatchedLinearLoRA(bsz, dim, kv_dim, rank) for _ in range(num_slots)] + ) + if k_lora + else None + ) + self.mlp_loras = ( + nn.ModuleList( + [BatchedLinearLoRA(bsz, dim, dim, rank) for _ in range(num_slots)] + ) + if mlp_lora + else None + ) + self.o_loras = ( + nn.ModuleList( + [BatchedLinearLoRA(bsz, dim, dim, rank) for _ in range(num_slots)] + ) + if o_lora + else None + ) + + def reset(self): + with torch.no_grad(): + self.lm_head_lora.reset() + for loras in [self.q_loras, self.v_loras, self.k_loras, + self.mlp_loras, self.o_loras]: + if loras is not None: + for lora in loras: + lora.reset() + + +# Polar Express per-iteration minimax Newton-Schulz coefficients (PR #1344). +# Replaces the fixed (3.4445, -4.775, 2.0315) coefficients when +# POLAR_EXPRESS_NS=1 (default). Applied at backend_steps=5. +_PE_COEFFS = ( + (8.156554524902461, -22.48329292557795, 15.878769915207462), + (4.042929935166739, -2.808917465908714, 0.5000178451051316), + (3.8916678022926607, -2.772484153217685, 0.5060648178503393), + (3.285753657755655, -2.3681294933425376, 0.46449024233003106), + (2.3465413258596377, -1.7097828382687081, 0.42323551169305323), +) +_POLAR_EXPRESS_NS = bool(int(os.environ.get("POLAR_EXPRESS_NS", "1"))) + + +@torch.compile +def zeropower_via_newtonschulz5(G, steps=10, eps=1e-07): + was_2d = G.ndim == 2 + if was_2d: + G = G.unsqueeze(0) + X = G.bfloat16() + transposed = X.size(-2) > X.size(-1) + if transposed: + X = X.mT + X = X / (X.norm(dim=(-2, -1), keepdim=True) + eps) + if _POLAR_EXPRESS_NS: + coeffs = _PE_COEFFS[:steps] if steps <= len(_PE_COEFFS) else _PE_COEFFS + for a, b, c in coeffs: + A = X @ X.mT + B = b * A + c * (A @ A) + X = a * X + B @ X + else: + a, b, c = 3.4445, -4.775, 2.0315 + for _ in range(steps): + A = X @ X.mT + B = b * A + c * (A @ A) + X = a * X + B @ X + if transposed: + X = X.mT + if was_2d: + X = X.squeeze(0) + return X + + +_NM_PARAM_TO_MODULE = {} +_NM_HANDLES = [] +_NM_STEP_COUNTER = [0] + + +def _nm_install_hooks(model, capture_every=1): + """Install forward-pre hooks on every CastedLinear so we can accumulate Z^T@Z for Newton-Muon preconditioning. Idempotent.""" + for h in _NM_HANDLES: + h.remove() + _NM_HANDLES.clear() + _NM_PARAM_TO_MODULE.clear() + _NM_STEP_COUNTER[0] = 0 + + def make_hook(m): + def hook(module, args): + if _NM_STEP_COUNTER[0] % capture_every != 0: + return + x = args[0] + x_flat = x.reshape(-1, x.size(-1)).float() + outer = x_flat.transpose(0, 1) @ x_flat + n = x_flat.size(0) + if getattr(module, "_nm_K_acc", None) is None or module._nm_K_acc.shape != outer.shape: + module._nm_K_acc = outer.detach() + module._nm_K_count = n + else: + module._nm_K_acc.add_(outer.detach()) + module._nm_K_count += n + return hook + + count = 0 + for name, mod in model.named_modules(): + if isinstance(mod, CastedLinear) and mod.weight.ndim == 2: + mod._nm_K_acc = None + mod._nm_K_count = 0 + _NM_PARAM_TO_MODULE[id(mod.weight)] = mod + _NM_HANDLES.append(mod.register_forward_pre_hook(make_hook(mod))) + count += 1 + log(f"newton_muon: installed {count} forward-pre hooks (capture_every={capture_every})") + + +def _nm_uninstall_hooks(): + for h in _NM_HANDLES: + h.remove() + _NM_HANDLES.clear() + _NM_PARAM_TO_MODULE.clear() + + +def _nm_advance_step(): + _NM_STEP_COUNTER[0] += 1 + + +class Muon(torch.optim.Optimizer): + def __init__( + self, + params, + lr, + momentum, + backend_steps, + nesterov=True, + weight_decay=0.0, + row_normalize=False, + newton_muon=False, + nm_beta=0.95, + nm_gamma=0.2, + nm_k=32, + nm_warmup=100, + nm_all_reduce_k=True, + ): + super().__init__( + params, + dict( + lr=lr, + momentum=momentum, + backend_steps=backend_steps, + nesterov=nesterov, + weight_decay=weight_decay, + row_normalize=row_normalize, + ), + ) + self._built = False + self.newton_muon = newton_muon + self.nm_beta = nm_beta + self.nm_gamma = nm_gamma + self.nm_k = nm_k + self.nm_warmup = nm_warmup + self.nm_all_reduce_k = nm_all_reduce_k + + def _build(self): + self._distributed = dist.is_available() and dist.is_initialized() + self._world_size = dist.get_world_size() if self._distributed else 1 + self._rank = dist.get_rank() if self._distributed else 0 + ws = self._world_size + self._bank_meta = [] + for group in self.param_groups: + for p in group["params"]: + B = p.shape[0] + padded_B = ((B + ws - 1) // ws) * ws + shard_B = padded_B // ws + tail = p.shape[1:] + dev = p.device + self._bank_meta.append({ + "p": p, + "B": B, + "padded_grad": torch.zeros(padded_B, *tail, device=dev, dtype=torch.bfloat16), + "shard": torch.zeros(shard_B, *tail, device=dev, dtype=torch.bfloat16), + "shard_mom": torch.zeros(shard_B, *tail, device=dev, dtype=torch.bfloat16), + "full_update": torch.zeros(padded_B, *tail, device=dev, dtype=torch.bfloat16), + "scale": max(1, p.shape[-2] / p.shape[-1]) ** 0.5, + }) + self._bank_meta.sort(key=lambda m: -m["p"].numel()) + self._built = True + + def _nm_precondition(self): + """Apply Newton-Muon right-preconditioning g <- g @ K_inv on each param's gradient before reduce_scatter.""" + if _NM_STEP_COUNTER[0] < self.nm_warmup: + return + for m in self._bank_meta: + p = m["p"] + if p.grad is None: + continue + mod = _NM_PARAM_TO_MODULE.get(id(p)) + if mod is None or getattr(mod, "_nm_K_acc", None) is None or mod._nm_K_count <= 0: + continue + state = self.state[p] + k_inst = mod._nm_K_acc / mod._nm_K_count + if self.nm_all_reduce_k and dist.is_available() and dist.is_initialized(): + dist.all_reduce(k_inst, op=dist.ReduceOp.AVG) + if "nm_K" not in state: + state["nm_K"] = k_inst.float().clone() + state["nm_K_inv"] = None + state["nm_step"] = 0 + else: + state["nm_K"].mul_(self.nm_beta).add_(k_inst.float(), alpha=1.0 - self.nm_beta) + if state["nm_K_inv"] is None or state["nm_step"] % self.nm_k == 0: + K = state["nm_K"] + n = K.size(0) + gamma_eff = self.nm_gamma * K.diagonal().mean() / max(n, 1) + I = torch.eye(n, device=K.device, dtype=K.dtype) + state["nm_K_inv"] = torch.linalg.inv(K + gamma_eff * I) + state["nm_step"] += 1 + K_inv = state["nm_K_inv"].to(p.grad.dtype) + p.grad = p.grad @ K_inv + mod._nm_K_acc.zero_() + mod._nm_K_count = 0 + + def launch_reduce_scatters(self): + if not self._built: + self._build() + if self.newton_muon: + self._nm_precondition() + if not self._distributed: + return + self._rs_futures = [] + for m in self._bank_meta: + p = m["p"] + if p.grad is None: + self._rs_futures.append(None) + continue + pg = m["padded_grad"] + pg[: m["B"]].copy_(p.grad.bfloat16()) + if pg.shape[0] > m["B"]: + pg[m["B"] :].zero_() + fut = dist.reduce_scatter_tensor( + m["shard"], pg, op=dist.ReduceOp.AVG, async_op=True + ) + self._rs_futures.append(fut) + + @torch.no_grad() + def step(self, closure=None): + loss = None + if closure is not None: + with torch.enable_grad(): + loss = closure() + if not self._built: + self._build() + for group in self.param_groups: + lr = group["lr"] + momentum = group["momentum"] + backend_steps = group["backend_steps"] + nesterov = group["nesterov"] + wd = group.get("weight_decay", 0.0) + row_normalize = group.get("row_normalize", False) + prev_ag_handle = None + prev_m = None + sharded = self._distributed and hasattr(self, "_rs_futures") + for idx, m in enumerate(self._bank_meta): + p = m["p"] + if p.grad is None: + continue + if prev_ag_handle is not None: + prev_ag_handle.wait() + pp = prev_m["p"] + upd = prev_m["full_update"][: prev_m["B"]] + if wd > 0.0: + pp.data.mul_(1.0 - lr * wd) + pp.add_(upd.to(dtype=pp.dtype), alpha=-lr * prev_m["scale"]) + if sharded and self._rs_futures[idx] is not None: + self._rs_futures[idx].wait() + g = m["shard"] + buf = m["shard_mom"] + else: + g = p.grad.bfloat16() + state = self.state[p] + if "momentum_buffer" not in state: + state["momentum_buffer"] = torch.zeros_like(g) + buf = state["momentum_buffer"] + buf.mul_(momentum).add_(g) + if nesterov: + update = g.add(buf, alpha=momentum) + else: + update = buf + if row_normalize: + rn = update.float().norm(dim=-1, keepdim=True).clamp_min(1e-07) + update = update / rn.to(update.dtype) + update = zeropower_via_newtonschulz5(update, steps=backend_steps) + if sharded: + prev_ag_handle = dist.all_gather_into_tensor( + m["full_update"], update, async_op=True + ) + prev_m = m + else: + if wd > 0.0: + p.data.mul_(1.0 - lr * wd) + p.add_(update.to(dtype=p.dtype), alpha=-lr * m["scale"]) + if prev_ag_handle is not None: + prev_ag_handle.wait() + pp = prev_m["p"] + upd = prev_m["full_update"][: prev_m["B"]] + if wd > 0.0: + pp.data.mul_(1.0 - lr * wd) + pp.add_(upd.to(dtype=pp.dtype), alpha=-lr * prev_m["scale"]) + if hasattr(self, "_rs_futures"): + del self._rs_futures + return loss + + +CONTROL_TENSOR_NAME_PATTERNS = tuple( + pattern + for pattern in os.environ.get( + "CONTROL_TENSOR_NAME_PATTERNS", + "attn_scale,attn_scales,mlp_scale,mlp_scales,resid_mix,resid_mixes,q_gain,skip_weight,skip_weights,skip_gates,parallel_post_lambdas,parallel_resid_lambdas,attn_gate_proj", + ).split(",") + if pattern +) + + +PACKED_REPLICATED_GRAD_MAX_NUMEL = 1 << 15 + + +class Optimizers: + def __init__(self, h, base_model): + matrix_params = [ + base_model.qo_bank, + base_model.kv_bank, + base_model.mlp_up_bank, + base_model.mlp_down_bank, + ] + block_named_params = list(base_model.blocks.named_parameters()) + scalar_params = [ + p + for (name, p) in block_named_params + if p.ndim < 2 + or any(pattern in name for pattern in CONTROL_TENSOR_NAME_PATTERNS) + ] + if base_model.skip_weights.numel() > 0: + scalar_params.append(base_model.skip_weights) + if base_model.skip_gates is not None and base_model.skip_gates.numel() > 0: + scalar_params.append(base_model.skip_gates) + if base_model.parallel_post_lambdas is not None: + scalar_params.append(base_model.parallel_post_lambdas) + if base_model.parallel_resid_lambdas is not None: + scalar_params.append(base_model.parallel_resid_lambdas) + if getattr(base_model, "smear_gate_enabled", False): + scalar_params.append(base_model.smear_gate.weight) + scalar_params.append(base_model.smear_lambda) + token_lr = h.tied_embed_lr if h.tie_embeddings else h.embed_lr + tok_params = [ + {"params": [base_model.tok_emb.weight], "lr": token_lr, "base_lr": token_lr} + ] + self.optimizer_tok = torch.optim.AdamW( + tok_params, + betas=(h.beta1, h.beta2), + eps=h.adam_eps, + weight_decay=h.embed_wd, + fused=True, + ) + self.optimizer_muon = Muon( + matrix_params, + lr=h.matrix_lr, + momentum=h.muon_momentum, + backend_steps=h.muon_backend_steps, + weight_decay=h.muon_wd, + row_normalize=h.muon_row_normalize, + newton_muon=h.newton_muon_enabled, + nm_beta=h.newton_muon_beta, + nm_gamma=h.newton_muon_gamma, + nm_k=h.newton_muon_k_refresh, + nm_warmup=h.newton_muon_warmup, + nm_all_reduce_k=h.newton_muon_all_reduce_k, + ) + for group in self.optimizer_muon.param_groups: + group["base_lr"] = h.matrix_lr + self.optimizer_scalar = torch.optim.AdamW( + [{"params": scalar_params, "lr": h.scalar_lr, "base_lr": h.scalar_lr}], + betas=(h.beta1, h.beta2), + eps=h.adam_eps, + weight_decay=h.adam_wd, + fused=True, + ) + self.optimizers = [ + self.optimizer_tok, + self.optimizer_muon, + self.optimizer_scalar, + ] + self.replicated_params = list(tok_params[0]["params"]) + self.replicated_params.extend(scalar_params) + self.replicated_large_params = [] + self.replicated_packed_params = [] + for p in self.replicated_params: + if p.numel() <= PACKED_REPLICATED_GRAD_MAX_NUMEL: + self.replicated_packed_params.append(p) + else: + self.replicated_large_params.append(p) + + def __iter__(self): + return iter(self.optimizers) + + def zero_grad_all(self): + for opt in self.optimizers: + opt.zero_grad(set_to_none=True) + + def _all_reduce_packed_grads(self): + grads_by_key = collections.defaultdict(list) + for p in self.replicated_packed_params: + if p.grad is not None: + grads_by_key[(p.grad.device, p.grad.dtype)].append(p.grad) + for grads in grads_by_key.values(): + flat = torch.empty( + sum(g.numel() for g in grads), + device=grads[0].device, + dtype=grads[0].dtype, + ) + offset = 0 + for g in grads: + n = g.numel() + flat[offset : offset + n].copy_(g.contiguous().view(-1)) + offset += n + dist.all_reduce(flat, op=dist.ReduceOp.AVG) + offset = 0 + for g in grads: + n = g.numel() + g.copy_(flat[offset : offset + n].view_as(g)) + offset += n + + def step(self, distributed=False): + self.optimizer_muon.launch_reduce_scatters() + if distributed: + reduce_handles = [ + dist.all_reduce(p.grad, op=dist.ReduceOp.AVG, async_op=True) + for p in self.replicated_large_params + if p.grad is not None + ] + self._all_reduce_packed_grads() + for handle in reduce_handles: + handle.wait() + self.optimizer_tok.step() + self.optimizer_scalar.step() + self.optimizer_muon.step() + self.zero_grad_all() + + +def restore_fp32_params(model): + for module in model.modules(): + if isinstance(module, CastedLinear): + module.float() + for name, param in model.named_parameters(): + if ( + param.ndim < 2 + or any(pattern in name for pattern in CONTROL_TENSOR_NAME_PATTERNS) + ) and param.dtype != torch.float32: + param.data = param.data.float() + if hasattr(model, "qo_bank") and model.qo_bank is not None: + model.qo_bank.data = model.qo_bank.data.float() + model.kv_bank.data = model.kv_bank.data.float() + model.mlp_up_bank.data = model.mlp_up_bank.data.float() + model.mlp_down_bank.data = model.mlp_down_bank.data.float() + + +def collect_hessians(model, train_loader, h, device, n_calibration_batches=64): + hessians = {} + hooks = [] + for i, block in enumerate(model.blocks): + block.attn._calib = True + block.mlp._calib = True + block.mlp.use_fused = False + + def make_attn_hook(layer_idx): + def hook_fn(module, inp, out): + x = inp[0].detach().float() + if x.ndim == 3: + x = x.reshape(-1, x.shape[-1]) + for suffix in ["c_q", "c_k", "c_v"]: + name = f"blocks.{layer_idx}.attn.{suffix}.weight" + if name not in hessians: + hessians[name] = torch.zeros( + x.shape[1], x.shape[1], dtype=torch.float32, device=device + ) + hessians[name].addmm_(x.T, x) + y = module._last_proj_input + if y is not None: + y = y.float() + if y.ndim == 3: + y = y.reshape(-1, y.shape[-1]) + name = f"blocks.{layer_idx}.attn.proj.weight" + if name not in hessians: + hessians[name] = torch.zeros( + y.shape[1], y.shape[1], dtype=torch.float32, device=device + ) + hessians[name].addmm_(y.T, y) + return hook_fn + + def make_mlp_hook(layer_idx): + def hook_fn(module, inp, out): + x = inp[0].detach().float() + if x.ndim == 3: + x = x.reshape(-1, x.shape[-1]) + name = f"blocks.{layer_idx}.mlp.fc.weight" + if name not in hessians: + hessians[name] = torch.zeros( + x.shape[1], x.shape[1], dtype=torch.float32, device=device + ) + hessians[name].addmm_(x.T, x) + h_act = module._last_down_input + if h_act is not None: + h_act = h_act.float() + if h_act.ndim == 3: + h_act = h_act.reshape(-1, h_act.shape[-1]) + name = f"blocks.{layer_idx}.mlp.proj.weight" + if name not in hessians: + hessians[name] = torch.zeros( + h_act.shape[1], h_act.shape[1], dtype=torch.float32, device=device + ) + hessians[name].addmm_(h_act.T, h_act) + return hook_fn + + for i, block in enumerate(model.blocks): + hooks.append(block.attn.register_forward_hook(make_attn_hook(i))) + hooks.append(block.mlp.register_forward_hook(make_mlp_hook(i))) + + # Hessian hooks for embedding factorization projection layers + def make_linear_input_hook(weight_name): + def hook_fn(module, inp, out): + x = inp[0].detach().float() + if x.ndim == 3: + x = x.reshape(-1, x.shape[-1]) + if weight_name not in hessians: + hessians[weight_name] = torch.zeros( + x.shape[1], x.shape[1], dtype=torch.float32, device=device + ) + hessians[weight_name].addmm_(x.T, x) + return hook_fn + + if model.tie_embeddings: + hook_module = model.final_norm + + def make_output_hook(name): + def hook_fn(module, inp, out): + x = out.detach().float() + if x.ndim == 3: + x = x.reshape(-1, x.shape[-1]) + if name not in hessians: + hessians[name] = torch.zeros( + x.shape[1], x.shape[1], dtype=torch.float32, device=device + ) + hessians[name].addmm_(x.T, x) + return hook_fn + + hooks.append( + hook_module.register_forward_hook(make_output_hook("tok_emb.weight")) + ) + model.eval() + with torch.no_grad(): + for _ in range(n_calibration_batches): + x, _ = train_loader.next_batch(h.train_batch_tokens, h.grad_accum_steps) + model.forward_logits(x) + for hook in hooks: + hook.remove() + for i, block in enumerate(model.blocks): + block.attn._calib = False + block.mlp._calib = False + block.mlp.use_fused = True + for name in hessians: + hessians[name] = hessians[name].cpu() / n_calibration_batches + return hessians + + +def gptq_quantize_weight(w, H, clip_sigmas=3.0, clip_range=63, block_size=128): + W_orig = w.float().clone() + rows, cols = W_orig.shape + H = H.float().clone() + dead = torch.diag(H) == 0 + H[dead, dead] = 1 + damp = 0.01 * H.diag().mean() + H.diagonal().add_(damp) + perm = torch.argsort(H.diag(), descending=True) + invperm = torch.argsort(perm) + W_perm = W_orig[:, perm].clone() + W_perm[:, dead[perm]] = 0 + H = H[perm][:, perm] + Hinv = torch.cholesky_inverse(torch.linalg.cholesky(H)) + Hinv = torch.linalg.cholesky(Hinv, upper=True) + row_std = W_orig.std(dim=1) + s = (clip_sigmas * row_std / clip_range).clamp_min(1e-10).to(torch.float16) + sf = s.float() + Q = torch.zeros(rows, cols, dtype=torch.int8) + W_work = W_perm.clone() + for i1 in range(0, cols, block_size): + i2 = min(i1 + block_size, cols) + W_block = W_work[:, i1:i2].clone() + Hinv_block = Hinv[i1:i2, i1:i2] + Err = torch.zeros(rows, i2 - i1) + for j in range(i2 - i1): + w_col = W_block[:, j] + d = Hinv_block[j, j] + q_col = torch.clamp(torch.round(w_col / sf), -clip_range, clip_range) + Q[:, i1 + j] = q_col.to(torch.int8) + err = (w_col - q_col.float() * sf) / d + Err[:, j] = err + W_block[:, j:] -= err.unsqueeze(1) * Hinv_block[j, j:].unsqueeze(0) + if i2 < cols: + W_work[:, i2:] -= Err @ Hinv[i1:i2, i2:] + return Q[:, invperm], s + + +def _lqer_pack(A, B, bits): + rng = 2 ** (bits - 1) - 1 + sA = (A.abs().amax(dim=1).clamp_min(1e-10) / rng).to(torch.float16) + sB = (B.abs().amax(dim=1).clamp_min(1e-10) / rng).to(torch.float16) + qA = torch.clamp(torch.round(A / sA.float().view(-1, 1)), -rng, rng).to(torch.int8) + qB = torch.clamp(torch.round(B / sB.float().view(-1, 1)), -rng, rng).to(torch.int8) + return qA, sA, qB, sB + + +def _lqer_pack_asym(A, B, g=64): + # A: INT2 per-matrix scalar (signed [-2,1], scale = |A|max/1.5). + sA = (A.abs().amax().clamp_min(1e-10) / 1.5).to(torch.float16) + qA = torch.clamp(torch.round(A / sA.float()), -2, 1).to(torch.int8) + # B: INT4 groupwise g over flattened B (signed [-8,7], per-group scale). + Bf = B.reshape(-1, g) + Bmax = Bf.abs().amax(dim=-1, keepdim=True).clamp_min(1e-10) + sB = (Bmax / 7.5).to(torch.float16).reshape(-1) + qB = torch.clamp(torch.round(Bf / sB.float().reshape(-1, 1)), -8, 7).to( + torch.int8 + ).reshape(B.shape) + return qA, sA, qB, sB + + +def gptq_mixed_quantize(state_dict, hessians, h): + result = {} + meta = {} + lqer_on = bool(getattr(h, "lqer_enabled", False)) + lqer_cands = {} + for (name, tensor) in state_dict.items(): + t = tensor.detach().cpu().contiguous() + if not t.is_floating_point() or t.numel() <= 65536: + result[name] = t.to(torch.float16) if t.is_floating_point() else t + meta[name] = "passthrough (float16)" + continue + if "tok_emb" in name: + cs = h.embed_clip_sigmas + elif ".mlp." in name: + cs = h.mlp_clip_sigmas + elif ".attn." in name: + cs = h.attn_clip_sigmas + else: + cs = h.matrix_clip_sigmas + bits = h.embed_bits if "tok_emb" in name else h.matrix_bits + clip_range = 2 ** (bits - 1) - 1 + ret = gptq_quantize_weight( + t, hessians[name], clip_sigmas=cs, clip_range=clip_range + ) + q, s = ret + result[name + ".q"] = q + result[name + ".scale"] = s + meta[name] = f"gptq (int{bits})" + if lqer_on: + W_q = q.float() * s.float().view(-1, 1) + E = t.float() - W_q + lqer_cands[name] = (E, float(E.norm())) + if lqer_on and lqer_cands: + top = sorted(lqer_cands.items(), key=lambda kv: -kv[1][1])[: h.lqer_top_k] + asym_on = bool(getattr(h, "lqer_asym_enabled", False)) + asym_g = int(getattr(h, "lqer_asym_group", 64)) + for (name, (E, _)) in top: + U, S, Vh = torch.linalg.svd(E, full_matrices=False) + r = min(h.lqer_rank, S.numel()) + A = (U[:, :r] * S[:r]).contiguous() + B = Vh[:r, :].contiguous() + if asym_on and B.numel() % asym_g == 0: + qA, sA, qB, sB = _lqer_pack_asym(A, B, asym_g) + result[name + ".lqA_a"] = qA + result[name + ".lqAs_a"] = sA + result[name + ".lqB_a"] = qB + result[name + ".lqBs_a"] = sB + meta[name] = meta[name] + "+lqer_asym" + else: + qA, sA, qB, sB = _lqer_pack(A, B, h.lqer_factor_bits) + result[name + ".lqA"] = qA + result[name + ".lqAs"] = sA + result[name + ".lqB"] = qB + result[name + ".lqBs"] = sB + meta[name] = meta[name] + "+lqer" + categories = collections.defaultdict(set) + for (name, cat) in meta.items(): + short = re.sub("\\.\\d+$", "", re.sub("blocks\\.\\d+", "blocks", name)) + categories[cat].add(short) + log("Quantized weights:") + for cat in sorted(categories): + log(f" {cat}: {', '.join(sorted(categories[cat]))}") + return result, meta + + +def dequantize_mixed(result, meta, template_sd): + out = {} + for (name, orig) in template_sd.items(): + info = meta.get(name) + if info is None: + continue + orig_dtype = orig.dtype + if "passthrough" in info: + t = result[name] + if t.dtype == torch.float16 and orig_dtype in ( + torch.float32, + torch.bfloat16, + ): + t = t.to(orig_dtype) + out[name] = t + continue + q, s = result[name + ".q"], result[name + ".scale"] + if s.ndim > 0: + W = q.float() * s.float().view(q.shape[0], *[1] * (q.ndim - 1)) + else: + W = q.float() * float(s.item()) + if "lqer_asym" in info: + qA_t = result[name + ".lqA_a"] + sA_t = result[name + ".lqAs_a"] + qB_t = result[name + ".lqB_a"] + sB_t = result[name + ".lqBs_a"] + qA = qA_t.float() * float(sA_t) + g_sz = qB_t.numel() // sB_t.numel() + qB = (qB_t.reshape(-1, g_sz).float() * sB_t.float().view(-1, 1)).reshape( + qB_t.shape + ) + W = W + qA @ qB + elif "lqer" in info: + qA = result[name + ".lqA"].float() * result[name + ".lqAs"].float().view(-1, 1) + qB = result[name + ".lqB"].float() * result[name + ".lqBs"].float().view(-1, 1) + W = W + qA @ qB + out[name] = W.to(orig_dtype) + return out + + +_BSHF_MAGIC = b"BSHF" + + +def _byte_shuffle(data, stride=2): + if stride <= 1 or len(data) < stride: + return data + src = np.frombuffer(data, dtype=np.uint8) + n = len(src) + out = np.empty(n, dtype=np.uint8) + dest_off = 0 + for pos in range(stride): + chunk = src[pos::stride] + out[dest_off : dest_off + len(chunk)] = chunk + dest_off += len(chunk) + return _BSHF_MAGIC + bytes([stride]) + out.tobytes() + + +def _byte_unshuffle(data): + if len(data) < 5 or data[:4] != _BSHF_MAGIC: + return data + stride = data[4] + if stride < 2: + return data[5:] + payload = np.frombuffer(data, dtype=np.uint8, offset=5) + n = len(payload) + out = np.empty(n, dtype=np.uint8) + src_off = 0 + for pos in range(stride): + chunk_len = n // stride + (1 if pos < n % stride else 0) + out[pos::stride][:chunk_len] = payload[src_off : src_off + chunk_len] + src_off += chunk_len + return out.tobytes() + + +def _compress(data, compressor): + data = _byte_shuffle(data) + if compressor == "lzma": + return lzma.compress(data, preset=6) + elif compressor == "brotli": + import brotli + + return brotli.compress(data, quality=11) + raise ValueError(f"Unknown compressor: {compressor!r}") + + +def _decompress(data, compressor): + if compressor == "lzma": + raw = lzma.decompress(data) + elif compressor == "brotli": + import brotli + + raw = brotli.decompress(data) + else: + raise ValueError(f"Unknown compressor: {compressor!r}") + raw = _byte_unshuffle(raw) + return raw + + +def _unbank_state_dict(state_dict, num_layers): + sd = {} + n = num_layers + for k, v in state_dict.items(): + t = v.detach().cpu() if v is not None else None + if k == "qo_bank": + for i in range(n): + sd[f"blocks.{i}.attn.c_q.weight"] = t[i] + sd[f"blocks.{i}.attn.proj.weight"] = t[n + i] + elif k == "kv_bank": + for i in range(n): + sd[f"blocks.{i}.attn.c_k.weight"] = t[i] + sd[f"blocks.{i}.attn.c_v.weight"] = t[n + i] + elif k == "mlp_up_bank": + for i in range(n): + sd[f"blocks.{i}.mlp.fc.weight"] = t[i] + elif k == "mlp_down_bank": + for i in range(n): + sd[f"blocks.{i}.mlp.proj.weight"] = t[i] + else: + if t is not None: + sd[k] = t + return sd + + +def _rebank_state_dict(flat_sd, num_layers, model_dim, kv_dim, hidden_dim): + sd = {} + n = num_layers + sd["qo_bank"] = torch.zeros(2 * n, model_dim, model_dim) + sd["kv_bank"] = torch.zeros(2 * n, kv_dim, model_dim) + for i in range(n): + sd["qo_bank"][i] = flat_sd[f"blocks.{i}.attn.c_q.weight"] + sd["qo_bank"][n + i] = flat_sd[f"blocks.{i}.attn.proj.weight"] + sd["kv_bank"][i] = flat_sd[f"blocks.{i}.attn.c_k.weight"] + sd["kv_bank"][n + i] = flat_sd[f"blocks.{i}.attn.c_v.weight"] + sd["mlp_up_bank"] = torch.zeros(n, hidden_dim, model_dim) + sd["mlp_down_bank"] = torch.zeros(n, model_dim, hidden_dim) + for i in range(n): + sd["mlp_up_bank"][i] = flat_sd[f"blocks.{i}.mlp.fc.weight"] + sd["mlp_down_bank"][i] = flat_sd[f"blocks.{i}.mlp.proj.weight"] + for k, v in flat_sd.items(): + if not ( + k.startswith("blocks.") + and any( + p in k + for p in [ + ".attn.c_q.", ".attn.c_k.", ".attn.c_v.", + ".attn.proj.", ".mlp.fc.", ".mlp.proj.", + ] + ) + ): + sd[k] = v + return sd + + + +def _compressed_code_size(code): + code_raw = code.encode("utf-8") + try: + result = subprocess.run( + ["pyminify", "--no-rename-locals", "--no-hoist-literals", "--remove-literal-statements", "-"], + input=code_raw, capture_output=True, check=True, + ) + minified = result.stdout + except (FileNotFoundError, subprocess.CalledProcessError): + minified = code_raw + compressed = lzma.compress(minified) + encoded = base64.b85encode(compressed) + wrapper = b'import lzma as L,base64 as B\nexec(L.decompress(B.b85decode("' + encoded + b'")))\n' + return len(code_raw), len(wrapper) + + +def serialize(h, base_model, code): + code_bytes_uncompressed, code_bytes = _compressed_code_size(code) + if h.is_main_process: + torch.save(base_model.state_dict(), h.model_path) + model_bytes = os.path.getsize(h.model_path) + log(f"Serialized model: {model_bytes} bytes") + log(f"Code size (uncompressed): {code_bytes_uncompressed} bytes") + log(f"Code size (compressed): {code_bytes} bytes") + sd_cpu = _unbank_state_dict(base_model.state_dict(), h.num_layers) + device = torch.device("cuda", h.local_rank) + log("GPTQ:collecting Hessians from calibration data...") + t0 = time.perf_counter() + calib_loader = ShuffledSequenceLoader(h, device) + hessians = collect_hessians( + base_model, + calib_loader, + h, + device, + n_calibration_batches=h.gptq_calibration_batches, + ) + log(f"GPTQ:collected {len(hessians)} Hessians in {time.perf_counter()-t0:.1f}s") + quant_result, quant_meta = gptq_mixed_quantize(sd_cpu, hessians, h) + quant_buf = io.BytesIO() + torch.save({"w": quant_result, "m": quant_meta}, quant_buf) + quant_raw = quant_buf.getvalue() + quant_blob = _compress(quant_raw, h.compressor) + quant_file_bytes = len(quant_blob) + bytes_total = quant_file_bytes + code_bytes + if h.is_main_process: + with open(h.quantized_model_path, "wb") as f: + f.write(quant_blob) + log(f"Serialized model quantized+{h.compressor}: {quant_file_bytes} bytes") + log(f"Total submission size quantized+{h.compressor}: {bytes_total} bytes") + return bytes_total, quant_file_bytes + + +def deserialize(h, device): + eval_model = GPT(h).to(device).bfloat16() + restore_fp32_params(eval_model) + flat_template = _unbank_state_dict(eval_model.state_dict(), h.num_layers) + with open(h.quantized_model_path, "rb") as f: + quant_blob_disk = f.read() + quant_state = torch.load( + io.BytesIO(_decompress(quant_blob_disk, h.compressor)), map_location="cpu" + ) + deq_flat = dequantize_mixed(quant_state["w"], quant_state["m"], flat_template) + head_dim = h.model_dim // h.num_heads + kv_dim = h.num_kv_heads * head_dim + hidden_dim = int(h.mlp_mult * h.model_dim) + deq_state = _rebank_state_dict(deq_flat, h.num_layers, h.model_dim, kv_dim, hidden_dim) + eval_model.load_state_dict(deq_state, strict=True) + return eval_model + + +def _loss_bpb(loss_sum, token_count, byte_count): + val_loss = (loss_sum / token_count).item() + val_bpb = val_loss / math.log(2.0) * (token_count.item() / byte_count.item()) + return val_loss, val_bpb + + +def eval_val(h, device, val_data, model, forward_logits_fn=None): + seq_len = h.eval_seq_len + local_batch_tokens = h.val_batch_tokens // (h.world_size * h.grad_accum_steps) + if local_batch_tokens < seq_len: + raise ValueError( + f"VAL_BATCH_SIZE must provide at least one sequence per rank; got VAL_BATCH_SIZE={h.val_batch_tokens}, WORLD_SIZE={h.world_size}, GRAD_ACCUM_STEPS={h.grad_accum_steps}, seq_len={seq_len}" + ) + local_batch_seqs = local_batch_tokens // seq_len + total_seqs = (val_data.val_tokens.numel() - 1) // seq_len + seq_start = total_seqs * h.rank // h.world_size + seq_end = total_seqs * (h.rank + 1) // h.world_size + + # TODO: Don't truncate this. + seq_end = seq_start + ((seq_end - seq_start) // local_batch_seqs) * local_batch_seqs + + val_loss_sum = torch.zeros((), device=device, dtype=torch.float64) + val_token_count = torch.zeros((), device=device, dtype=torch.float64) + val_byte_count = torch.zeros((), device=device, dtype=torch.float64) + run_forward_logits = ( + (model.module.forward_logits if hasattr(model, "module") else model.forward_logits) + if forward_logits_fn is None + else forward_logits_fn + ) + model.eval() + global BOS_ID + if BOS_ID is None: + BOS_ID = 1 + with torch.no_grad(): + for batch_seq_start in range(seq_start, seq_end, local_batch_seqs): + batch_seq_end = min(batch_seq_start + local_batch_seqs, seq_end) + raw_start = batch_seq_start * seq_len + raw_end = batch_seq_end * seq_len + 1 + local = val_data.val_tokens[raw_start:raw_end].to( + device=device, dtype=torch.int64, non_blocking=True + ) + x = local[:-1] + y = local[1:] + bos_pos = (x == BOS_ID).nonzero(as_tuple=True)[0].tolist() + cu_seqlens, max_seqlen = _build_cu_seqlens( + bos_pos, x.numel(), x.device, h.eval_seq_len, 64 + ) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + logits = run_forward_logits( + x[None], cu_seqlens=cu_seqlens, max_seqlen=max_seqlen + ).detach() + per_token_loss = F.cross_entropy( + logits.reshape(-1, logits.size(-1)).float(), + y.reshape(-1), + reduction="none", + ) + val_loss_sum += per_token_loss.to(torch.float64).sum() + val_token_count += float(y.numel()) + prev_ids = x + tgt_ids = y + token_bytes = val_data.base_bytes_lut[tgt_ids].to(dtype=torch.int16) + token_bytes += ( + val_data.has_leading_space_lut[tgt_ids] + & ~val_data.is_boundary_token_lut[prev_ids] + ).to(dtype=torch.int16) + val_byte_count += token_bytes.to(torch.float64).sum() + if dist.is_available() and dist.is_initialized(): + dist.all_reduce(val_loss_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(val_token_count, op=dist.ReduceOp.SUM) + dist.all_reduce(val_byte_count, op=dist.ReduceOp.SUM) + model.train() + return _loss_bpb(val_loss_sum, val_token_count, val_byte_count) + + +def eval_val_sliding(h, device, val_data, base_model, forward_logits_fn=None, batch_seqs=32): + global BOS_ID + if BOS_ID is None: + BOS_ID = 1 + base_model.eval() + run_forward_logits = base_model.forward_logits if forward_logits_fn is None else forward_logits_fn + seq_len = h.eval_seq_len + stride = h.eval_stride + total_tokens = val_data.val_tokens.numel() - 1 + context_size = seq_len - stride + window_starts = [ws for ws in range(0, total_tokens, stride) + if ws + context_size < total_tokens] + total_windows = len(window_starts) + my_s = (total_windows * h.rank) // h.world_size + my_e = (total_windows * (h.rank + 1)) // h.world_size + my_windows = window_starts[my_s:my_e] + loss_sum = torch.zeros((), device=device, dtype=torch.float64) + token_count = torch.zeros((), device=device, dtype=torch.float64) + byte_count = torch.zeros((), device=device, dtype=torch.float64) + total_batches = (len(my_windows) + batch_seqs - 1) // batch_seqs + is_master = h.rank == 0 + cu_bucket = 64 + t_sw_start = time.perf_counter() + with torch.no_grad(): + for bi in range(0, len(my_windows), batch_seqs): + batch_idx = bi // batch_seqs + if is_master and (batch_idx % 50 == 0 or batch_idx == total_batches - 1): + elapsed = time.perf_counter() - t_sw_start + rl = float(loss_sum.item() / token_count.item()) if token_count.item() > 0 else 0.0 + rb = float((rl / math.log(2.0)) * token_count.item() / byte_count.item()) if byte_count.item() > 0 else 0.0 + log(f"sliding_progress: batch {batch_idx+1}/{total_batches} " + f"tokens:{int(token_count.item())} running_loss:{rl:.4f} running_bpb:{rb:.4f} " + f"elapsed:{elapsed:.1f}s") + batch_ws = my_windows[bi:bi + batch_seqs] + x_parts = [] + y_parts = [] + cu_starts = [] + score_ranges = [] + offset = 0 + for ws in batch_ws: + end = min(ws + seq_len, total_tokens) + wlen = end - ws + chunk_cpu = val_data.val_tokens[ws:end + 1] + bos_pos = (chunk_cpu[:-1] == BOS_ID).nonzero(as_tuple=True)[0].tolist() + if not bos_pos or bos_pos[0] != 0: + bos_pos = [0] + bos_pos + cu_starts.extend(offset + pos for pos in bos_pos) + chunk = chunk_cpu.to(dtype=torch.int64, device=device) + x_parts.append(chunk[:-1]) + y_parts.append(chunk[1:]) + score_ranges.append((offset, wlen, ws)) + offset += wlen + x_cat = torch.cat(x_parts, dim=0)[None] + y_cat = torch.cat(y_parts, dim=0) + boundaries = cu_starts + [offset] + padded_len = get_next_multiple_of_n(len(boundaries), cu_bucket) + cu_seqlens = torch.full((padded_len,), offset, dtype=torch.int32, device=device) + cu_seqlens[:len(boundaries)] = torch.tensor(boundaries, dtype=torch.int32, device=device) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16): + logits = run_forward_logits(x_cat, cu_seqlens=cu_seqlens, max_seqlen=seq_len) + flat_nll = F.cross_entropy( + logits.reshape(-1, logits.size(-1)).float(), + y_cat, + reduction="none", + ) + flat_x = x_cat.reshape(-1) + for off, wlen, ws in score_ranges: + s = 0 if ws == 0 else context_size + lo = off + s + hi = off + wlen + scored_nll = flat_nll[lo:hi].to(torch.float64) + loss_sum += scored_nll.sum() + token_count += float(hi - lo) + tgt = y_cat[lo:hi] + prev = flat_x[lo:hi] + tb = val_data.base_bytes_lut[tgt].to(torch.float64) + tb += (val_data.has_leading_space_lut[tgt] & ~val_data.is_boundary_token_lut[prev]).to(torch.float64) + byte_count += tb.sum() + if dist.is_available() and dist.is_initialized(): + dist.all_reduce(loss_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(token_count, op=dist.ReduceOp.SUM) + dist.all_reduce(byte_count, op=dist.ReduceOp.SUM) + base_model.train() + return _loss_bpb(loss_sum, token_count, byte_count) + + +def _find_docs(all_tokens): + bos_positions = (all_tokens == BOS_ID).nonzero(as_tuple=True)[0].numpy() + docs = [] + for i in range(len(bos_positions)): + start = int(bos_positions[i]) + end = ( + int(bos_positions[i + 1]) + if i + 1 < len(bos_positions) + else all_tokens.numel() + ) + if i + 1 < len(bos_positions): + end += 1 + assert end - start >= 2 + docs.append((start, end - start)) + return docs + + +def _build_ttt_global_batches(doc_entries, h, ascending=False): + batch_size = h.ttt_batch_size + global_doc_entries = sorted(doc_entries, key=lambda x: x[1][1]) + global_batches = [ + global_doc_entries[i : i + batch_size] + for i in range(0, len(global_doc_entries), batch_size) + ] + indexed = list(enumerate(global_batches)) + if not ascending: + indexed.sort(key=lambda ib: -max(dl for _, (_, dl) in ib[1])) + return indexed + + +def _init_batch_counter(path): + with open(path, "wb") as f: + f.write((0).to_bytes(4, "little")) + + +def _claim_next_batch(counter_path, queue_len): + try: + with open(counter_path, "r+b") as f: + fcntl.flock(f, fcntl.LOCK_EX) + idx = int.from_bytes(f.read(4), "little") + f.seek(0) + f.write((idx + 1).to_bytes(4, "little")) + f.flush() + except FileNotFoundError: + return queue_len + return idx + + +def _compute_chunk_window(ci, pred_len, num_chunks, chunk_size, eval_seq_len): + chunk_end = pred_len if ci == num_chunks - 1 else (ci + 1) * chunk_size + win_start = max(0, chunk_end - eval_seq_len) + win_len = chunk_end - win_start + chunk_start = ci * chunk_size + chunk_offset = chunk_start - win_start + chunk_len = chunk_end - chunk_start + return win_start, win_len, chunk_offset, chunk_len + + +def _accumulate_bpb( + ptl, + x, + y, + chunk_offsets, + chunk_lens, + pos_idx, + base_bytes_lut, + has_leading_space_lut, + is_boundary_token_lut, + loss_sum, + byte_sum, + token_count, +): + pos = pos_idx[: x.size(1)].unsqueeze(0) + mask = ( + (chunk_lens.unsqueeze(1) > 0) + & (pos >= chunk_offsets.unsqueeze(1)) + & (pos < (chunk_offsets + chunk_lens).unsqueeze(1)) + ) + mask_f64 = mask.to(torch.float64) + tok_bytes = base_bytes_lut[y].to(torch.float64) + tok_bytes += (has_leading_space_lut[y] & ~is_boundary_token_lut[x]).to( + torch.float64 + ) + loss_sum += (ptl.to(torch.float64) * mask_f64).sum() + byte_sum += (tok_bytes * mask_f64).sum() + token_count += chunk_lens.to(torch.float64).sum() + + +def _loss_bpb_from_sums(loss_sum, token_count, byte_sum): + val_loss = (loss_sum / token_count).item() + val_bpb = val_loss / math.log(2.0) * (token_count.item() / byte_sum.item()) + return val_loss, val_bpb + + +def _split_doc_entries_for_phased(doc_entries, prefix_docs): + prefix_docs = max(0, min(len(doc_entries), int(prefix_docs))) + return doc_entries[:prefix_docs], doc_entries[prefix_docs:] + + +def _add_to_counter(path, delta): + try: + with open(path, "r+b") as f: + fcntl.flock(f, fcntl.LOCK_EX) + cur = int.from_bytes(f.read(8), "little", signed=True) + cur += int(delta) + f.seek(0) + f.write(int(cur).to_bytes(8, "little", signed=True)) + f.flush() + return cur + except FileNotFoundError: + return int(delta) + + +def _init_int64_counter(path): + with open(path, "wb") as f: + f.write((0).to_bytes(8, "little", signed=True)) + + +def _select_ttt_doc_entries(docs, h): + doc_entries = list(enumerate(docs)) + if h.val_doc_fraction < 1.0: + sample_n = max(1, int(round(len(docs) * h.val_doc_fraction))) + sampled_indices = sorted( + random.Random(h.seed).sample(range(len(docs)), sample_n) + ) + return [(i, docs[i]) for i in sampled_indices] + return doc_entries + + +def train_val_ttt_global_sgd_distributed(h, device, val_data, base_model, val_tokens, batch_seqs=None): + global BOS_ID + if BOS_ID is None: + BOS_ID = 1 + base_model.eval() + seq_len = h.eval_seq_len + total_tokens = val_tokens.numel() - 1 + ttt_chunk = h.global_ttt_chunk_tokens + batch_seqs = h.global_ttt_batch_seqs if batch_seqs is None else batch_seqs + num_chunks = (total_tokens + ttt_chunk - 1) // ttt_chunk + ttt_params = [p for p in base_model.parameters()] + for p in ttt_params: + p.requires_grad_(True) + optimizer = torch.optim.SGD( + ttt_params, lr=h.global_ttt_lr, momentum=h.global_ttt_momentum + ) + t_start = time.perf_counter() + for ci in range(num_chunks): + chunk_start = ci * ttt_chunk + chunk_end = min((ci + 1) * ttt_chunk, total_tokens) + is_last_chunk = ci == num_chunks - 1 + if is_last_chunk or h.global_ttt_epochs <= 0: + continue + base_model.train() + chunk_seqs = (chunk_end - chunk_start) // seq_len + if chunk_seqs <= 0: + continue + warmup_chunks = max(0, min(h.global_ttt_warmup_chunks, num_chunks - 1)) + if warmup_chunks > 0 and ci < warmup_chunks: + warmup_denom = max(warmup_chunks - 1, 1) + warmup_t = ci / warmup_denom + lr_now = ( + h.global_ttt_warmup_start_lr + + (h.global_ttt_lr - h.global_ttt_warmup_start_lr) * warmup_t + ) + else: + decay_steps = max(num_chunks - 1 - warmup_chunks, 1) + decay_ci = max(ci - warmup_chunks, 0) + lr_now = h.global_ttt_lr * 0.5 * ( + 1.0 + math.cos(math.pi * decay_ci / decay_steps) + ) + for pg in optimizer.param_groups: + pg["lr"] = lr_now + my_seq_s = chunk_seqs * h.rank // h.world_size + my_seq_e = chunk_seqs * (h.rank + 1) // h.world_size + my_chunk_seqs = my_seq_e - my_seq_s + for _ in range(h.global_ttt_epochs): + for bs in range(0, my_chunk_seqs, batch_seqs): + be = min(bs + batch_seqs, my_chunk_seqs) + actual_bs = my_seq_s + bs + start_tok = chunk_start + actual_bs * seq_len + end_tok = chunk_start + (my_seq_s + be) * seq_len + 1 + if end_tok > val_tokens.numel(): + continue + local = val_tokens[start_tok:end_tok].to(device=device, dtype=torch.int64) + x_flat = local[:-1] + y_flat = local[1:] + optimizer.zero_grad(set_to_none=True) + with torch.enable_grad(): + with torch.autocast(device_type="cuda", dtype=torch.bfloat16): + if h.global_ttt_respect_doc_boundaries: + bos_pos = (x_flat == BOS_ID).nonzero(as_tuple=True)[0].tolist() + cu_seqlens, max_seqlen = _build_cu_seqlens( + bos_pos, x_flat.numel(), x_flat.device, h.eval_seq_len, 64 + ) + loss = base_model( + x_flat[None], + y_flat[None], + cu_seqlens=cu_seqlens, + max_seqlen=max_seqlen, + ) + else: + x = x_flat.reshape(-1, seq_len) + y = y_flat.reshape(-1, seq_len) + loss = base_model(x, y) + loss.backward() + if dist.is_available() and dist.is_initialized(): + for p in ttt_params: + if p.grad is not None: + dist.all_reduce(p.grad, op=dist.ReduceOp.SUM) + p.grad.mul_(1.0 / h.world_size) + if h.global_ttt_grad_clip > 0: + torch.nn.utils.clip_grad_norm_(ttt_params, h.global_ttt_grad_clip) + optimizer.step() + base_model.eval() + if h.rank == 0: + elapsed = time.perf_counter() - t_start + log( + f"tttg: c{ci+1}/{num_chunks} lr:{lr_now:.6f} t:{elapsed:.1f}s" + ) + for p in base_model.parameters(): + p.requires_grad_(True) + base_model.eval() + + +def eval_val_ttt_phased(h, base_model, device, val_data, forward_ttt_train): + global BOS_ID + if BOS_ID is None: + BOS_ID = 1 + base_model.eval() + for p in base_model.parameters(): + p.requires_grad_(False) + all_tokens = val_data.val_tokens + all_tokens_idx = all_tokens.to(torch.int32) + docs = _find_docs(all_tokens) + doc_entries = _select_ttt_doc_entries(docs, h) + prefix_doc_limit = max(0, min(len(doc_entries), int(h.phased_ttt_prefix_docs))) + num_phases = max(1, int(h.phased_ttt_num_phases)) + phase_boundaries = [] + for pi in range(num_phases): + boundary = prefix_doc_limit * (pi + 1) // num_phases + phase_boundaries.append(boundary) + current_phase = 0 + current_phase_boundary = phase_boundaries[0] + log( + "ttt_phased:" + f" total_docs:{len(doc_entries)} prefix_docs:{prefix_doc_limit} " + f"suffix_docs:{len(doc_entries) - prefix_doc_limit}" + f" num_phases:{num_phases} boundaries:{phase_boundaries}" + ) + chunk_size, eval_seq_len = h.ttt_chunk_size, h.ttt_eval_seq_len + eval_batch_set = None + if h.ttt_eval_batches: + eval_batch_set = set(int(x) for x in h.ttt_eval_batches.split(",") if x.strip()) + use_ascending = eval_batch_set is not None + global_batches_sorted = _build_ttt_global_batches( + doc_entries, h, ascending=use_ascending + ) + queue_len = len(global_batches_sorted) + counter_path = f"/tmp/ttt_counter_{h.run_id}" + prefix_counter_path = f"/tmp/ttt_prefix_counter_{h.run_id}" + pause_flag_path = f"/tmp/ttt_pause_flag_{h.run_id}" + if h.rank == 0: + _init_batch_counter(counter_path) + _init_int64_counter(prefix_counter_path) + try: + os.remove(pause_flag_path) + except FileNotFoundError: + pass + if dist.is_available() and dist.is_initialized(): + path_list = [counter_path, prefix_counter_path, pause_flag_path] + dist.broadcast_object_list(path_list, src=0) + counter_path, prefix_counter_path, pause_flag_path = path_list + dist.barrier() + loss_sum = torch.zeros((), device=device, dtype=torch.float64) + byte_sum = torch.zeros((), device=device, dtype=torch.float64) + token_count = torch.zeros((), device=device, dtype=torch.float64) + t_start = time.perf_counter() + reusable_lora = BatchedTTTLoRA( + h.ttt_batch_size, base_model, h.ttt_lora_rank, + k_lora=h.ttt_k_lora, mlp_lora=h.ttt_mlp_lora, o_lora=h.ttt_o_lora, + ).to(device) + + def _build_opt(lora): + if h.ttt_optimizer == "sgd": + return torch.optim.SGD( + lora.parameters(), lr=h.ttt_lora_lr, + momentum=h.ttt_beta1, weight_decay=h.ttt_weight_decay, + ) + return torch.optim.AdamW( + lora.parameters(), lr=h.ttt_lora_lr, + betas=(h.ttt_beta1, h.ttt_beta2), + eps=1e-10, weight_decay=h.ttt_weight_decay, fused=True, + ) + + reusable_opt = _build_opt(reusable_lora) + local_scored_docs = [] + global_ttt_done = prefix_doc_limit == 0 + try: + while True: + queue_idx = _claim_next_batch(counter_path, queue_len) + if queue_idx >= queue_len: + break + orig_batch_idx, batch_entries = global_batches_sorted[queue_idx] + batch = [doc for _, doc in batch_entries] + bsz = len(batch) + prev_loss = loss_sum.item() + prev_bytes = byte_sum.item() + prev_tokens = token_count.item() + if bsz == reusable_lora.bsz: + reusable_lora.reset() + for s in reusable_opt.state.values(): + for k, v in s.items(): + if isinstance(v, torch.Tensor): + v.zero_() + elif k == "step": + s[k] = 0 + cur_lora = reusable_lora + cur_opt = reusable_opt + else: + cur_lora = BatchedTTTLoRA( + bsz, base_model, h.ttt_lora_rank, + k_lora=h.ttt_k_lora, mlp_lora=h.ttt_mlp_lora, o_lora=h.ttt_o_lora, + ).to(device) + cur_opt = _build_opt(cur_lora) + pred_lens = [doc_len - 1 for _, doc_len in batch] + num_chunks = [(pl + chunk_size - 1) // chunk_size for pl in pred_lens] + max_nc = max(num_chunks) + num_chunks_t = torch.tensor(num_chunks, dtype=torch.int64, device=device) + for ci in range(max_nc): + active = [ci < nc for nc in num_chunks] + needs_train = any(ci < nc - 1 for nc in num_chunks) + tok_starts = torch.zeros(bsz, dtype=torch.int64) + tok_wls = torch.zeros(bsz, dtype=torch.int64) + chunk_offsets_cpu = torch.zeros(bsz, dtype=torch.int64) + chunk_lens_cpu = torch.zeros(bsz, dtype=torch.int64) + for b in range(bsz): + if not active[b]: + continue + doc_start, doc_len = batch[b] + win_start, win_len, chunk_offset, chunk_len = _compute_chunk_window( + ci, pred_lens[b], num_chunks[b], chunk_size, eval_seq_len + ) + tok_starts[b] = doc_start + win_start + tok_wls[b] = win_len + chunk_offsets_cpu[b] = chunk_offset + chunk_lens_cpu[b] = chunk_len + _, context_size, chunk_offset, _ = _compute_chunk_window( + ci, (ci + 1) * chunk_size, ci + 1, chunk_size, eval_seq_len + ) + col_idx = torch.arange(context_size + 1) + idx = tok_starts.unsqueeze(1) + col_idx.unsqueeze(0) + idx.clamp_(max=all_tokens.numel() - 1) + gathered_gpu = all_tokens_idx[idx].to( + device=device, dtype=torch.int64, non_blocking=True + ) + valid = (col_idx[:context_size].unsqueeze(0) < tok_wls.unsqueeze(1)).to( + device, non_blocking=True + ) + chunk_offsets = chunk_offsets_cpu.to(device, non_blocking=True) + chunk_lens = chunk_lens_cpu.to(device, non_blocking=True) + x = torch.where(valid, gathered_gpu[:, :context_size], 0) + y = torch.where(valid, gathered_gpu[:, 1 : context_size + 1], 0) + ctx_pos = torch.arange(context_size, device=device, dtype=torch.int64) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16): + per_tok_loss = forward_ttt_train(x, y, lora=cur_lora) + with torch.no_grad(): + _accumulate_bpb( + per_tok_loss, + x, + y, + chunk_offsets, + chunk_lens, + ctx_pos, + val_data.base_bytes_lut, + val_data.has_leading_space_lut, + val_data.is_boundary_token_lut, + loss_sum, + byte_sum, + token_count, + ) + if needs_train: + activate_chunk_mask = (num_chunks_t - 1 > ci).float() + for gi in range(h.ttt_grad_steps): + if gi > 0: + with torch.autocast(device_type="cuda", dtype=torch.bfloat16): + per_tok_loss = forward_ttt_train(x, y, lora=cur_lora) + per_doc = per_tok_loss[ + :, chunk_offset : chunk_offset + chunk_size + ].mean(dim=-1) + cur_opt.zero_grad(set_to_none=True) + (per_doc * activate_chunk_mask).sum().backward() + cur_opt.step() + else: + del per_tok_loss + batch_num = orig_batch_idx + 1 + doc_lens = [dl for _, dl in batch] + should_report = batch_num in eval_batch_set if eval_batch_set is not None else True + if should_report: + cur_tokens = token_count.item() + cur_loss_val = loss_sum.item() + cur_bytes_val = byte_sum.item() + dt = cur_tokens - prev_tokens + db = cur_bytes_val - prev_bytes + if dt > 0 and db > 0: + b_loss = (cur_loss_val - prev_loss) / dt + b_bpb = b_loss / math.log(2.0) * (dt / db) + else: + b_loss = b_bpb = 0.0 + r_loss = cur_loss_val / max(cur_tokens, 1) + r_bpb = r_loss / math.log(2.0) * (cur_tokens / max(cur_bytes_val, 1)) + elapsed = time.perf_counter() - t_start + log( + f"ttp: b{batch_num}/{queue_len} bl:{b_loss:.4f} bb:{b_bpb:.4f} " + f"rl:{r_loss:.4f} rb:{r_bpb:.4f} dl:{min(doc_lens)}-{max(doc_lens)} " + f"gd:{int(global_ttt_done)}" + ) + if not global_ttt_done: + local_scored_docs.extend( + (orig_batch_idx, pos, doc_start, doc_len) + for pos, (doc_start, doc_len) in enumerate(batch) + ) + prefix_done = _add_to_counter(prefix_counter_path, len(batch_entries)) + if prefix_done >= current_phase_boundary: + try: + with open(pause_flag_path, "x"): + pass + except FileExistsError: + pass + should_pause = os.path.exists(pause_flag_path) + if should_pause: + if dist.is_available() and dist.is_initialized(): + dist.barrier() + gathered_scored_docs = [None] * h.world_size + if dist.is_available() and dist.is_initialized(): + dist.all_gather_object(gathered_scored_docs, local_scored_docs) + else: + gathered_scored_docs = [local_scored_docs] + scored_docs_for_global = [] + for rank_docs in gathered_scored_docs: + if rank_docs: + scored_docs_for_global.extend(rank_docs) + scored_docs_for_global.sort(key=lambda x: (x[0], x[1])) + scored_docs_for_global = scored_docs_for_global[:current_phase_boundary] + scored_token_chunks = [ + val_data.val_tokens[doc_start : doc_start + doc_len] + for _, _, doc_start, doc_len in scored_docs_for_global + ] + if scored_token_chunks: + global_ttt_tokens = torch.cat(scored_token_chunks) + else: + global_ttt_tokens = val_data.val_tokens[:0] + if h.rank == 0: + prefix_done = 0 + try: + with open(prefix_counter_path, "rb") as f: + prefix_done = int.from_bytes( + f.read(8), "little", signed=True + ) + except FileNotFoundError: + pass + log( + f"ttpp: phase:{current_phase + 1}/{num_phases} pd:{prefix_done} " + f"gd:{len(scored_docs_for_global)} " + f"t:{time.perf_counter() - t_start:.1f}s" + ) + train_val_ttt_global_sgd_distributed( + h, device, val_data, base_model, global_ttt_tokens + ) + for p in base_model.parameters(): + p.requires_grad_(False) + reusable_lora = BatchedTTTLoRA( + h.ttt_batch_size, base_model, h.ttt_lora_rank, + k_lora=h.ttt_k_lora, mlp_lora=h.ttt_mlp_lora, o_lora=h.ttt_o_lora, + ).to(device) + reusable_opt = _build_opt(reusable_lora) + current_phase += 1 + if current_phase >= num_phases: + global_ttt_done = True + else: + current_phase_boundary = phase_boundaries[current_phase] + if h.rank == 0: + try: + os.remove(pause_flag_path) + except FileNotFoundError: + pass + if dist.is_available() and dist.is_initialized(): + dist.barrier() + if h.rank == 0: + log(f"ttpr: phase:{current_phase}/{num_phases} t:{time.perf_counter() - t_start:.1f}s") + del cur_lora, cur_opt + finally: + pass + if dist.is_available() and dist.is_initialized(): + dist.all_reduce(loss_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(byte_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(token_count, op=dist.ReduceOp.SUM) + for p in base_model.parameters(): + p.requires_grad_(True) + base_model.train() + return _loss_bpb_from_sums(loss_sum, token_count, byte_sum) + + +def timed_eval(label, fn, *args, **kwargs): + torch.cuda.synchronize() + t0 = time.perf_counter() + val_loss, val_bpb = fn(*args, **kwargs) + torch.cuda.synchronize() + elapsed_ms = 1e3 * (time.perf_counter() - t0) + log( + f"{label} val_loss:{val_loss:.8f} val_bpb:{val_bpb:.8f} eval_time:{elapsed_ms:.0f}ms" + ) + return val_loss, val_bpb + + +def train_model(h, device, val_data): + base_model = GPT(h).to(device).bfloat16() + restore_fp32_params(base_model) + if h.newton_muon_enabled: + log(f"newton_muon: enabled (compile fullgraph=False, capture_every={h.newton_muon_capture_every})") + compiled_model = torch.compile(base_model, dynamic=False, fullgraph=False) + compiled_forward_logits = torch.compile( + base_model.forward_logits, dynamic=False, fullgraph=False + ) + _nm_install_hooks(base_model, capture_every=h.newton_muon_capture_every) + else: + compiled_model = torch.compile(base_model, dynamic=False, fullgraph=True) + compiled_forward_logits = torch.compile( + base_model.forward_logits, dynamic=False, fullgraph=True + ) + model = compiled_model + log(f"model_params:{sum(p.numel()for p in base_model.parameters())}") + optimizers = Optimizers(h, base_model) + train_loader = DocumentPackingLoader(h, device) + max_wallclock_ms = ( + 1e3 * h.max_wallclock_seconds if h.max_wallclock_seconds > 0 else None + ) + if max_wallclock_ms is not None: + max_wallclock_ms -= h.gptq_reserve_seconds * 1e3 + log( + f"gptq:reserving {h.gptq_reserve_seconds:.0f}s, effective={max_wallclock_ms:.0f}ms" + ) + + def training_frac(step, elapsed_ms): + if max_wallclock_ms is None: + return step / max(h.iterations, 1) + return elapsed_ms / max(max_wallclock_ms, 1e-09) + + def lr_mul(frac): + if h.warmdown_frac <= 0: + return 1.0 + if frac >= 1.0 - h.warmdown_frac: + return max((1.0 - frac) / h.warmdown_frac, h.min_lr) + return 1.0 + + def step_fn(step, lr_scale): + optimizers.zero_grad_all() + train_loss = torch.zeros((), device=device) + for micro_step in range(h.grad_accum_steps): + x, y, cu_seqlens, _max_seqlen = train_loader.next_batch( + h.train_batch_tokens, h.grad_accum_steps + ) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + loss = model(x, y, cu_seqlens=cu_seqlens, max_seqlen=h.train_seq_len) + train_loss += loss.detach() + (loss / h.grad_accum_steps).backward() + train_loss /= h.grad_accum_steps + frac = ( + min(step / h.muon_momentum_warmup_steps, 1.0) + if h.muon_momentum_warmup_steps > 0 + else 1.0 + ) + muon_momentum = ( + 1 - frac + ) * h.muon_momentum_warmup_start + frac * h.muon_momentum + for group in optimizers.optimizer_muon.param_groups: + group["momentum"] = muon_momentum + for opt in optimizers: + for group in opt.param_groups: + group["lr"] = group["base_lr"] * lr_scale + if h.grad_clip_norm > 0: + torch.nn.utils.clip_grad_norm_(base_model.parameters(), h.grad_clip_norm) + optimizers.step(distributed=h.distributed) + if h.newton_muon_enabled: + _nm_advance_step() + return train_loss + + if h.warmup_steps > 0: + initial_model_state = { + name: tensor.detach().cpu().clone() + for (name, tensor) in base_model.state_dict().items() + } + initial_optimizer_states = [ + copy.deepcopy(opt.state_dict()) for opt in optimizers + ] + model.train() + num_tokens_local = h.train_batch_tokens // h.world_size + for blk in base_model.blocks: + blk.attn.rotary(num_tokens_local, device, torch.bfloat16) + cu_bucket_size = train_loader.cu_bucket_size + warmup_cu_buckets = tuple(cu_bucket_size * i for i in range(1, 5)) + warmup_cu_iters = 3 + x, y, cu_seqlens, _ = train_loader.next_batch( + h.train_batch_tokens, h.grad_accum_steps + ) + log(f"warmup_cu_buckets:{','.join(str(b) for b in warmup_cu_buckets)} iters_each:{warmup_cu_iters}") + def _run_cu_bucket_warmup(): + for bucket_len in warmup_cu_buckets: + boundaries = list(range(0, x.size(1), max(h.train_seq_len, 1))) + if boundaries[-1] != x.size(1): + boundaries.append(x.size(1)) + cu = torch.full((bucket_len,), x.size(1), dtype=torch.int32, device=device) + cu[: len(boundaries)] = torch.tensor(boundaries, dtype=torch.int32, device=device) + for _ in range(warmup_cu_iters): + optimizers.zero_grad_all() + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + wloss = model(x, y, cu_seqlens=cu, max_seqlen=h.train_seq_len) + (wloss / h.grad_accum_steps).backward() + optimizers.zero_grad_all() + _run_cu_bucket_warmup() + if h.num_loops > 0: + base_model.looping_active = True + _run_cu_bucket_warmup() + base_model.looping_active = False + for warmup_step in range(h.warmup_steps): + step_fn(warmup_step, 1.0) + if ( + warmup_step <= 5 + or (warmup_step + 1) % 10 == 0 + or warmup_step + 1 == h.warmup_steps + ): + log(f"warmup_step: {warmup_step+1}/{h.warmup_steps}") + if h.num_loops > 0: + base_model.looping_active = True + log( + f"loop_warmup:enabled encoder:{base_model.encoder_indices} decoder:{base_model.decoder_indices}" + ) + for warmup_step in range(h.warmup_steps): + step_fn(warmup_step, 1.0) + if ( + warmup_step <= 5 + or (warmup_step + 1) % 10 == 0 + or warmup_step + 1 == h.warmup_steps + ): + log(f"loop_warmup_step: {warmup_step+1}/{h.warmup_steps}") + base_model.looping_active = False + base_model.load_state_dict(initial_model_state, strict=True) + for (opt, state) in zip(optimizers, initial_optimizer_states, strict=True): + opt.load_state_dict(state) + optimizers.zero_grad_all() + train_loader = DocumentPackingLoader(h, device) + ema_state = { + name: t.detach().float().clone() + for (name, t) in base_model.state_dict().items() + } + ema_decay = h.ema_decay + training_time_ms = 0.0 + stop_after_step = None + torch.cuda.synchronize() + t0 = time.perf_counter() + step = 0 + while True: + last_step = ( + step == h.iterations + or stop_after_step is not None + and step >= stop_after_step + ) + should_validate = ( + last_step or h.val_loss_every > 0 and step % h.val_loss_every == 0 + ) + if should_validate: + torch.cuda.synchronize() + training_time_ms += 1e3 * (time.perf_counter() - t0) + val_loss, val_bpb = eval_val( + h, device, val_data, model, compiled_forward_logits + ) + log( + f"{step}/{h.iterations} val_loss: {val_loss:.4f} val_bpb: {val_bpb:.4f}" + ) + torch.cuda.synchronize() + t0 = time.perf_counter() + if last_step: + if stop_after_step is not None and step < h.iterations: + log( + f"stopping_early: wallclock_cap train_time: {training_time_ms:.0f}ms step: {step}/{h.iterations}" + ) + break + elapsed_ms = training_time_ms + 1e3 * (time.perf_counter() - t0) + frac = training_frac(step, elapsed_ms) + scale = lr_mul(frac) + if ( + h.num_loops > 0 + and not base_model.looping_active + and frac >= h.enable_looping_at + ): + base_model.looping_active = True + log( + f"layer_loop:enabled step:{step} frac:{frac:.3f} encoder:{base_model.encoder_indices} decoder:{base_model.decoder_indices}" + ) + train_loss = step_fn(step, scale) + with torch.no_grad(): + for (name, t) in base_model.state_dict().items(): + ema_state[name].mul_(ema_decay).add_( + t.detach().float(), alpha=1.0 - ema_decay + ) + step += 1 + approx_training_time_ms = training_time_ms + 1e3 * (time.perf_counter() - t0) + should_log_train = h.train_log_every > 0 and ( + step <= 5 or step % h.train_log_every == 0 or stop_after_step is not None + ) + if should_log_train: + tok_per_sec = step * h.train_batch_tokens / (approx_training_time_ms / 1e3) + log( + f"{step}/{h.iterations} train_loss: {train_loss.item():.4f} train_time: {approx_training_time_ms/60000:.1f}m tok/s: {tok_per_sec:.0f}" + ) + reached_cap = ( + max_wallclock_ms is not None and approx_training_time_ms >= max_wallclock_ms + ) + if h.distributed and max_wallclock_ms is not None: + reached_cap_tensor = torch.tensor(int(reached_cap), device=device) + dist.all_reduce(reached_cap_tensor, op=dist.ReduceOp.MAX) + reached_cap = bool(reached_cap_tensor.item()) + if stop_after_step is None and reached_cap: + stop_after_step = step + log( + f"peak memory allocated: {torch.cuda.max_memory_allocated()//1024//1024} MiB reserved: {torch.cuda.max_memory_reserved()//1024//1024} MiB" + ) + log("ema:applying EMA weights") + current_state = base_model.state_dict() + avg_state = { + name: t.to(dtype=current_state[name].dtype) for (name, t) in ema_state.items() + } + base_model.load_state_dict(avg_state, strict=True) + if h.newton_muon_enabled: + _nm_uninstall_hooks() + log("newton_muon: hooks uninstalled before EMA/quant/eval") + return base_model, compiled_model, compiled_forward_logits + + +def train_and_eval(h, device): + random.seed(h.seed) + np.random.seed(h.seed) + torch.manual_seed(h.seed) + torch.cuda.manual_seed_all(h.seed) + if h.artifact_dir and h.is_main_process: + os.makedirs(h.artifact_dir, exist_ok=True) + val_data = ValidationData(h, device) + log( + f"train_shards: {len(list(Path(h.datasets_dir).resolve().glob('fineweb_train_*.bin')))}" + ) + log(f"val_tokens: {val_data.val_tokens.numel()-1}") + base_model, compiled_model, compiled_forward_logits = train_model( + h, device, val_data + ) + torch._dynamo.reset() + timed_eval( + "diagnostic pre-quantization post-ema", + eval_val, + h, + device, + val_data, + compiled_model, + compiled_forward_logits, + ) + serialize(h, base_model, Path(__file__).read_text(encoding="utf-8")) + if h.distributed: + dist.barrier() + eval_model = deserialize(h, device) + if h.num_loops > 0: + eval_model.looping_active = True + compiled_model = torch.compile(eval_model, dynamic=False, fullgraph=True) + compiled_forward_logits = torch.compile( + eval_model.forward_logits, dynamic=False, fullgraph=True + ) + timed_eval( + "diagnostic quantized", + eval_val, + h, + device, + val_data, + compiled_model, + compiled_forward_logits, + ) + if h.sliding_window_enabled: + timed_eval( + "diagnostic quantized_sliding_window", + eval_val_sliding, + h, + device, + val_data, + eval_model, + forward_logits_fn=compiled_forward_logits, + ) + if h.ttt_enabled: + del eval_model, compiled_model + torch._dynamo.reset() + torch.cuda.empty_cache() + ttt_model = deserialize(h, device) + if h.num_loops > 0: + ttt_model.looping_active = True + for p in ttt_model.parameters(): + p.requires_grad_(False) + + if h.rope_yarn: + _yarn_seqlen = h.train_batch_tokens // h.grad_accum_steps + for block in ttt_model.blocks: + block.attn.rotary(_yarn_seqlen, device, torch.bfloat16) + else: + for block in ttt_model.blocks: + block.attn.rotary._cos_cached = None + block.attn.rotary._sin_cached = None + block.attn.rotary._seq_len_cached = 0 + block.attn.rotary(h.ttt_eval_seq_len, device, torch.bfloat16) + + def _fwd_ttt_inner(input_ids, target_ids, lora): + return ttt_model.forward_ttt(input_ids, target_ids, lora=lora) + + _fwd_ttt_compiled_inner = None + + def _fwd_ttt(input_ids, target_ids, lora): + nonlocal _fwd_ttt_compiled_inner + if _fwd_ttt_compiled_inner is None: + _fwd_ttt_compiled_inner = torch.compile(_fwd_ttt_inner, dynamic=True) + return _fwd_ttt_compiled_inner(input_ids, target_ids, lora=lora) + + fwd_ttt_compiled = _fwd_ttt + log(f"ttt_lora:warming up compile (random tokens, no val data)") + global BOS_ID + if BOS_ID is None: + BOS_ID = 1 + t_warmup = time.perf_counter() + warmup_bszes = [h.ttt_batch_size] + for bsz in warmup_bszes: + wl = BatchedTTTLoRA( + bsz, ttt_model, h.ttt_lora_rank, + k_lora=h.ttt_k_lora, mlp_lora=h.ttt_mlp_lora, o_lora=h.ttt_o_lora, + ).to(device) + wo = torch.optim.AdamW( + wl.parameters(), + lr=h.ttt_lora_lr, + betas=(h.ttt_beta1, h.ttt_beta2), + eps=1e-10, + weight_decay=h.ttt_weight_decay, + fused=True, + ) + for ctx_len in (h.ttt_chunk_size, h.ttt_eval_seq_len): + xw = torch.randint(0, h.vocab_size, (bsz, ctx_len), device=device, dtype=torch.int64) + yw = torch.randint(0, h.vocab_size, (bsz, ctx_len), device=device, dtype=torch.int64) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16): + ptl = fwd_ttt_compiled(xw, yw, lora=wl) + ptl[:, : min(h.ttt_chunk_size, ctx_len)].mean(dim=-1).sum().backward() + wo.step() + wo.zero_grad(set_to_none=True) + del wl, wo + torch.cuda.empty_cache() + compile_elapsed = time.perf_counter() - t_warmup + log(f"ttt_lora:compile warmup done ({compile_elapsed:.1f}s)") + log("\nbeginning TTT eval timer") + torch.cuda.synchronize() + t_ttt = time.perf_counter() + ttt_val_loss, ttt_val_bpb = eval_val_ttt_phased( + h, ttt_model, device, val_data, forward_ttt_train=fwd_ttt_compiled + ) + torch.cuda.synchronize() + ttt_eval_elapsed = time.perf_counter() - t_ttt + log( + "quantized_ttt_phased " + f"val_loss:{ttt_val_loss:.8f} val_bpb:{ttt_val_bpb:.8f} " + f"eval_time:{1e3*ttt_eval_elapsed:.0f}ms" + ) + log(f"total_eval_time:{ttt_eval_elapsed:.1f}s") + del ttt_model + + +def main(): + world_size = int(os.environ.get("WORLD_SIZE", "1")) + local_rank = int(os.environ.get("LOCAL_RANK", "0")) + distributed = "RANK" in os.environ and "WORLD_SIZE" in os.environ + if not torch.cuda.is_available(): + raise RuntimeError("CUDA is required") + if world_size <= 0: + raise ValueError(f"WORLD_SIZE must be positive, got {world_size}") + if 8 % world_size != 0: + raise ValueError( + f"WORLD_SIZE={world_size} must divide 8 so grad_accum_steps stays integral" + ) + device = torch.device("cuda", local_rank) + torch.cuda.set_device(device) + if distributed: + dist.init_process_group(backend="nccl", device_id=device) + dist.barrier() + torch.backends.cuda.matmul.allow_tf32 = True + torch.backends.cudnn.allow_tf32 = True + torch.set_float32_matmul_precision("high") + from torch.backends.cuda import ( + enable_cudnn_sdp, + enable_flash_sdp, + enable_math_sdp, + enable_mem_efficient_sdp, + ) + + enable_cudnn_sdp(False) + enable_flash_sdp(True) + enable_mem_efficient_sdp(False) + enable_math_sdp(False) + torch._dynamo.config.optimize_ddp = False + torch._dynamo.config.cache_size_limit = 16 + h = Hyperparameters() + set_logging_hparams(h) + if h.is_main_process: + os.makedirs(h.artifact_dir if h.artifact_dir else "logs", exist_ok=True) + log(100 * "=", console=False) + log("Hyperparameters:", console=True) + for (k, v) in sorted(vars(type(h)).items()): + if not k.startswith("_"): + log(f" {k}: {v}", console=True) + log("=" * 100, console=False) + log("Source code:", console=False) + log("=" * 100, console=False) + with open(__file__, "r", encoding="utf-8") as _src: + log(_src.read(), console=False) + log("=" * 100, console=False) + log(f"Running Python {sys.version}", console=False) + log(f"Running PyTorch {torch.__version__}", console=False) + log("=" * 100, console=False) + train_and_eval(h, device) + if distributed: + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_default.log b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_default.log new file mode 100644 index 0000000000..71108512ba --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_default.log @@ -0,0 +1,841 @@ +W0428 06:37:53.640000 324982 torch/distributed/run.py:803] +W0428 06:37:53.640000 324982 torch/distributed/run.py:803] ***************************************** +W0428 06:37:53.640000 324982 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0428 06:37:53.640000 324982 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + beta1: 0.9 + beta2: 0.95 + compressor: brotli + data_dir: ./data/ + datasets_dir: ./data/datasets/fineweb10B_sp8192 + distributed: True + ema_decay: 0.9965 + embed_bits: 7 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.35 + eval_seq_len: 2048 + eval_stride: 64 + gate_attn_out: True + gate_attn_width: 24 + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 4.0 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 20000 + ln_scale: True + local_rank: 0 + logfile: logs/nm_default.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 3 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 600.0 + min_lr: 0.0 + mlp_clip_sigmas: 12.0 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + newton_muon_all_reduce_k: True + newton_muon_beta: 0.95 + newton_muon_capture_every: 4 + newton_muon_enabled: True + newton_muon_gamma: 0.2 + newton_muon_k_refresh: 32 + newton_muon_warmup: 100 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 8 + phased_ttt_enabled: True + phased_ttt_num_phases: 3 + phased_ttt_prefix_docs: 2000 + polar_express_ns: True + qk_gain_init: 5.25 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: nm_default + scalar_lr: 0.02 + seed: 42 + skip_gates_enabled: True + sliding_window_enabled: False + smear_gate_enabled: True + smear_gate_width: 12 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./data/tokenizers/fineweb_8192_bpe.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: True + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 128 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192/fineweb_val_*.bin + val_loss_every: 4000 + vocab_size: 8192 + warmdown_frac: 0.75 + warmup_steps: 20 + world_size: 8 + xsa_last_n: 11 +train_shards: 80 +val_tokens: 40540160 +newton_muon: enabled (compile fullgraph=False, capture_every=4) +newton_muon: installed 12 forward-pre hooks (capture_every=4) +model_params:35946727 +gptq:reserving 4s, effective=596000ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +[rank4]:W0428 06:42:06.733000 325054 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank4]:W0428 06:42:06.733000 325054 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank4]:W0428 06:42:06.733000 325054 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank4]:W0428 06:42:06.733000 325054 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank4]:W0428 06:42:06.733000 325054 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank1]:W0428 06:42:07.010000 325051 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank1]:W0428 06:42:07.010000 325051 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank1]:W0428 06:42:07.010000 325051 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank1]:W0428 06:42:07.010000 325051 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank1]:W0428 06:42:07.010000 325051 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank5]:W0428 06:42:07.463000 325055 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank5]:W0428 06:42:07.463000 325055 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank5]:W0428 06:42:07.463000 325055 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank5]:W0428 06:42:07.463000 325055 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank5]:W0428 06:42:07.463000 325055 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank2]:W0428 06:42:08.363000 325052 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank2]:W0428 06:42:08.363000 325052 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank2]:W0428 06:42:08.363000 325052 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank2]:W0428 06:42:08.363000 325052 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank2]:W0428 06:42:08.363000 325052 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank6]:W0428 06:42:09.231000 325056 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank6]:W0428 06:42:09.231000 325056 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank6]:W0428 06:42:09.231000 325056 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank6]:W0428 06:42:09.231000 325056 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank6]:W0428 06:42:09.231000 325056 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank3]:W0428 06:42:09.277000 325053 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank3]:W0428 06:42:09.277000 325053 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank3]:W0428 06:42:09.277000 325053 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank3]:W0428 06:42:09.277000 325053 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank3]:W0428 06:42:09.277000 325053 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank7]:W0428 06:42:09.720000 325057 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank7]:W0428 06:42:09.720000 325057 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank7]:W0428 06:42:09.720000 325057 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank7]:W0428 06:42:09.720000 325057 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank7]:W0428 06:42:09.720000 325057 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank0]:W0428 06:42:10.085000 325050 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank0]:W0428 06:42:10.085000 325050 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank0]:W0428 06:42:10.085000 325050 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank0]:W0428 06:42:10.085000 325050 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank0]:W0428 06:42:10.085000 325050 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +warmup_step: 1/20 +warmup_step: 2/20 +warmup_step: 3/20 +warmup_step: 4/20 +warmup_step: 5/20 +warmup_step: 6/20 +warmup_step: 10/20 +warmup_step: 20/20 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/20 +loop_warmup_step: 2/20 +loop_warmup_step: 3/20 +loop_warmup_step: 4/20 +loop_warmup_step: 5/20 +loop_warmup_step: 6/20 +loop_warmup_step: 10/20 +loop_warmup_step: 20/20 +[rank7]:W0428 06:43:10.465000 325057 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank7]:W0428 06:43:10.465000 325057 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank7]:W0428 06:43:10.465000 325057 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank7]:W0428 06:43:10.465000 325057 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank7]:W0428 06:43:10.465000 325057 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank3]:W0428 06:43:10.535000 325053 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank3]:W0428 06:43:10.535000 325053 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank3]:W0428 06:43:10.535000 325053 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank3]:W0428 06:43:10.535000 325053 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank3]:W0428 06:43:10.535000 325053 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank5]:W0428 06:43:10.659000 325055 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank5]:W0428 06:43:10.659000 325055 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank5]:W0428 06:43:10.659000 325055 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank5]:W0428 06:43:10.659000 325055 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank5]:W0428 06:43:10.659000 325055 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank0]:W0428 06:43:11.312000 325050 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank0]:W0428 06:43:11.312000 325050 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank0]:W0428 06:43:11.312000 325050 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank0]:W0428 06:43:11.312000 325050 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank0]:W0428 06:43:11.312000 325050 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank2]:W0428 06:43:12.676000 325052 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank2]:W0428 06:43:12.676000 325052 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank2]:W0428 06:43:12.676000 325052 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank2]:W0428 06:43:12.676000 325052 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank2]:W0428 06:43:12.676000 325052 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank1]:W0428 06:43:13.210000 325051 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank1]:W0428 06:43:13.210000 325051 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank1]:W0428 06:43:13.210000 325051 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank1]:W0428 06:43:13.210000 325051 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank1]:W0428 06:43:13.210000 325051 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank4]:W0428 06:43:13.439000 325054 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank4]:W0428 06:43:13.439000 325054 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank4]:W0428 06:43:13.439000 325054 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 4325376 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank4]:W0428 06:43:13.439000 325054 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank4]:W0428 06:43:13.439000 325054 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank6]:W0428 06:43:24.940000 325056 torch/_dynamo/convert_frame.py:1358] [2/16] torch._dynamo hit config.recompile_limit (16) +[rank6]:W0428 06:43:24.940000 325056 torch/_dynamo/convert_frame.py:1358] [2/16] function: 'forward_logits' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:941) +[rank6]:W0428 06:43:24.940000 325056 torch/_dynamo/convert_frame.py:1358] [2/16] last reason: 2/15: tensor 'cu_seqlens' size mismatch at index 0. expected 64, actual 128 +[rank6]:W0428 06:43:24.940000 325056 torch/_dynamo/convert_frame.py:1358] [2/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank6]:W0428 06:43:24.940000 325056 torch/_dynamo/convert_frame.py:1358] [2/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +0/20000 val_loss: 9.0076 val_bpb: 3.4870 +1/20000 train_loss: 9.0080 train_time: 0.0m tok/s: 3636925 +2/20000 train_loss: 12.2485 train_time: 0.0m tok/s: 3587972 +3/20000 train_loss: 11.3002 train_time: 0.0m tok/s: 3516216 +4/20000 train_loss: 9.6261 train_time: 0.0m tok/s: 3470418 +5/20000 train_loss: 8.2273 train_time: 0.0m tok/s: 3447133 +500/20000 train_loss: 3.2466 train_time: 2.0m tok/s: 3341703 +layer_loop:enabled step:886 frac:0.350 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +1000/20000 train_loss: 2.9556 train_time: 4.1m tok/s: 3172612 +1500/20000 train_loss: 2.8844 train_time: 7.0m tok/s: 2796344 +2000/20000 train_loss: 2.7602 train_time: 9.9m tok/s: 2650490 +2008/20000 val_loss: 2.8623 val_bpb: 1.1080 +stopping_early: wallclock_cap train_time: 596195ms step: 2008/20000 +peak memory allocated: 54249 MiB reserved: 60418 MiB +ema:applying EMA weights +newton_muon: hooks uninstalled before EMA/quant/eval +diagnostic pre-quantization post-ema val_loss:2.87405954 val_bpb:1.11260216 eval_time:10200ms +Serialized model: 135422397 bytes +Code size (uncompressed): 134706 bytes +Code size (compressed): 33710 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians in 3.4s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + gptq (int7)+lqer_asym: tok_emb.weight + passthrough (float16): blocks.attn.attn_gate_proj.weight, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15928150 bytes +Total submission size quantized+brotli: 15961860 bytes +diagnostic quantized val_loss:2.89225739 val_bpb:1.11964688 eval_time:10067ms +ttt_lora:warming up compile (random tokens, no val data) +ttt_lora:compile warmup done (107.9s) + +beginning TTT eval timer +ttt_phased: total_docs:50000 prefix_docs:2000 suffix_docs:48000 num_phases:3 boundaries:[666, 1333, 2000] +ttp: b781/782 bl:2.6404 bb:1.0900 rl:2.6404 rb:1.0900 dl:14510-25988 gd:0 +ttpp: phase:1/3 pd:1104 gd:666 t:197.0s +tttg: c1/95 lr:0.001000 t:0.3s +tttg: c2/95 lr:0.001000 t:0.3s +tttg: c3/95 lr:0.000999 t:0.4s +tttg: c4/95 lr:0.000997 t:0.5s +tttg: c5/95 lr:0.000996 t:0.6s +tttg: c6/95 lr:0.000993 t:0.7s +tttg: c7/95 lr:0.000990 t:0.8s +tttg: c8/95 lr:0.000986 t:0.9s +tttg: c9/95 lr:0.000982 t:0.9s +tttg: c10/95 lr:0.000978 t:1.0s +tttg: c11/95 lr:0.000972 t:1.1s +tttg: c12/95 lr:0.000967 t:1.2s +tttg: c13/95 lr:0.000960 t:1.2s +tttg: c14/95 lr:0.000954 t:1.3s +tttg: c15/95 lr:0.000946 t:1.4s +tttg: c16/95 lr:0.000938 t:1.5s +tttg: c17/95 lr:0.000930 t:1.5s +tttg: c18/95 lr:0.000921 t:1.6s +tttg: c19/95 lr:0.000912 t:1.7s +tttg: c20/95 lr:0.000903 t:1.8s +tttg: c21/95 lr:0.000892 t:1.9s +tttg: c22/95 lr:0.000882 t:1.9s +tttg: c23/95 lr:0.000871 t:2.0s +tttg: c24/95 lr:0.000859 t:2.1s +tttg: c25/95 lr:0.000848 t:2.2s +tttg: c26/95 lr:0.000835 t:2.3s +tttg: c27/95 lr:0.000823 t:2.3s +tttg: c28/95 lr:0.000810 t:2.4s +tttg: c29/95 lr:0.000797 t:2.5s +tttg: c30/95 lr:0.000783 t:2.6s +tttg: c31/95 lr:0.000769 t:2.7s +tttg: c32/95 lr:0.000755 t:2.7s +tttg: c33/95 lr:0.000740 t:2.8s +tttg: c34/95 lr:0.000726 t:2.9s +tttg: c35/95 lr:0.000710 t:3.0s +tttg: c36/95 lr:0.000695 t:3.0s +tttg: c37/95 lr:0.000680 t:3.1s +tttg: c38/95 lr:0.000664 t:3.2s +tttg: c39/95 lr:0.000648 t:3.3s +tttg: c40/95 lr:0.000632 t:3.3s +tttg: c41/95 lr:0.000616 t:3.4s +tttg: c42/95 lr:0.000600 t:3.5s +tttg: c43/95 lr:0.000583 t:3.6s +tttg: c44/95 lr:0.000567 t:3.7s +tttg: c45/95 lr:0.000550 t:3.7s +tttg: c46/95 lr:0.000533 t:3.8s +tttg: c47/95 lr:0.000517 t:3.9s +tttg: c48/95 lr:0.000500 t:4.0s +tttg: c49/95 lr:0.000483 t:4.0s +tttg: c50/95 lr:0.000467 t:4.1s +tttg: c51/95 lr:0.000450 t:4.2s +tttg: c52/95 lr:0.000433 t:4.3s +tttg: c53/95 lr:0.000417 t:4.4s +tttg: c54/95 lr:0.000400 t:4.4s +tttg: c55/95 lr:0.000384 t:4.5s +tttg: c56/95 lr:0.000368 t:4.6s +tttg: c57/95 lr:0.000352 t:4.7s +tttg: c58/95 lr:0.000336 t:4.7s +tttg: c59/95 lr:0.000320 t:4.8s +tttg: c60/95 lr:0.000305 t:4.9s +tttg: c61/95 lr:0.000290 t:5.0s +tttg: c62/95 lr:0.000274 t:5.1s +tttg: c63/95 lr:0.000260 t:5.1s +tttg: c64/95 lr:0.000245 t:5.2s +tttg: c65/95 lr:0.000231 t:5.3s +tttg: c66/95 lr:0.000217 t:5.4s +tttg: c67/95 lr:0.000203 t:5.5s +tttg: c68/95 lr:0.000190 t:5.5s +tttg: c69/95 lr:0.000177 t:5.6s +tttg: c70/95 lr:0.000165 t:5.7s +tttg: c71/95 lr:0.000152 t:5.8s +tttg: c72/95 lr:0.000141 t:5.8s +tttg: c73/95 lr:0.000129 t:5.9s +tttg: c74/95 lr:0.000118 t:6.0s +tttg: c75/95 lr:0.000108 t:6.1s +tttg: c76/95 lr:0.000097 t:6.1s +tttg: c77/95 lr:0.000088 t:6.2s +tttg: c78/95 lr:0.000079 t:6.3s +tttg: c79/95 lr:0.000070 t:6.4s +tttg: c80/95 lr:0.000062 t:6.5s +tttg: c81/95 lr:0.000054 t:6.5s +tttg: c82/95 lr:0.000046 t:6.6s +tttg: c83/95 lr:0.000040 t:6.7s +tttg: c84/95 lr:0.000033 t:6.8s +tttg: c85/95 lr:0.000028 t:6.9s +tttg: c86/95 lr:0.000022 t:6.9s +tttg: c87/95 lr:0.000018 t:7.0s +tttg: c88/95 lr:0.000014 t:7.1s +tttg: c89/95 lr:0.000010 t:7.2s +tttg: c90/95 lr:0.000007 t:7.3s +tttg: c91/95 lr:0.000004 t:7.3s +tttg: c92/95 lr:0.000003 t:7.4s +tttg: c93/95 lr:0.000001 t:7.5s +tttg: c94/95 lr:0.000000 t:7.6s +ttpr: phase:1/3 t:207.4s +ttp: b760/782 bl:2.9296 bb:1.1505 rl:2.6836 rb:1.0994 dl:3255-3334 gd:0 +ttpp: phase:2/3 pd:1808 gd:1333 t:276.0s +tttg: c1/158 lr:0.001000 t:0.1s +tttg: c2/158 lr:0.001000 t:0.2s +tttg: c3/158 lr:0.001000 t:0.2s +tttg: c4/158 lr:0.000999 t:0.3s +tttg: c5/158 lr:0.000998 t:0.4s +tttg: c6/158 lr:0.000997 t:0.5s +tttg: c7/158 lr:0.000996 t:0.6s +tttg: c8/158 lr:0.000995 t:0.6s +tttg: c9/158 lr:0.000994 t:0.7s +tttg: c10/158 lr:0.000992 t:0.8s +tttg: c11/158 lr:0.000990 t:0.9s +tttg: c12/158 lr:0.000988 t:0.9s +tttg: c13/158 lr:0.000986 t:1.0s +tttg: c14/158 lr:0.000983 t:1.1s +tttg: c15/158 lr:0.000981 t:1.2s +tttg: c16/158 lr:0.000978 t:1.3s +tttg: c17/158 lr:0.000975 t:1.3s +tttg: c18/158 lr:0.000971 t:1.4s +tttg: c19/158 lr:0.000968 t:1.5s +tttg: c20/158 lr:0.000964 t:1.6s +tttg: c21/158 lr:0.000960 t:1.6s +tttg: c22/158 lr:0.000957 t:1.7s +tttg: c23/158 lr:0.000952 t:1.8s +tttg: c24/158 lr:0.000948 t:1.9s +tttg: c25/158 lr:0.000943 t:2.0s +tttg: c26/158 lr:0.000939 t:2.1s +tttg: c27/158 lr:0.000934 t:2.1s +tttg: c28/158 lr:0.000929 t:2.2s +tttg: c29/158 lr:0.000924 t:2.3s +tttg: c30/158 lr:0.000918 t:2.4s +tttg: c31/158 lr:0.000913 t:2.4s +tttg: c32/158 lr:0.000907 t:2.5s +tttg: c33/158 lr:0.000901 t:2.6s +tttg: c34/158 lr:0.000895 t:2.7s +tttg: c35/158 lr:0.000889 t:2.7s +tttg: c36/158 lr:0.000882 t:2.8s +tttg: c37/158 lr:0.000876 t:2.9s +tttg: c38/158 lr:0.000869 t:3.0s +tttg: c39/158 lr:0.000862 t:3.1s +tttg: c40/158 lr:0.000855 t:3.1s +tttg: c41/158 lr:0.000848 t:3.2s +tttg: c42/158 lr:0.000841 t:3.3s +tttg: c43/158 lr:0.000834 t:3.4s +tttg: c44/158 lr:0.000826 t:3.4s +tttg: c45/158 lr:0.000818 t:3.5s +tttg: c46/158 lr:0.000811 t:3.6s +tttg: c47/158 lr:0.000803 t:3.7s +tttg: c48/158 lr:0.000795 t:3.8s +tttg: c49/158 lr:0.000787 t:3.8s +tttg: c50/158 lr:0.000778 t:3.9s +tttg: c51/158 lr:0.000770 t:4.0s +tttg: c52/158 lr:0.000761 t:4.1s +tttg: c53/158 lr:0.000753 t:4.1s +tttg: c54/158 lr:0.000744 t:4.2s +tttg: c55/158 lr:0.000735 t:4.3s +tttg: c56/158 lr:0.000727 t:4.4s +tttg: c57/158 lr:0.000718 t:4.5s +tttg: c58/158 lr:0.000709 t:4.5s +tttg: c59/158 lr:0.000699 t:4.6s +tttg: c60/158 lr:0.000690 t:4.7s +tttg: c61/158 lr:0.000681 t:4.8s +tttg: c62/158 lr:0.000672 t:4.8s +tttg: c63/158 lr:0.000662 t:4.9s +tttg: c64/158 lr:0.000653 t:5.0s +tttg: c65/158 lr:0.000643 t:5.1s +tttg: c66/158 lr:0.000633 t:5.1s +tttg: c67/158 lr:0.000624 t:5.2s +tttg: c68/158 lr:0.000614 t:5.3s +tttg: c69/158 lr:0.000604 t:5.4s +tttg: c70/158 lr:0.000594 t:5.5s +tttg: c71/158 lr:0.000585 t:5.5s +tttg: c72/158 lr:0.000575 t:5.6s +tttg: c73/158 lr:0.000565 t:5.7s +tttg: c74/158 lr:0.000555 t:5.8s +tttg: c75/158 lr:0.000545 t:5.8s +tttg: c76/158 lr:0.000535 t:5.9s +tttg: c77/158 lr:0.000525 t:6.0s +tttg: c78/158 lr:0.000515 t:6.1s +tttg: c79/158 lr:0.000505 t:6.2s +tttg: c80/158 lr:0.000495 t:6.2s +tttg: c81/158 lr:0.000485 t:6.3s +tttg: c82/158 lr:0.000475 t:6.4s +tttg: c83/158 lr:0.000465 t:6.5s +tttg: c84/158 lr:0.000455 t:6.6s +tttg: c85/158 lr:0.000445 t:6.7s +tttg: c86/158 lr:0.000435 t:6.8s +tttg: c87/158 lr:0.000425 t:6.8s +tttg: c88/158 lr:0.000415 t:6.9s +tttg: c89/158 lr:0.000406 t:7.0s +tttg: c90/158 lr:0.000396 t:7.1s +tttg: c91/158 lr:0.000386 t:7.1s +tttg: c92/158 lr:0.000376 t:7.2s +tttg: c93/158 lr:0.000367 t:7.3s +tttg: c94/158 lr:0.000357 t:7.4s +tttg: c95/158 lr:0.000347 t:7.4s +tttg: c96/158 lr:0.000338 t:7.5s +tttg: c97/158 lr:0.000328 t:7.6s +tttg: c98/158 lr:0.000319 t:7.7s +tttg: c99/158 lr:0.000310 t:7.7s +tttg: c100/158 lr:0.000301 t:7.8s +tttg: c101/158 lr:0.000291 t:7.9s +tttg: c102/158 lr:0.000282 t:8.0s +tttg: c103/158 lr:0.000273 t:8.1s +tttg: c104/158 lr:0.000265 t:8.1s +tttg: c105/158 lr:0.000256 t:10.9s +tttg: c106/158 lr:0.000247 t:11.0s +tttg: c107/158 lr:0.000239 t:11.1s +tttg: c108/158 lr:0.000230 t:11.1s +tttg: c109/158 lr:0.000222 t:11.2s +tttg: c110/158 lr:0.000213 t:11.3s +tttg: c111/158 lr:0.000205 t:11.4s +tttg: c112/158 lr:0.000197 t:11.5s +tttg: c113/158 lr:0.000189 t:11.6s +tttg: c114/158 lr:0.000182 t:11.6s +tttg: c115/158 lr:0.000174 t:11.7s +tttg: c116/158 lr:0.000166 t:11.8s +tttg: c117/158 lr:0.000159 t:11.9s +tttg: c118/158 lr:0.000152 t:11.9s +tttg: c119/158 lr:0.000145 t:12.0s +tttg: c120/158 lr:0.000138 t:12.1s +tttg: c121/158 lr:0.000131 t:12.2s +tttg: c122/158 lr:0.000124 t:12.2s +tttg: c123/158 lr:0.000118 t:12.3s +tttg: c124/158 lr:0.000111 t:12.4s +tttg: c125/158 lr:0.000105 t:12.5s +tttg: c126/158 lr:0.000099 t:12.5s +tttg: c127/158 lr:0.000093 t:12.6s +tttg: c128/158 lr:0.000087 t:12.7s +tttg: c129/158 lr:0.000082 t:12.8s +tttg: c130/158 lr:0.000076 t:12.9s +tttg: c131/158 lr:0.000071 t:12.9s +tttg: c132/158 lr:0.000066 t:13.0s +tttg: c133/158 lr:0.000061 t:13.1s +tttg: c134/158 lr:0.000057 t:13.2s +tttg: c135/158 lr:0.000052 t:13.2s +tttg: c136/158 lr:0.000048 t:13.3s +tttg: c137/158 lr:0.000043 t:13.4s +tttg: c138/158 lr:0.000040 t:13.5s +tttg: c139/158 lr:0.000036 t:13.6s +tttg: c140/158 lr:0.000032 t:13.6s +tttg: c141/158 lr:0.000029 t:13.7s +tttg: c142/158 lr:0.000025 t:13.8s +tttg: c143/158 lr:0.000022 t:13.9s +tttg: c144/158 lr:0.000019 t:13.9s +tttg: c145/158 lr:0.000017 t:14.0s +tttg: c146/158 lr:0.000014 t:14.1s +tttg: c147/158 lr:0.000012 t:14.2s +tttg: c148/158 lr:0.000010 t:14.2s +tttg: c149/158 lr:0.000008 t:14.3s +tttg: c150/158 lr:0.000006 t:14.4s +tttg: c151/158 lr:0.000005 t:14.5s +tttg: c152/158 lr:0.000004 t:14.6s +tttg: c153/158 lr:0.000003 t:14.7s +tttg: c154/158 lr:0.000002 t:14.7s +tttg: c155/158 lr:0.000001 t:14.8s +tttg: c156/158 lr:0.000000 t:14.9s +tttg: c157/158 lr:0.000000 t:15.0s +ttpr: phase:2/3 t:293.8s +ttp: b747/782 bl:2.8703 bb:1.0917 rl:2.7027 rb:1.0986 dl:2501-2538 gd:0 +ttp: b744/782 bl:2.7403 bb:1.0917 rl:2.7060 rb:1.0980 dl:2388-2419 gd:0 +ttpp: phase:3/3 pd:2448 gd:2000 t:307.8s +tttg: c1/213 lr:0.001000 t:0.1s +tttg: c2/213 lr:0.001000 t:0.2s +tttg: c3/213 lr:0.001000 t:0.2s +tttg: c4/213 lr:0.001000 t:0.3s +tttg: c5/213 lr:0.000999 t:0.4s +tttg: c6/213 lr:0.000999 t:0.5s +tttg: c7/213 lr:0.000998 t:0.5s +tttg: c8/213 lr:0.000997 t:0.6s +tttg: c9/213 lr:0.000996 t:0.7s +tttg: c10/213 lr:0.000996 t:0.8s +tttg: c11/213 lr:0.000995 t:0.8s +tttg: c12/213 lr:0.000993 t:0.9s +tttg: c13/213 lr:0.000992 t:1.0s +tttg: c14/213 lr:0.000991 t:1.1s +tttg: c15/213 lr:0.000989 t:1.1s +tttg: c16/213 lr:0.000988 t:1.2s +tttg: c17/213 lr:0.000986 t:1.3s +tttg: c18/213 lr:0.000984 t:1.4s +tttg: c19/213 lr:0.000982 t:1.5s +tttg: c20/213 lr:0.000980 t:1.5s +tttg: c21/213 lr:0.000978 t:1.6s +tttg: c22/213 lr:0.000976 t:1.7s +tttg: c23/213 lr:0.000974 t:1.8s +tttg: c24/213 lr:0.000971 t:1.8s +tttg: c25/213 lr:0.000969 t:1.9s +tttg: c26/213 lr:0.000966 t:2.0s +tttg: c27/213 lr:0.000963 t:2.1s +tttg: c28/213 lr:0.000961 t:2.2s +tttg: c29/213 lr:0.000958 t:2.2s +tttg: c30/213 lr:0.000955 t:2.3s +tttg: c31/213 lr:0.000951 t:2.4s +tttg: c32/213 lr:0.000948 t:2.5s +tttg: c33/213 lr:0.000945 t:2.5s +tttg: c34/213 lr:0.000941 t:2.6s +tttg: c35/213 lr:0.000938 t:2.7s +tttg: c36/213 lr:0.000934 t:2.8s +tttg: c37/213 lr:0.000931 t:2.8s +tttg: c38/213 lr:0.000927 t:2.9s +tttg: c39/213 lr:0.000923 t:3.0s +tttg: c40/213 lr:0.000919 t:3.1s +tttg: c41/213 lr:0.000915 t:3.2s +tttg: c42/213 lr:0.000911 t:3.2s +tttg: c43/213 lr:0.000906 t:3.3s +tttg: c44/213 lr:0.000902 t:3.4s +tttg: c45/213 lr:0.000897 t:3.5s +tttg: c46/213 lr:0.000893 t:3.5s +tttg: c47/213 lr:0.000888 t:3.6s +tttg: c48/213 lr:0.000884 t:3.7s +tttg: c49/213 lr:0.000879 t:3.8s +tttg: c50/213 lr:0.000874 t:3.9s +tttg: c51/213 lr:0.000869 t:3.9s +tttg: c52/213 lr:0.000864 t:4.0s +tttg: c53/213 lr:0.000859 t:4.1s +tttg: c54/213 lr:0.000854 t:4.2s +tttg: c55/213 lr:0.000848 t:4.3s +tttg: c56/213 lr:0.000843 t:4.3s +tttg: c57/213 lr:0.000837 t:4.4s +tttg: c58/213 lr:0.000832 t:4.5s +tttg: c59/213 lr:0.000826 t:4.6s +tttg: c60/213 lr:0.000821 t:4.6s +tttg: c61/213 lr:0.000815 t:4.7s +tttg: c62/213 lr:0.000809 t:4.8s +tttg: c63/213 lr:0.000803 t:4.9s +tttg: c64/213 lr:0.000797 t:5.0s +tttg: c65/213 lr:0.000791 t:5.0s +tttg: c66/213 lr:0.000785 t:5.1s +tttg: c67/213 lr:0.000779 t:5.2s +tttg: c68/213 lr:0.000773 t:5.3s +tttg: c69/213 lr:0.000767 t:5.4s +tttg: c70/213 lr:0.000761 t:5.4s +tttg: c71/213 lr:0.000754 t:5.5s +tttg: c72/213 lr:0.000748 t:5.6s +tttg: c73/213 lr:0.000741 t:5.7s +tttg: c74/213 lr:0.000735 t:5.7s +tttg: c75/213 lr:0.000728 t:5.8s +tttg: c76/213 lr:0.000722 t:5.9s +tttg: c77/213 lr:0.000715 t:6.0s +tttg: c78/213 lr:0.000708 t:6.0s +tttg: c79/213 lr:0.000702 t:6.1s +tttg: c80/213 lr:0.000695 t:6.2s +tttg: c81/213 lr:0.000688 t:6.3s +tttg: c82/213 lr:0.000681 t:6.4s +tttg: c83/213 lr:0.000674 t:6.4s +tttg: c84/213 lr:0.000667 t:6.5s +tttg: c85/213 lr:0.000660 t:6.6s +tttg: c86/213 lr:0.000653 t:6.7s +tttg: c87/213 lr:0.000646 t:6.8s +tttg: c88/213 lr:0.000639 t:6.8s +tttg: c89/213 lr:0.000632 t:6.9s +tttg: c90/213 lr:0.000625 t:7.0s +tttg: c91/213 lr:0.000617 t:7.1s +tttg: c92/213 lr:0.000610 t:7.1s +tttg: c93/213 lr:0.000603 t:7.2s +tttg: c94/213 lr:0.000596 t:7.3s +tttg: c95/213 lr:0.000588 t:7.4s +tttg: c96/213 lr:0.000581 t:7.4s +tttg: c97/213 lr:0.000574 t:7.5s +tttg: c98/213 lr:0.000566 t:7.6s +tttg: c99/213 lr:0.000559 t:7.7s +tttg: c100/213 lr:0.000552 t:7.8s +tttg: c101/213 lr:0.000544 t:7.8s +tttg: c102/213 lr:0.000537 t:7.9s +tttg: c103/213 lr:0.000530 t:8.0s +tttg: c104/213 lr:0.000522 t:8.1s +tttg: c105/213 lr:0.000515 t:8.2s +tttg: c106/213 lr:0.000507 t:8.2s +tttg: c107/213 lr:0.000500 t:8.3s +tttg: c108/213 lr:0.000493 t:8.4s +tttg: c109/213 lr:0.000485 t:8.5s +tttg: c110/213 lr:0.000478 t:8.5s +tttg: c111/213 lr:0.000470 t:8.6s +tttg: c112/213 lr:0.000463 t:8.7s +tttg: c113/213 lr:0.000456 t:8.8s +tttg: c114/213 lr:0.000448 t:8.8s +tttg: c115/213 lr:0.000441 t:8.9s +tttg: c116/213 lr:0.000434 t:9.0s +tttg: c117/213 lr:0.000426 t:9.1s +tttg: c118/213 lr:0.000419 t:9.2s +tttg: c119/213 lr:0.000412 t:9.2s +tttg: c120/213 lr:0.000404 t:9.3s +tttg: c121/213 lr:0.000397 t:9.4s +tttg: c122/213 lr:0.000390 t:9.5s +tttg: c123/213 lr:0.000383 t:9.5s +tttg: c124/213 lr:0.000375 t:9.6s +tttg: c125/213 lr:0.000368 t:9.7s +tttg: c126/213 lr:0.000361 t:9.8s +tttg: c127/213 lr:0.000354 t:9.9s +tttg: c128/213 lr:0.000347 t:9.9s +tttg: c129/213 lr:0.000340 t:10.0s +tttg: c130/213 lr:0.000333 t:10.1s +tttg: c131/213 lr:0.000326 t:10.2s +tttg: c132/213 lr:0.000319 t:10.3s +tttg: c133/213 lr:0.000312 t:10.3s +tttg: c134/213 lr:0.000305 t:10.4s +tttg: c135/213 lr:0.000298 t:10.5s +tttg: c136/213 lr:0.000292 t:10.6s +tttg: c137/213 lr:0.000285 t:10.6s +tttg: c138/213 lr:0.000278 t:10.7s +tttg: c139/213 lr:0.000272 t:10.8s +tttg: c140/213 lr:0.000265 t:10.9s +tttg: c141/213 lr:0.000259 t:10.9s +tttg: c142/213 lr:0.000252 t:11.0s +tttg: c143/213 lr:0.000246 t:11.1s +tttg: c144/213 lr:0.000239 t:11.2s +tttg: c145/213 lr:0.000233 t:11.3s +tttg: c146/213 lr:0.000227 t:11.3s +tttg: c147/213 lr:0.000221 t:11.4s +tttg: c148/213 lr:0.000215 t:11.5s +tttg: c149/213 lr:0.000209 t:11.6s +tttg: c150/213 lr:0.000203 t:11.6s +tttg: c151/213 lr:0.000197 t:11.7s +tttg: c152/213 lr:0.000191 t:11.8s +tttg: c153/213 lr:0.000185 t:11.9s +tttg: c154/213 lr:0.000179 t:11.9s +tttg: c155/213 lr:0.000174 t:12.0s +tttg: c156/213 lr:0.000168 t:12.1s +tttg: c157/213 lr:0.000163 t:12.2s +tttg: c158/213 lr:0.000157 t:12.2s +tttg: c159/213 lr:0.000152 t:12.3s +tttg: c160/213 lr:0.000146 t:12.4s +tttg: c161/213 lr:0.000141 t:12.5s +tttg: c162/213 lr:0.000136 t:12.6s +tttg: c163/213 lr:0.000131 t:12.6s +tttg: c164/213 lr:0.000126 t:12.7s +tttg: c165/213 lr:0.000121 t:12.8s +tttg: c166/213 lr:0.000116 t:12.9s +tttg: c167/213 lr:0.000112 t:12.9s +tttg: c168/213 lr:0.000107 t:13.0s +tttg: c169/213 lr:0.000103 t:13.1s +tttg: c170/213 lr:0.000098 t:13.2s +tttg: c171/213 lr:0.000094 t:13.3s +tttg: c172/213 lr:0.000089 t:13.3s +tttg: c173/213 lr:0.000085 t:13.4s +tttg: c174/213 lr:0.000081 t:13.5s +tttg: c175/213 lr:0.000077 t:13.6s +tttg: c176/213 lr:0.000073 t:13.6s +tttg: c177/213 lr:0.000069 t:13.7s +tttg: c178/213 lr:0.000066 t:13.8s +tttg: c179/213 lr:0.000062 t:13.9s +tttg: c180/213 lr:0.000059 t:13.9s +tttg: c181/213 lr:0.000055 t:14.0s +tttg: c182/213 lr:0.000052 t:14.1s +tttg: c183/213 lr:0.000049 t:14.2s +tttg: c184/213 lr:0.000045 t:14.3s +tttg: c185/213 lr:0.000042 t:14.3s +tttg: c186/213 lr:0.000039 t:14.4s +tttg: c187/213 lr:0.000037 t:14.5s +tttg: c188/213 lr:0.000034 t:14.6s +tttg: c189/213 lr:0.000031 t:14.7s +tttg: c190/213 lr:0.000029 t:14.7s +tttg: c191/213 lr:0.000026 t:14.8s +tttg: c192/213 lr:0.000024 t:14.9s +tttg: c193/213 lr:0.000022 t:15.0s +tttg: c194/213 lr:0.000020 t:15.0s +tttg: c195/213 lr:0.000018 t:15.1s +tttg: c196/213 lr:0.000016 t:15.2s +tttg: c197/213 lr:0.000014 t:15.3s +tttg: c198/213 lr:0.000012 t:15.3s +tttg: c199/213 lr:0.000011 t:15.4s +tttg: c200/213 lr:0.000009 t:15.5s +tttg: c201/213 lr:0.000008 t:15.6s +tttg: c202/213 lr:0.000007 t:15.6s +tttg: c203/213 lr:0.000005 t:15.7s +tttg: c204/213 lr:0.000004 t:15.8s +tttg: c205/213 lr:0.000004 t:15.9s +tttg: c206/213 lr:0.000003 t:16.0s +tttg: c207/213 lr:0.000002 t:16.1s +tttg: c208/213 lr:0.000001 t:16.1s +tttg: c209/213 lr:0.000001 t:16.2s +tttg: c210/213 lr:0.000000 t:16.3s +tttg: c211/213 lr:0.000000 t:16.4s +tttg: c212/213 lr:0.000000 t:16.5s +ttpr: phase:3/3 t:327.1s +ttp: b743/782 bl:2.7948 bb:1.0760 rl:2.7132 rb:1.0961 dl:2355-2388 gd:1 +ttp: b729/782 bl:2.8033 bb:1.0683 rl:2.7189 rb:1.0942 dl:1978-1994 gd:1 +ttp: b721/782 bl:2.8352 bb:1.0583 rl:2.7253 rb:1.0921 dl:1832-1846 gd:1 +ttp: b716/782 bl:2.8902 bb:1.0665 rl:2.7336 rb:1.0907 dl:1739-1754 gd:1 +ttp: b709/782 bl:2.8742 bb:1.0921 rl:2.7399 rb:1.0908 dl:1649-1661 gd:1 +ttp: b701/782 bl:2.8425 bb:1.0810 rl:2.7441 rb:1.0904 dl:1562-1572 gd:1 +ttp: b694/782 bl:2.8456 bb:1.0983 rl:2.7480 rb:1.0907 dl:1494-1504 gd:1 +ttp: b683/782 bl:2.8570 bb:1.1000 rl:2.7517 rb:1.0910 dl:1400-1406 gd:1 +ttp: b675/782 bl:2.9220 bb:1.0969 rl:2.7571 rb:1.0912 dl:1341-1347 gd:1 +ttp: b666/782 bl:2.9115 bb:1.0942 rl:2.7616 rb:1.0913 dl:1282-1288 gd:1 +ttp: b663/782 bl:2.8898 bb:1.0968 rl:2.7652 rb:1.0915 dl:1264-1269 gd:1 +ttp: b652/782 bl:2.8865 bb:1.1060 rl:2.7684 rb:1.0918 dl:1198-1203 gd:1 +ttp: b644/782 bl:2.8205 bb:1.0643 rl:2.7697 rb:1.0911 dl:1155-1160 gd:1 +ttp: b635/782 bl:2.8301 bb:1.0955 rl:2.7711 rb:1.0912 dl:1111-1116 gd:1 +ttp: b626/782 bl:2.8972 bb:1.0765 rl:2.7738 rb:1.0909 dl:1068-1073 gd:1 +ttp: b618/782 bl:2.8237 bb:1.0825 rl:2.7748 rb:1.0907 dl:1031-1037 gd:1 +ttp: b612/782 bl:2.9108 bb:1.0752 rl:2.7775 rb:1.0904 dl:1007-1012 gd:1 +ttp: b601/782 bl:2.8538 bb:1.0968 rl:2.7789 rb:1.0905 dl:963-966 gd:1 +ttp: b593/782 bl:2.8825 bb:1.0780 rl:2.7807 rb:1.0903 dl:933-937 gd:1 +ttp: b587/782 bl:2.8711 bb:1.0963 rl:2.7822 rb:1.0904 dl:914-917 gd:1 +ttp: b578/782 bl:2.8887 bb:1.1006 rl:2.7839 rb:1.0906 dl:884-887 gd:1 +ttp: b570/782 bl:2.8688 bb:1.1165 rl:2.7852 rb:1.0910 dl:858-862 gd:1 +ttp: b567/782 bl:2.7597 bb:1.0630 rl:2.7848 rb:1.0905 dl:849-852 gd:1 +ttp: b559/782 bl:2.8375 bb:1.0784 rl:2.7856 rb:1.0904 dl:824-827 gd:1 +ttp: b548/782 bl:2.8505 bb:1.0808 rl:2.7865 rb:1.0902 dl:793-795 gd:1 +ttp: b539/782 bl:2.8186 bb:1.0792 rl:2.7869 rb:1.0901 dl:769-771 gd:1 +ttp: b529/782 bl:2.8636 bb:1.0907 rl:2.7878 rb:1.0901 dl:745-747 gd:1 +ttp: b521/782 bl:2.8583 bb:1.0845 rl:2.7887 rb:1.0900 dl:725-727 gd:1 +ttp: b514/782 bl:2.9948 bb:1.1296 rl:2.7910 rb:1.0905 dl:707-710 gd:1 +ttp: b509/782 bl:2.8406 bb:1.1054 rl:2.7916 rb:1.0907 dl:695-698 gd:1 +ttp: b501/782 bl:2.8809 bb:1.0732 rl:2.7926 rb:1.0905 dl:677-680 gd:1 +ttp: b492/782 bl:2.8915 bb:1.0874 rl:2.7936 rb:1.0904 dl:657-659 gd:1 +ttp: b484/782 bl:2.8882 bb:1.1023 rl:2.7945 rb:1.0906 dl:641-643 gd:1 +ttp: b477/782 bl:2.8666 bb:1.0890 rl:2.7952 rb:1.0905 dl:626-628 gd:1 +ttp: b466/782 bl:2.8868 bb:1.0975 rl:2.7961 rb:1.0906 dl:604-606 gd:1 +ttp: b458/782 bl:2.8975 bb:1.0983 rl:2.7970 rb:1.0907 dl:589-591 gd:1 +ttp: b450/782 bl:2.8508 bb:1.0640 rl:2.7974 rb:1.0904 dl:575-576 gd:1 +ttp: b448/782 bl:2.8116 bb:1.0681 rl:2.7976 rb:1.0902 dl:571-573 gd:1 +ttp: b440/782 bl:2.9534 bb:1.1276 rl:2.7988 rb:1.0906 dl:556-559 gd:1 +ttp: b432/782 bl:2.8495 bb:1.0841 rl:2.7992 rb:1.0905 dl:542-544 gd:1 +ttp: b424/782 bl:2.8806 bb:1.1133 rl:2.7999 rb:1.0907 dl:528-530 gd:1 +ttp: b411/782 bl:2.9032 bb:1.1069 rl:2.8006 rb:1.0908 dl:507-508 gd:1 +ttp: b405/782 bl:2.9226 bb:1.1046 rl:2.8015 rb:1.0909 dl:497-498 gd:1 +ttp: b398/782 bl:2.9725 bb:1.1290 rl:2.8027 rb:1.0912 dl:486-487 gd:1 +ttp: b391/782 bl:2.9161 bb:1.1357 rl:2.8034 rb:1.0915 dl:475-476 gd:1 +ttp: b385/782 bl:2.9846 bb:1.1373 rl:2.8046 rb:1.0918 dl:466-467 gd:1 +ttp: b377/782 bl:2.8953 bb:1.1226 rl:2.8052 rb:1.0920 dl:454-455 gd:1 +ttp: b369/782 bl:3.0238 bb:1.1224 rl:2.8065 rb:1.0922 dl:443-444 gd:1 +ttp: b362/782 bl:2.9096 bb:1.1002 rl:2.8072 rb:1.0922 dl:433-434 gd:1 +ttp: b351/782 bl:2.9236 bb:1.1252 rl:2.8078 rb:1.0924 dl:418-419 gd:1 +ttp: b342/782 bl:2.9620 bb:1.1396 rl:2.8087 rb:1.0927 dl:406-407 gd:1 +ttp: b335/782 bl:2.8052 bb:1.1243 rl:2.8087 rb:1.0928 dl:396-398 gd:1 +ttp: b327/782 bl:2.8675 bb:1.1132 rl:2.8090 rb:1.0929 dl:387-388 gd:1 +ttp: b319/782 bl:2.9171 bb:1.1444 rl:2.8095 rb:1.0932 dl:376-377 gd:1 +ttp: b311/782 bl:2.9526 bb:1.1311 rl:2.8102 rb:1.0934 dl:365-367 gd:1 +ttp: b303/782 bl:2.8998 bb:1.1234 rl:2.8106 rb:1.0935 dl:355-357 gd:1 +ttp: b295/782 bl:2.9316 bb:1.1559 rl:2.8112 rb:1.0938 dl:345-347 gd:1 +ttp: b287/782 bl:2.9478 bb:1.1499 rl:2.8118 rb:1.0941 dl:336-337 gd:1 +ttp: b284/782 bl:2.9689 bb:1.1193 rl:2.8125 rb:1.0942 dl:333-334 gd:1 +ttp: b276/782 bl:2.9384 bb:1.1389 rl:2.8130 rb:1.0944 dl:324-325 gd:1 +ttp: b268/782 bl:2.9476 bb:1.1333 rl:2.8135 rb:1.0945 dl:315-316 gd:1 +ttp: b260/782 bl:2.9194 bb:1.1388 rl:2.8140 rb:1.0947 dl:306-307 gd:1 +ttp: b252/782 bl:2.9829 bb:1.1623 rl:2.8146 rb:1.0950 dl:297-298 gd:1 +ttp: b244/782 bl:3.0394 bb:1.1921 rl:2.8154 rb:1.0953 dl:289-290 gd:1 +ttp: b235/782 bl:3.0207 bb:1.1482 rl:2.8162 rb:1.0955 dl:280-281 gd:1 +ttp: b227/782 bl:2.8921 bb:1.1216 rl:2.8164 rb:1.0956 dl:272-273 gd:1 +ttp: b219/782 bl:2.9903 bb:1.1667 rl:2.8170 rb:1.0958 dl:264-265 gd:1 +ttp: b209/782 bl:3.0017 bb:1.1883 rl:2.8176 rb:1.0961 dl:254-255 gd:1 +ttp: b199/782 bl:3.0280 bb:1.1605 rl:2.8183 rb:1.0963 dl:246-247 gd:1 +ttp: b192/782 bl:2.9983 bb:1.1819 rl:2.8188 rb:1.0966 dl:239-240 gd:1 +ttp: b188/782 bl:3.0066 bb:1.1911 rl:2.8194 rb:1.0969 dl:236-237 gd:1 +ttp: b178/782 bl:2.9516 bb:1.1772 rl:2.8198 rb:1.0971 dl:227-228 gd:1 +ttp: b172/782 bl:3.0938 bb:1.2167 rl:2.8205 rb:1.0974 dl:222-223 gd:1 +ttp: b165/782 bl:3.0288 bb:1.1985 rl:2.8211 rb:1.0977 dl:216-217 gd:1 +ttp: b160/782 bl:2.9580 bb:1.1626 rl:2.8214 rb:1.0978 dl:212-212 gd:1 +ttp: b152/782 bl:2.9781 bb:1.1620 rl:2.8218 rb:1.0980 dl:205-206 gd:1 +ttp: b144/782 bl:2.9308 bb:1.1658 rl:2.8221 rb:1.0982 dl:199-200 gd:1 +ttp: b139/782 bl:3.0869 bb:1.1947 rl:2.8227 rb:1.0984 dl:195-195 gd:1 +ttp: b132/782 bl:3.0445 bb:1.1717 rl:2.8232 rb:1.0986 dl:189-189 gd:1 +ttp: b124/782 bl:2.9703 bb:1.1882 rl:2.8236 rb:1.0988 dl:183-184 gd:1 +ttp: b116/782 bl:3.1007 bb:1.2262 rl:2.8242 rb:1.0990 dl:177-178 gd:1 +ttp: b109/782 bl:3.1554 bb:1.2435 rl:2.8249 rb:1.0993 dl:172-173 gd:1 +ttp: b102/782 bl:2.8998 bb:1.1677 rl:2.8250 rb:1.0995 dl:167-168 gd:1 +ttp: b96/782 bl:3.0349 bb:1.1862 rl:2.8254 rb:1.0996 dl:162-163 gd:1 +ttp: b90/782 bl:3.1213 bb:1.2312 rl:2.8260 rb:1.0999 dl:158-158 gd:1 +ttp: b82/782 bl:3.0831 bb:1.2411 rl:2.8265 rb:1.1001 dl:151-152 gd:1 +ttp: b75/782 bl:3.1678 bb:1.2437 rl:2.8271 rb:1.1004 dl:146-147 gd:1 +ttp: b69/782 bl:3.1908 bb:1.2659 rl:2.8277 rb:1.1007 dl:142-142 gd:1 +ttp: b62/782 bl:3.0822 bb:1.2468 rl:2.8281 rb:1.1009 dl:136-137 gd:1 +ttp: b53/782 bl:3.2351 bb:1.2754 rl:2.8287 rb:1.1012 dl:129-130 gd:1 +ttp: b47/782 bl:3.0245 bb:1.2087 rl:2.8290 rb:1.1013 dl:124-125 gd:1 +ttp: b40/782 bl:3.0808 bb:1.2392 rl:2.8294 rb:1.1015 dl:119-119 gd:1 +ttp: b30/782 bl:3.2204 bb:1.2900 rl:2.8299 rb:1.1018 dl:110-111 gd:1 +ttp: b23/782 bl:3.2222 bb:1.2843 rl:2.8304 rb:1.1020 dl:104-105 gd:1 +ttp: b15/782 bl:3.3198 bb:1.2700 rl:2.8309 rb:1.1022 dl:95-97 gd:1 +ttp: b11/782 bl:3.3294 bb:1.3011 rl:2.8315 rb:1.1024 dl:90-92 gd:1 +ttp: b3/782 bl:3.4104 bb:1.2934 rl:2.8320 rb:1.1026 dl:75-78 gd:1 +quantized_ttt_phased val_loss:2.85963649 val_bpb:1.10705440 eval_time:412607ms +total_eval_time:412.6s diff --git a/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_smoke.log b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_smoke.log new file mode 100644 index 0000000000..8f99436e6f --- /dev/null +++ b/records/track_non_record_16mb/2026-04-28_NewtonMuon_DocPacking_NegativeResult/train_nm_smoke.log @@ -0,0 +1,788 @@ +W0428 06:20:47.610000 286141 torch/distributed/run.py:803] +W0428 06:20:47.610000 286141 torch/distributed/run.py:803] ***************************************** +W0428 06:20:47.610000 286141 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0428 06:20:47.610000 286141 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + beta1: 0.9 + beta2: 0.95 + compressor: brotli + data_dir: ./data/ + datasets_dir: ./data/datasets/fineweb10B_sp8192 + distributed: True + ema_decay: 0.9965 + embed_bits: 7 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.35 + eval_seq_len: 2048 + eval_stride: 64 + gate_attn_out: True + gate_attn_width: 24 + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 4.0 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 100 + ln_scale: True + local_rank: 0 + logfile: logs/pr1874_nm_smoke.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 3 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 180.0 + min_lr: 0.0 + mlp_clip_sigmas: 12.0 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + newton_muon_all_reduce_k: True + newton_muon_beta: 0.95 + newton_muon_capture_every: 4 + newton_muon_enabled: True + newton_muon_gamma: 0.2 + newton_muon_k_refresh: 32 + newton_muon_warmup: 100 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 8 + phased_ttt_enabled: True + phased_ttt_num_phases: 3 + phased_ttt_prefix_docs: 2000 + polar_express_ns: True + qk_gain_init: 5.25 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: pr1874_nm_smoke + scalar_lr: 0.02 + seed: 42 + skip_gates_enabled: True + sliding_window_enabled: False + smear_gate_enabled: True + smear_gate_width: 12 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./data/tokenizers/fineweb_8192_bpe.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: True + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 128 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192/fineweb_val_*.bin + val_loss_every: 0 + vocab_size: 8192 + warmdown_frac: 0.75 + warmup_steps: 5 + world_size: 8 + xsa_last_n: 11 +train_shards: 80 +val_tokens: 40540160 +newton_muon: enabled (compile fullgraph=False, capture_every=4) +newton_muon: installed 12 forward-pre hooks (capture_every=4) +model_params:35946727 +gptq:reserving 4s, effective=176000ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +[rank3]:W0428 06:26:21.317000 286212 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank3]:W0428 06:26:21.317000 286212 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank3]:W0428 06:26:21.317000 286212 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank3]:W0428 06:26:21.317000 286212 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank3]:W0428 06:26:21.317000 286212 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank1]:W0428 06:26:21.854000 286210 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank1]:W0428 06:26:21.854000 286210 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank1]:W0428 06:26:21.854000 286210 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank1]:W0428 06:26:21.854000 286210 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank1]:W0428 06:26:21.854000 286210 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank2]:W0428 06:26:24.619000 286211 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank2]:W0428 06:26:24.619000 286211 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank2]:W0428 06:26:24.619000 286211 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank2]:W0428 06:26:24.619000 286211 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank2]:W0428 06:26:24.619000 286211 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank4]:W0428 06:26:25.049000 286213 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank4]:W0428 06:26:25.049000 286213 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank4]:W0428 06:26:25.049000 286213 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank4]:W0428 06:26:25.049000 286213 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank4]:W0428 06:26:25.049000 286213 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank5]:W0428 06:26:25.566000 286214 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank5]:W0428 06:26:25.566000 286214 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank5]:W0428 06:26:25.566000 286214 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank5]:W0428 06:26:25.566000 286214 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank5]:W0428 06:26:25.566000 286214 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank0]:W0428 06:26:28.461000 286209 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank0]:W0428 06:26:28.461000 286209 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank0]:W0428 06:26:28.461000 286209 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank0]:W0428 06:26:28.461000 286209 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank0]:W0428 06:26:28.461000 286209 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank7]:W0428 06:26:29.783000 286216 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank7]:W0428 06:26:29.783000 286216 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank7]:W0428 06:26:29.783000 286216 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank7]:W0428 06:26:29.783000 286216 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank7]:W0428 06:26:29.783000 286216 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +[rank6]:W0428 06:26:35.771000 286215 torch/_dynamo/convert_frame.py:1358] [0/16] torch._dynamo hit config.recompile_limit (16) +[rank6]:W0428 06:26:35.771000 286215 torch/_dynamo/convert_frame.py:1358] [0/16] function: 'forward' (/workspace/parameter-golf/Opus/code/pr1874_nm.py:1009) +[rank6]:W0428 06:26:35.771000 286215 torch/_dynamo/convert_frame.py:1358] [0/16] last reason: 0/15: self._modules['blocks']._modules['0']._modules['attn']._modules['attn_gate_proj']._nm_K_count == 1474560 # module._nm_K_count += n # orkspace/parameter-golf/Opus/code/pr1874_nm.py:1337 in hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.) +[rank6]:W0428 06:26:35.771000 286215 torch/_dynamo/convert_frame.py:1358] [0/16] To log all recompilation reasons, use TORCH_LOGS="recompiles". +[rank6]:W0428 06:26:35.771000 286215 torch/_dynamo/convert_frame.py:1358] [0/16] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html +warmup_step: 1/5 +warmup_step: 2/5 +warmup_step: 3/5 +warmup_step: 4/5 +warmup_step: 5/5 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/5 +loop_warmup_step: 2/5 +loop_warmup_step: 3/5 +loop_warmup_step: 4/5 +loop_warmup_step: 5/5 +1/100 train_loss: 9.0080 train_time: 0.0m tok/s: 3695530 +2/100 train_loss: 12.1832 train_time: 0.0m tok/s: 3599625 +3/100 train_loss: 11.2459 train_time: 0.0m tok/s: 3520893 +4/100 train_loss: 9.6099 train_time: 0.0m tok/s: 3473238 +5/100 train_loss: 8.1912 train_time: 0.0m tok/s: 3450926 +100/100 val_loss: 4.1581 val_bpb: 1.6097 +peak memory allocated: 54113 MiB reserved: 60400 MiB +ema:applying EMA weights +newton_muon: hooks uninstalled before EMA/quant/eval +diagnostic pre-quantization post-ema val_loss:7.35094726 val_bpb:2.84568905 eval_time:4921ms +Serialized model: 135422397 bytes +Code size (uncompressed): 134706 bytes +Code size (compressed): 33710 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians in 2.4s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + gptq (int7)+lqer_asym: tok_emb.weight + passthrough (float16): blocks.attn.attn_gate_proj.weight, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15943987 bytes +Total submission size quantized+brotli: 15977697 bytes +diagnostic quantized val_loss:7.35229814 val_bpb:2.84621200 eval_time:12836ms +ttt_lora:warming up compile (random tokens, no val data) +ttt_lora:compile warmup done (107.7s) + +beginning TTT eval timer +ttt_phased: total_docs:50000 prefix_docs:2000 suffix_docs:48000 num_phases:3 boundaries:[666, 1333, 2000] +ttp: b775/782 bl:5.7064 bb:2.2590 rl:5.7064 rb:2.2590 dl:5853-6355 gd:0 +ttp: b774/782 bl:5.5512 bb:2.1923 rl:5.6315 rb:2.2268 dl:5552-5852 gd:0 +ttp: b769/782 bl:5.6107 bb:2.2204 rl:5.6259 rb:2.2251 dl:4307-4479 gd:0 +ttpp: phase:1/3 pd:1104 gd:666 t:196.6s +tttg: c1/95 lr:0.001000 t:0.3s +tttg: c2/95 lr:0.001000 t:0.4s +tttg: c3/95 lr:0.000999 t:0.4s +tttg: c4/95 lr:0.000997 t:0.5s +tttg: c5/95 lr:0.000996 t:0.6s +tttg: c6/95 lr:0.000993 t:0.7s +tttg: c7/95 lr:0.000990 t:0.7s +tttg: c8/95 lr:0.000986 t:0.8s +tttg: c9/95 lr:0.000982 t:0.9s +tttg: c10/95 lr:0.000978 t:1.0s +tttg: c11/95 lr:0.000972 t:1.1s +tttg: c12/95 lr:0.000967 t:1.1s +tttg: c13/95 lr:0.000960 t:1.2s +tttg: c14/95 lr:0.000954 t:1.3s +tttg: c15/95 lr:0.000946 t:1.4s +tttg: c16/95 lr:0.000938 t:1.4s +tttg: c17/95 lr:0.000930 t:1.5s +tttg: c18/95 lr:0.000921 t:1.6s +tttg: c19/95 lr:0.000912 t:1.7s +tttg: c20/95 lr:0.000903 t:1.8s +tttg: c21/95 lr:0.000892 t:1.8s +tttg: c22/95 lr:0.000882 t:1.9s +tttg: c23/95 lr:0.000871 t:2.0s +tttg: c24/95 lr:0.000859 t:2.1s +tttg: c25/95 lr:0.000848 t:2.1s +tttg: c26/95 lr:0.000835 t:2.2s +tttg: c27/95 lr:0.000823 t:2.3s +tttg: c28/95 lr:0.000810 t:2.4s +tttg: c29/95 lr:0.000797 t:2.5s +tttg: c30/95 lr:0.000783 t:2.5s +tttg: c31/95 lr:0.000769 t:2.6s +tttg: c32/95 lr:0.000755 t:2.7s +tttg: c33/95 lr:0.000740 t:2.8s +tttg: c34/95 lr:0.000726 t:2.8s +tttg: c35/95 lr:0.000710 t:2.9s +tttg: c36/95 lr:0.000695 t:3.0s +tttg: c37/95 lr:0.000680 t:3.1s +tttg: c38/95 lr:0.000664 t:3.1s +tttg: c39/95 lr:0.000648 t:3.2s +tttg: c40/95 lr:0.000632 t:3.3s +tttg: c41/95 lr:0.000616 t:3.4s +tttg: c42/95 lr:0.000600 t:3.4s +tttg: c43/95 lr:0.000583 t:3.5s +tttg: c44/95 lr:0.000567 t:3.6s +tttg: c45/95 lr:0.000550 t:3.7s +tttg: c46/95 lr:0.000533 t:3.7s +tttg: c47/95 lr:0.000517 t:3.8s +tttg: c48/95 lr:0.000500 t:3.9s +tttg: c49/95 lr:0.000483 t:4.0s +tttg: c50/95 lr:0.000467 t:4.1s +tttg: c51/95 lr:0.000450 t:4.1s +tttg: c52/95 lr:0.000433 t:4.2s +tttg: c53/95 lr:0.000417 t:4.3s +tttg: c54/95 lr:0.000400 t:4.4s +tttg: c55/95 lr:0.000384 t:4.4s +tttg: c56/95 lr:0.000368 t:4.5s +tttg: c57/95 lr:0.000352 t:4.6s +tttg: c58/95 lr:0.000336 t:4.7s +tttg: c59/95 lr:0.000320 t:4.8s +tttg: c60/95 lr:0.000305 t:4.8s +tttg: c61/95 lr:0.000290 t:4.9s +tttg: c62/95 lr:0.000274 t:5.0s +tttg: c63/95 lr:0.000260 t:5.1s +tttg: c64/95 lr:0.000245 t:5.1s +tttg: c65/95 lr:0.000231 t:5.2s +tttg: c66/95 lr:0.000217 t:5.3s +tttg: c67/95 lr:0.000203 t:5.4s +tttg: c68/95 lr:0.000190 t:5.5s +tttg: c69/95 lr:0.000177 t:5.5s +tttg: c70/95 lr:0.000165 t:5.6s +tttg: c71/95 lr:0.000152 t:5.7s +tttg: c72/95 lr:0.000141 t:5.8s +tttg: c73/95 lr:0.000129 t:5.8s +tttg: c74/95 lr:0.000118 t:5.9s +tttg: c75/95 lr:0.000108 t:6.0s +tttg: c76/95 lr:0.000097 t:6.1s +tttg: c77/95 lr:0.000088 t:6.1s +tttg: c78/95 lr:0.000079 t:6.2s +tttg: c79/95 lr:0.000070 t:6.3s +tttg: c80/95 lr:0.000062 t:6.4s +tttg: c81/95 lr:0.000054 t:6.4s +tttg: c82/95 lr:0.000046 t:6.5s +tttg: c83/95 lr:0.000040 t:6.6s +tttg: c84/95 lr:0.000033 t:6.7s +tttg: c85/95 lr:0.000028 t:6.7s +tttg: c86/95 lr:0.000022 t:6.8s +tttg: c87/95 lr:0.000018 t:6.9s +tttg: c88/95 lr:0.000014 t:7.0s +tttg: c89/95 lr:0.000010 t:7.0s +tttg: c90/95 lr:0.000007 t:7.1s +tttg: c91/95 lr:0.000004 t:7.2s +tttg: c92/95 lr:0.000003 t:7.3s +tttg: c93/95 lr:0.000001 t:7.4s +tttg: c94/95 lr:0.000000 t:7.4s +ttpr: phase:1/3 t:206.7s +ttp: b758/782 bl:5.8772 bb:2.2175 rl:5.6667 rb:2.2238 dl:3108-3187 gd:0 +ttp: b756/782 bl:5.7047 bb:2.2111 rl:5.6718 rb:2.2221 dl:2973-3032 gd:0 +ttpp: phase:2/3 pd:1808 gd:1333 t:275.7s +tttg: c1/158 lr:0.001000 t:0.1s +tttg: c2/158 lr:0.001000 t:0.2s +tttg: c3/158 lr:0.001000 t:0.2s +tttg: c4/158 lr:0.000999 t:0.3s +tttg: c5/158 lr:0.000998 t:0.4s +tttg: c6/158 lr:0.000997 t:0.5s +tttg: c7/158 lr:0.000996 t:0.5s +tttg: c8/158 lr:0.000995 t:0.6s +tttg: c9/158 lr:0.000994 t:0.7s +tttg: c10/158 lr:0.000992 t:0.8s +tttg: c11/158 lr:0.000990 t:0.8s +tttg: c12/158 lr:0.000988 t:0.9s +tttg: c13/158 lr:0.000986 t:1.0s +tttg: c14/158 lr:0.000983 t:1.1s +tttg: c15/158 lr:0.000981 t:1.1s +tttg: c16/158 lr:0.000978 t:1.2s +tttg: c17/158 lr:0.000975 t:1.3s +tttg: c18/158 lr:0.000971 t:1.4s +tttg: c19/158 lr:0.000968 t:1.4s +tttg: c20/158 lr:0.000964 t:1.5s +tttg: c21/158 lr:0.000960 t:1.6s +tttg: c22/158 lr:0.000957 t:1.7s +tttg: c23/158 lr:0.000952 t:1.7s +tttg: c24/158 lr:0.000948 t:1.8s +tttg: c25/158 lr:0.000943 t:1.9s +tttg: c26/158 lr:0.000939 t:2.0s +tttg: c27/158 lr:0.000934 t:2.0s +tttg: c28/158 lr:0.000929 t:2.1s +tttg: c29/158 lr:0.000924 t:2.2s +tttg: c30/158 lr:0.000918 t:2.3s +tttg: c31/158 lr:0.000913 t:2.4s +tttg: c32/158 lr:0.000907 t:2.4s +tttg: c33/158 lr:0.000901 t:2.5s +tttg: c34/158 lr:0.000895 t:2.6s +tttg: c35/158 lr:0.000889 t:2.6s +tttg: c36/158 lr:0.000882 t:2.7s +tttg: c37/158 lr:0.000876 t:2.8s +tttg: c38/158 lr:0.000869 t:2.9s +tttg: c39/158 lr:0.000862 t:2.9s +tttg: c40/158 lr:0.000855 t:3.0s +tttg: c41/158 lr:0.000848 t:3.1s +tttg: c42/158 lr:0.000841 t:3.2s +tttg: c43/158 lr:0.000834 t:3.3s +tttg: c44/158 lr:0.000826 t:3.3s +tttg: c45/158 lr:0.000818 t:3.4s +tttg: c46/158 lr:0.000811 t:3.5s +tttg: c47/158 lr:0.000803 t:3.6s +tttg: c48/158 lr:0.000795 t:3.6s +tttg: c49/158 lr:0.000787 t:3.7s +tttg: c50/158 lr:0.000778 t:3.8s +tttg: c51/158 lr:0.000770 t:3.9s +tttg: c52/158 lr:0.000761 t:3.9s +tttg: c53/158 lr:0.000753 t:4.0s +tttg: c54/158 lr:0.000744 t:4.1s +tttg: c55/158 lr:0.000735 t:4.2s +tttg: c56/158 lr:0.000727 t:4.2s +tttg: c57/158 lr:0.000718 t:4.3s +tttg: c58/158 lr:0.000709 t:4.4s +tttg: c59/158 lr:0.000699 t:4.5s +tttg: c60/158 lr:0.000690 t:4.5s +tttg: c61/158 lr:0.000681 t:4.6s +tttg: c62/158 lr:0.000672 t:4.7s +tttg: c63/158 lr:0.000662 t:4.8s +tttg: c64/158 lr:0.000653 t:4.8s +tttg: c65/158 lr:0.000643 t:4.9s +tttg: c66/158 lr:0.000633 t:5.0s +tttg: c67/158 lr:0.000624 t:5.1s +tttg: c68/158 lr:0.000614 t:5.2s +tttg: c69/158 lr:0.000604 t:5.2s +tttg: c70/158 lr:0.000594 t:5.3s +tttg: c71/158 lr:0.000585 t:5.4s +tttg: c72/158 lr:0.000575 t:5.5s +tttg: c73/158 lr:0.000565 t:5.5s +tttg: c74/158 lr:0.000555 t:5.6s +tttg: c75/158 lr:0.000545 t:5.7s +tttg: c76/158 lr:0.000535 t:5.8s +tttg: c77/158 lr:0.000525 t:5.8s +tttg: c78/158 lr:0.000515 t:5.9s +tttg: c79/158 lr:0.000505 t:6.0s +tttg: c80/158 lr:0.000495 t:6.1s +tttg: c81/158 lr:0.000485 t:6.1s +tttg: c82/158 lr:0.000475 t:6.2s +tttg: c83/158 lr:0.000465 t:6.3s +tttg: c84/158 lr:0.000455 t:6.4s +tttg: c85/158 lr:0.000445 t:6.4s +tttg: c86/158 lr:0.000435 t:6.5s +tttg: c87/158 lr:0.000425 t:6.6s +tttg: c88/158 lr:0.000415 t:6.7s +tttg: c89/158 lr:0.000406 t:6.7s +tttg: c90/158 lr:0.000396 t:6.8s +tttg: c91/158 lr:0.000386 t:6.9s +tttg: c92/158 lr:0.000376 t:7.0s +tttg: c93/158 lr:0.000367 t:7.0s +tttg: c94/158 lr:0.000357 t:7.1s +tttg: c95/158 lr:0.000347 t:7.2s +tttg: c96/158 lr:0.000338 t:7.3s +tttg: c97/158 lr:0.000328 t:7.3s +tttg: c98/158 lr:0.000319 t:7.4s +tttg: c99/158 lr:0.000310 t:7.5s +tttg: c100/158 lr:0.000301 t:7.6s +tttg: c101/158 lr:0.000291 t:7.7s +tttg: c102/158 lr:0.000282 t:7.7s +tttg: c103/158 lr:0.000273 t:7.8s +tttg: c104/158 lr:0.000265 t:7.9s +tttg: c105/158 lr:0.000256 t:7.9s +tttg: c106/158 lr:0.000247 t:8.0s +tttg: c107/158 lr:0.000239 t:8.1s +tttg: c108/158 lr:0.000230 t:8.2s +tttg: c109/158 lr:0.000222 t:8.3s +tttg: c110/158 lr:0.000213 t:8.3s +tttg: c111/158 lr:0.000205 t:8.4s +tttg: c112/158 lr:0.000197 t:8.5s +tttg: c113/158 lr:0.000189 t:8.6s +tttg: c114/158 lr:0.000182 t:8.7s +tttg: c115/158 lr:0.000174 t:8.7s +tttg: c116/158 lr:0.000166 t:8.8s +tttg: c117/158 lr:0.000159 t:8.9s +tttg: c118/158 lr:0.000152 t:9.0s +tttg: c119/158 lr:0.000145 t:9.0s +tttg: c120/158 lr:0.000138 t:9.1s +tttg: c121/158 lr:0.000131 t:9.2s +tttg: c122/158 lr:0.000124 t:9.3s +tttg: c123/158 lr:0.000118 t:9.3s +tttg: c124/158 lr:0.000111 t:9.4s +tttg: c125/158 lr:0.000105 t:9.5s +tttg: c126/158 lr:0.000099 t:9.6s +tttg: c127/158 lr:0.000093 t:9.6s +tttg: c128/158 lr:0.000087 t:9.7s +tttg: c129/158 lr:0.000082 t:9.8s +tttg: c130/158 lr:0.000076 t:9.9s +tttg: c131/158 lr:0.000071 t:10.0s +tttg: c132/158 lr:0.000066 t:10.0s +tttg: c133/158 lr:0.000061 t:10.1s +tttg: c134/158 lr:0.000057 t:10.2s +tttg: c135/158 lr:0.000052 t:10.3s +tttg: c136/158 lr:0.000048 t:10.3s +tttg: c137/158 lr:0.000043 t:10.4s +tttg: c138/158 lr:0.000040 t:10.5s +tttg: c139/158 lr:0.000036 t:10.6s +tttg: c140/158 lr:0.000032 t:10.6s +tttg: c141/158 lr:0.000029 t:10.7s +tttg: c142/158 lr:0.000025 t:10.8s +tttg: c143/158 lr:0.000022 t:10.9s +tttg: c144/158 lr:0.000019 t:10.9s +tttg: c145/158 lr:0.000017 t:11.0s +tttg: c146/158 lr:0.000014 t:11.1s +tttg: c147/158 lr:0.000012 t:11.2s +tttg: c148/158 lr:0.000010 t:11.2s +tttg: c149/158 lr:0.000008 t:11.3s +tttg: c150/158 lr:0.000006 t:11.4s +tttg: c151/158 lr:0.000005 t:11.5s +tttg: c152/158 lr:0.000004 t:11.6s +tttg: c153/158 lr:0.000003 t:11.6s +tttg: c154/158 lr:0.000002 t:11.7s +tttg: c155/158 lr:0.000001 t:11.8s +tttg: c156/158 lr:0.000000 t:11.9s +tttg: c157/158 lr:0.000000 t:11.9s +ttpr: phase:2/3 t:290.3s +ttp: b752/782 bl:5.5547 bb:2.1311 rl:5.6589 rb:2.2119 dl:2740-2793 gd:0 +ttpp: phase:3/3 pd:2448 gd:2000 t:304.3s +tttg: c1/213 lr:0.001000 t:0.1s +tttg: c2/213 lr:0.001000 t:0.1s +tttg: c3/213 lr:0.001000 t:0.2s +tttg: c4/213 lr:0.001000 t:0.3s +tttg: c5/213 lr:0.000999 t:0.4s +tttg: c6/213 lr:0.000999 t:0.4s +tttg: c7/213 lr:0.000998 t:0.5s +tttg: c8/213 lr:0.000997 t:0.6s +tttg: c9/213 lr:0.000996 t:0.7s +tttg: c10/213 lr:0.000996 t:0.7s +tttg: c11/213 lr:0.000995 t:0.8s +tttg: c12/213 lr:0.000993 t:0.9s +tttg: c13/213 lr:0.000992 t:1.0s +tttg: c14/213 lr:0.000991 t:1.0s +tttg: c15/213 lr:0.000989 t:1.1s +tttg: c16/213 lr:0.000988 t:1.2s +tttg: c17/213 lr:0.000986 t:1.3s +tttg: c18/213 lr:0.000984 t:1.4s +tttg: c19/213 lr:0.000982 t:1.4s +tttg: c20/213 lr:0.000980 t:1.5s +tttg: c21/213 lr:0.000978 t:1.6s +tttg: c22/213 lr:0.000976 t:1.7s +tttg: c23/213 lr:0.000974 t:1.7s +tttg: c24/213 lr:0.000971 t:1.8s +tttg: c25/213 lr:0.000969 t:1.9s +tttg: c26/213 lr:0.000966 t:2.0s +tttg: c27/213 lr:0.000963 t:2.0s +tttg: c28/213 lr:0.000961 t:2.1s +tttg: c29/213 lr:0.000958 t:2.2s +tttg: c30/213 lr:0.000955 t:2.3s +tttg: c31/213 lr:0.000951 t:2.3s +tttg: c32/213 lr:0.000948 t:2.4s +tttg: c33/213 lr:0.000945 t:2.5s +tttg: c34/213 lr:0.000941 t:2.6s +tttg: c35/213 lr:0.000938 t:2.6s +tttg: c36/213 lr:0.000934 t:2.7s +tttg: c37/213 lr:0.000931 t:2.8s +tttg: c38/213 lr:0.000927 t:2.9s +tttg: c39/213 lr:0.000923 t:2.9s +tttg: c40/213 lr:0.000919 t:3.0s +tttg: c41/213 lr:0.000915 t:3.1s +tttg: c42/213 lr:0.000911 t:3.2s +tttg: c43/213 lr:0.000906 t:3.2s +tttg: c44/213 lr:0.000902 t:3.3s +tttg: c45/213 lr:0.000897 t:3.4s +tttg: c46/213 lr:0.000893 t:3.5s +tttg: c47/213 lr:0.000888 t:3.6s +tttg: c48/213 lr:0.000884 t:3.6s +tttg: c49/213 lr:0.000879 t:3.7s +tttg: c50/213 lr:0.000874 t:3.8s +tttg: c51/213 lr:0.000869 t:3.9s +tttg: c52/213 lr:0.000864 t:4.0s +tttg: c53/213 lr:0.000859 t:4.0s +tttg: c54/213 lr:0.000854 t:4.1s +tttg: c55/213 lr:0.000848 t:4.2s +tttg: c56/213 lr:0.000843 t:4.3s +tttg: c57/213 lr:0.000837 t:4.3s +tttg: c58/213 lr:0.000832 t:4.4s +tttg: c59/213 lr:0.000826 t:4.5s +tttg: c60/213 lr:0.000821 t:4.6s +tttg: c61/213 lr:0.000815 t:4.6s +tttg: c62/213 lr:0.000809 t:4.7s +tttg: c63/213 lr:0.000803 t:4.8s +tttg: c64/213 lr:0.000797 t:4.9s +tttg: c65/213 lr:0.000791 t:4.9s +tttg: c66/213 lr:0.000785 t:5.0s +tttg: c67/213 lr:0.000779 t:5.1s +tttg: c68/213 lr:0.000773 t:5.2s +tttg: c69/213 lr:0.000767 t:5.2s +tttg: c70/213 lr:0.000761 t:5.3s +tttg: c71/213 lr:0.000754 t:5.4s +tttg: c72/213 lr:0.000748 t:5.5s +tttg: c73/213 lr:0.000741 t:5.6s +tttg: c74/213 lr:0.000735 t:5.6s +tttg: c75/213 lr:0.000728 t:5.7s +tttg: c76/213 lr:0.000722 t:5.8s +tttg: c77/213 lr:0.000715 t:5.8s +tttg: c78/213 lr:0.000708 t:5.9s +tttg: c79/213 lr:0.000702 t:6.0s +tttg: c80/213 lr:0.000695 t:6.1s +tttg: c81/213 lr:0.000688 t:6.2s +tttg: c82/213 lr:0.000681 t:6.2s +tttg: c83/213 lr:0.000674 t:6.3s +tttg: c84/213 lr:0.000667 t:6.4s +tttg: c85/213 lr:0.000660 t:6.4s +tttg: c86/213 lr:0.000653 t:6.5s +tttg: c87/213 lr:0.000646 t:6.6s +tttg: c88/213 lr:0.000639 t:6.7s +tttg: c89/213 lr:0.000632 t:6.7s +tttg: c90/213 lr:0.000625 t:6.8s +tttg: c91/213 lr:0.000617 t:6.9s +tttg: c92/213 lr:0.000610 t:7.0s +tttg: c93/213 lr:0.000603 t:7.0s +tttg: c94/213 lr:0.000596 t:7.1s +tttg: c95/213 lr:0.000588 t:7.2s +tttg: c96/213 lr:0.000581 t:7.3s +tttg: c97/213 lr:0.000574 t:7.4s +tttg: c98/213 lr:0.000566 t:7.4s +tttg: c99/213 lr:0.000559 t:7.5s +tttg: c100/213 lr:0.000552 t:7.6s +tttg: c101/213 lr:0.000544 t:7.7s +tttg: c102/213 lr:0.000537 t:7.7s +tttg: c103/213 lr:0.000530 t:7.8s +tttg: c104/213 lr:0.000522 t:7.9s +tttg: c105/213 lr:0.000515 t:8.0s +tttg: c106/213 lr:0.000507 t:8.0s +tttg: c107/213 lr:0.000500 t:8.1s +tttg: c108/213 lr:0.000493 t:8.2s +tttg: c109/213 lr:0.000485 t:8.3s +tttg: c110/213 lr:0.000478 t:8.3s +tttg: c111/213 lr:0.000470 t:8.4s +tttg: c112/213 lr:0.000463 t:8.5s +tttg: c113/213 lr:0.000456 t:8.6s +tttg: c114/213 lr:0.000448 t:8.6s +tttg: c115/213 lr:0.000441 t:8.7s +tttg: c116/213 lr:0.000434 t:8.8s +tttg: c117/213 lr:0.000426 t:8.9s +tttg: c118/213 lr:0.000419 t:8.9s +tttg: c119/213 lr:0.000412 t:9.0s +tttg: c120/213 lr:0.000404 t:9.1s +tttg: c121/213 lr:0.000397 t:9.2s +tttg: c122/213 lr:0.000390 t:9.3s +tttg: c123/213 lr:0.000383 t:9.3s +tttg: c124/213 lr:0.000375 t:9.4s +tttg: c125/213 lr:0.000368 t:9.5s +tttg: c126/213 lr:0.000361 t:9.6s +tttg: c127/213 lr:0.000354 t:9.6s +tttg: c128/213 lr:0.000347 t:9.7s +tttg: c129/213 lr:0.000340 t:9.8s +tttg: c130/213 lr:0.000333 t:9.9s +tttg: c131/213 lr:0.000326 t:9.9s +tttg: c132/213 lr:0.000319 t:10.0s +tttg: c133/213 lr:0.000312 t:10.1s +tttg: c134/213 lr:0.000305 t:10.2s +tttg: c135/213 lr:0.000298 t:10.2s +tttg: c136/213 lr:0.000292 t:10.3s +tttg: c137/213 lr:0.000285 t:10.4s +tttg: c138/213 lr:0.000278 t:10.5s +tttg: c139/213 lr:0.000272 t:10.5s +tttg: c140/213 lr:0.000265 t:10.6s +tttg: c141/213 lr:0.000259 t:10.7s +tttg: c142/213 lr:0.000252 t:10.8s +tttg: c143/213 lr:0.000246 t:10.8s +tttg: c144/213 lr:0.000239 t:10.9s +tttg: c145/213 lr:0.000233 t:11.0s +tttg: c146/213 lr:0.000227 t:11.1s +tttg: c147/213 lr:0.000221 t:11.1s +tttg: c148/213 lr:0.000215 t:11.2s +tttg: c149/213 lr:0.000209 t:11.3s +tttg: c150/213 lr:0.000203 t:11.4s +tttg: c151/213 lr:0.000197 t:11.5s +tttg: c152/213 lr:0.000191 t:11.5s +tttg: c153/213 lr:0.000185 t:11.6s +tttg: c154/213 lr:0.000179 t:11.7s +tttg: c155/213 lr:0.000174 t:11.8s +tttg: c156/213 lr:0.000168 t:11.8s +tttg: c157/213 lr:0.000163 t:11.9s +tttg: c158/213 lr:0.000157 t:12.0s +tttg: c159/213 lr:0.000152 t:12.0s +tttg: c160/213 lr:0.000146 t:12.1s +tttg: c161/213 lr:0.000141 t:12.2s +tttg: c162/213 lr:0.000136 t:12.3s +tttg: c163/213 lr:0.000131 t:12.4s +tttg: c164/213 lr:0.000126 t:12.4s +tttg: c165/213 lr:0.000121 t:12.5s +tttg: c166/213 lr:0.000116 t:12.6s +tttg: c167/213 lr:0.000112 t:12.6s +tttg: c168/213 lr:0.000107 t:12.7s +tttg: c169/213 lr:0.000103 t:12.8s +tttg: c170/213 lr:0.000098 t:12.9s +tttg: c171/213 lr:0.000094 t:13.0s +tttg: c172/213 lr:0.000089 t:13.0s +tttg: c173/213 lr:0.000085 t:13.1s +tttg: c174/213 lr:0.000081 t:13.2s +tttg: c175/213 lr:0.000077 t:13.3s +tttg: c176/213 lr:0.000073 t:13.4s +tttg: c177/213 lr:0.000069 t:13.5s +tttg: c178/213 lr:0.000066 t:13.5s +tttg: c179/213 lr:0.000062 t:13.6s +tttg: c180/213 lr:0.000059 t:13.7s +tttg: c181/213 lr:0.000055 t:13.8s +tttg: c182/213 lr:0.000052 t:13.8s +tttg: c183/213 lr:0.000049 t:13.9s +tttg: c184/213 lr:0.000045 t:14.0s +tttg: c185/213 lr:0.000042 t:14.1s +tttg: c186/213 lr:0.000039 t:14.2s +tttg: c187/213 lr:0.000037 t:14.2s +tttg: c188/213 lr:0.000034 t:14.3s +tttg: c189/213 lr:0.000031 t:14.4s +tttg: c190/213 lr:0.000029 t:14.5s +tttg: c191/213 lr:0.000026 t:14.5s +tttg: c192/213 lr:0.000024 t:14.6s +tttg: c193/213 lr:0.000022 t:14.7s +tttg: c194/213 lr:0.000020 t:14.8s +tttg: c195/213 lr:0.000018 t:14.9s +tttg: c196/213 lr:0.000016 t:14.9s +tttg: c197/213 lr:0.000014 t:15.0s +tttg: c198/213 lr:0.000012 t:15.1s +tttg: c199/213 lr:0.000011 t:15.2s +tttg: c200/213 lr:0.000009 t:15.2s +tttg: c201/213 lr:0.000008 t:15.3s +tttg: c202/213 lr:0.000007 t:15.4s +tttg: c203/213 lr:0.000005 t:15.4s +tttg: c204/213 lr:0.000004 t:15.5s +tttg: c205/213 lr:0.000004 t:15.6s +tttg: c206/213 lr:0.000003 t:15.7s +tttg: c207/213 lr:0.000002 t:15.8s +tttg: c208/213 lr:0.000001 t:15.8s +tttg: c209/213 lr:0.000001 t:15.9s +tttg: c210/213 lr:0.000000 t:16.0s +tttg: c211/213 lr:0.000000 t:19.4s +tttg: c212/213 lr:0.000000 t:19.5s +ttpr: phase:3/3 t:326.5s +ttp: b741/782 bl:5.4600 bb:2.1494 rl:5.6422 rb:2.2066 dl:2286-2319 gd:1 +ttp: b729/782 bl:5.4251 bb:2.0674 rl:5.6275 rb:2.1970 dl:1978-1994 gd:1 +ttp: b722/782 bl:5.4195 bb:2.0721 rl:5.6152 rb:2.1895 dl:1846-1861 gd:1 +ttp: b719/782 bl:5.3911 bb:2.0673 rl:5.6030 rb:2.1827 dl:1793-1816 gd:1 +ttp: b705/782 bl:5.4509 bb:2.0996 rl:5.5959 rb:2.1788 dl:1606-1617 gd:1 +ttp: b701/782 bl:5.3849 bb:2.0479 rl:5.5868 rb:2.1730 dl:1562-1572 gd:1 +ttp: b692/782 bl:5.3814 bb:2.0418 rl:5.5788 rb:2.1677 dl:1477-1484 gd:1 +ttp: b687/782 bl:5.3280 bb:2.0575 rl:5.5696 rb:2.1637 dl:1432-1441 gd:1 +ttp: b679/782 bl:5.4678 bb:2.0831 rl:5.5661 rb:2.1609 dl:1368-1374 gd:1 +ttp: b670/782 bl:5.4871 bb:2.0516 rl:5.5636 rb:2.1573 dl:1308-1315 gd:1 +ttp: b658/782 bl:5.4002 bb:2.0670 rl:5.5590 rb:2.1547 dl:1234-1239 gd:1 +ttp: b654/782 bl:5.3922 bb:2.0469 rl:5.5544 rb:2.1517 dl:1209-1215 gd:1 +ttp: b642/782 bl:5.4458 bb:2.1186 rl:5.5516 rb:2.1509 dl:1144-1150 gd:1 +ttp: b639/782 bl:5.3871 bb:2.0406 rl:5.5477 rb:2.1481 dl:1129-1134 gd:1 +ttp: b629/782 bl:5.3586 bb:2.0530 rl:5.5434 rb:2.1460 dl:1082-1086 gd:1 +ttp: b621/782 bl:5.4119 bb:2.0727 rl:5.5405 rb:2.1444 dl:1046-1050 gd:1 +ttp: b615/782 bl:5.4481 bb:2.0456 rl:5.5386 rb:2.1423 dl:1020-1023 gd:1 +ttp: b606/782 bl:5.4630 bb:2.1019 rl:5.5372 rb:2.1415 dl:982-986 gd:1 +ttp: b598/782 bl:5.4311 bb:2.0687 rl:5.5352 rb:2.1401 dl:950-954 gd:1 +ttp: b592/782 bl:5.4385 bb:2.0475 rl:5.5335 rb:2.1384 dl:930-933 gd:1 +ttp: b584/782 bl:5.4188 bb:2.0359 rl:5.5316 rb:2.1367 dl:904-907 gd:1 +ttp: b576/782 bl:5.4233 bb:2.0425 rl:5.5298 rb:2.1351 dl:877-880 gd:1 +ttp: b567/782 bl:5.3725 bb:2.0693 rl:5.5274 rb:2.1341 dl:849-852 gd:1 +ttp: b559/782 bl:5.3852 bb:2.0467 rl:5.5253 rb:2.1328 dl:824-827 gd:1 +ttp: b549/782 bl:5.4397 bb:2.0929 rl:5.5241 rb:2.1322 dl:795-798 gd:1 +ttp: b543/782 bl:5.4517 bb:2.0466 rl:5.5231 rb:2.1310 dl:779-782 gd:1 +ttp: b534/782 bl:5.4203 bb:2.0618 rl:5.5218 rb:2.1301 dl:757-759 gd:1 +ttp: b526/782 bl:5.4091 bb:2.0656 rl:5.5204 rb:2.1293 dl:737-739 gd:1 +ttp: b518/782 bl:5.4364 bb:2.0939 rl:5.5194 rb:2.1289 dl:717-720 gd:1 +ttp: b513/782 bl:5.4494 bb:2.0170 rl:5.5186 rb:2.1275 dl:705-707 gd:1 +ttp: b505/782 bl:5.4460 bb:2.0808 rl:5.5177 rb:2.1270 dl:686-688 gd:1 +ttp: b497/782 bl:5.5102 bb:2.1010 rl:5.5177 rb:2.1267 dl:668-671 gd:1 +ttp: b490/782 bl:5.4221 bb:2.0719 rl:5.5167 rb:2.1261 dl:653-655 gd:1 +ttp: b483/782 bl:5.4348 bb:2.0784 rl:5.5158 rb:2.1256 dl:639-641 gd:1 +ttp: b476/782 bl:5.4621 bb:2.0861 rl:5.5153 rb:2.1253 dl:624-626 gd:1 +ttp: b466/782 bl:5.4404 bb:2.0683 rl:5.5146 rb:2.1247 dl:604-606 gd:1 +ttp: b459/782 bl:5.4416 bb:2.0649 rl:5.5139 rb:2.1242 dl:591-593 gd:1 +ttp: b452/782 bl:5.4339 bb:2.0961 rl:5.5132 rb:2.1239 dl:579-580 gd:1 +ttp: b446/782 bl:5.4630 bb:2.1086 rl:5.5128 rb:2.1238 dl:568-569 gd:1 +ttp: b439/782 bl:5.4351 bb:2.0599 rl:5.5122 rb:2.1232 dl:555-556 gd:1 +ttp: b431/782 bl:5.4674 bb:2.1105 rl:5.5118 rb:2.1231 dl:540-542 gd:1 +ttp: b423/782 bl:5.4687 bb:2.0540 rl:5.5115 rb:2.1226 dl:526-528 gd:1 +ttp: b419/782 bl:5.4929 bb:2.0414 rl:5.5113 rb:2.1220 dl:519-521 gd:1 +ttp: b411/782 bl:5.4925 bb:2.0940 rl:5.5112 rb:2.1217 dl:507-508 gd:1 +ttp: b402/782 bl:5.5087 bb:2.0755 rl:5.5112 rb:2.1214 dl:492-493 gd:1 +ttp: b394/782 bl:5.5512 bb:2.1409 rl:5.5114 rb:2.1215 dl:479-481 gd:1 +ttp: b387/782 bl:5.4841 bb:2.0759 rl:5.5113 rb:2.1212 dl:468-470 gd:1 +ttp: b379/782 bl:5.4840 bb:2.1000 rl:5.5111 rb:2.1211 dl:457-459 gd:1 +ttp: b371/782 bl:5.4954 bb:2.1005 rl:5.5110 rb:2.1210 dl:446-447 gd:1 +ttp: b363/782 bl:5.4886 bb:2.1887 rl:5.5108 rb:2.1214 dl:434-436 gd:1 +ttp: b352/782 bl:5.5225 bb:2.1953 rl:5.5109 rb:2.1218 dl:419-420 gd:1 +ttp: b344/782 bl:5.5714 bb:2.1356 rl:5.5113 rb:2.1219 dl:408-410 gd:1 +ttp: b337/782 bl:5.5919 bb:2.1291 rl:5.5117 rb:2.1219 dl:399-400 gd:1 +ttp: b329/782 bl:5.5800 bb:2.1765 rl:5.5121 rb:2.1222 dl:389-390 gd:1 +ttp: b320/782 bl:5.4901 bb:2.1417 rl:5.5119 rb:2.1223 dl:377-378 gd:1 +ttp: b312/782 bl:5.5025 bb:2.1480 rl:5.5119 rb:2.1224 dl:367-368 gd:1 +ttp: b304/782 bl:5.5810 bb:2.1736 rl:5.5122 rb:2.1226 dl:357-358 gd:1 +ttp: b296/782 bl:5.5922 bb:2.1627 rl:5.5126 rb:2.1228 dl:347-348 gd:1 +ttp: b288/782 bl:5.5445 bb:2.1769 rl:5.5127 rb:2.1231 dl:337-339 gd:1 +ttp: b277/782 bl:5.6055 bb:2.2077 rl:5.5131 rb:2.1234 dl:325-326 gd:1 +ttp: b269/782 bl:5.6290 bb:2.1697 rl:5.5136 rb:2.1236 dl:316-318 gd:1 +ttp: b261/782 bl:5.6475 bb:2.2090 rl:5.5142 rb:2.1240 dl:308-309 gd:1 +ttp: b253/782 bl:5.5531 bb:2.1807 rl:5.5143 rb:2.1242 dl:298-299 gd:1 +ttp: b245/782 bl:5.6241 bb:2.1596 rl:5.5147 rb:2.1243 dl:290-291 gd:1 +ttp: b237/782 bl:5.6871 bb:2.2359 rl:5.5154 rb:2.1247 dl:282-283 gd:1 +ttp: b230/782 bl:5.6162 bb:2.1491 rl:5.5157 rb:2.1248 dl:275-276 gd:1 +ttp: b222/782 bl:5.6294 bb:2.1870 rl:5.5161 rb:2.1250 dl:267-268 gd:1 +ttp: b217/782 bl:5.6010 bb:2.1939 rl:5.5164 rb:2.1252 dl:262-263 gd:1 +ttp: b209/782 bl:5.5978 bb:2.2161 rl:5.5167 rb:2.1255 dl:254-255 gd:1 +ttp: b201/782 bl:5.6540 bb:2.2038 rl:5.5171 rb:2.1258 dl:247-248 gd:1 +ttp: b193/782 bl:5.6301 bb:2.2684 rl:5.5175 rb:2.1262 dl:240-241 gd:1 +ttp: b187/782 bl:5.5915 bb:2.1560 rl:5.5177 rb:2.1263 dl:235-236 gd:1 +ttp: b179/782 bl:5.7422 bb:2.2430 rl:5.5183 rb:2.1266 dl:228-229 gd:1 +ttp: b171/782 bl:5.6732 bb:2.1820 rl:5.5188 rb:2.1268 dl:221-222 gd:1 +ttp: b163/782 bl:5.6659 bb:2.2239 rl:5.5192 rb:2.1270 dl:214-215 gd:1 +ttp: b155/782 bl:5.6943 bb:2.2380 rl:5.5196 rb:2.1273 dl:207-208 gd:1 +ttp: b147/782 bl:5.7202 bb:2.2631 rl:5.5201 rb:2.1276 dl:201-202 gd:1 +ttp: b138/782 bl:5.7238 bb:2.2783 rl:5.5206 rb:2.1280 dl:194-195 gd:1 +ttp: b128/782 bl:5.7126 bb:2.1932 rl:5.5211 rb:2.1282 dl:186-187 gd:1 +ttp: b121/782 bl:5.6618 bb:2.2431 rl:5.5214 rb:2.1284 dl:181-181 gd:1 +ttp: b112/782 bl:5.7121 bb:2.2094 rl:5.5218 rb:2.1286 dl:174-175 gd:1 +ttp: b107/782 bl:5.6872 bb:2.2325 rl:5.5221 rb:2.1288 dl:171-171 gd:1 +ttp: b97/782 bl:5.7504 bb:2.2464 rl:5.5226 rb:2.1290 dl:163-164 gd:1 +ttp: b89/782 bl:5.7293 bb:2.2849 rl:5.5230 rb:2.1293 dl:157-158 gd:1 +ttp: b82/782 bl:5.7797 bb:2.3265 rl:5.5235 rb:2.1297 dl:151-152 gd:1 +ttp: b71/782 bl:5.7049 bb:2.2258 rl:5.5238 rb:2.1298 dl:143-144 gd:1 +ttp: b66/782 bl:5.7252 bb:2.3451 rl:5.5241 rb:2.1302 dl:139-140 gd:1 +ttp: b55/782 bl:5.7344 bb:2.3030 rl:5.5245 rb:2.1304 dl:130-131 gd:1 +ttp: b47/782 bl:5.7837 bb:2.3114 rl:5.5249 rb:2.1307 dl:124-125 gd:1 +ttp: b40/782 bl:5.7500 bb:2.3128 rl:5.5252 rb:2.1310 dl:119-119 gd:1 +ttp: b31/782 bl:5.8385 bb:2.3137 rl:5.5256 rb:2.1312 dl:111-112 gd:1 +ttp: b23/782 bl:5.8357 bb:2.3260 rl:5.5260 rb:2.1314 dl:104-105 gd:1 +ttp: b15/782 bl:5.8566 bb:2.2405 rl:5.5264 rb:2.1316 dl:95-97 gd:1 +ttp: b7/782 bl:5.8085 bb:2.2287 rl:5.5267 rb:2.1317 dl:84-86 gd:1 +ttp: b2/782 bl:5.8275 bb:2.1614 rl:5.5269 rb:2.1317 dl:70-75 gd:1 +quantized_ttt_phased val_loss:5.47389905 val_bpb:2.11911690 eval_time:414807ms +total_eval_time:414.8s