diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/README.md b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/README.md new file mode 100644 index 0000000000..6fff94b600 --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/README.md @@ -0,0 +1,266 @@ +# Mercury in Retrograde* + +**This is a non-record submission for the Parameter Golf request for text diffusion.** Instead of taking a leading model and then sprinkling a little text diffusion on top for fun, we aimed to make text diffusion unmistakably present in the training objective and in the evaluation story, then to report, with as much honesty as possible, what happened when that idea was squeezed into a 16MB artifact and a 10-minute training budget. The result was a failure for `val_bpb` but I'm calling it a win for educational golfing. The model acquired a real diffusion-like interface. It learned to revise whole spans in parallel. It learned to infill. It learned a speed-quality knob. It also learned, with a kind of stubborn little diligence, how to be wrong at extremely high throughput. + +> *Mercury retrograde is an optical illusion in which the planet appears to move backward; it is superstitiously associated with communication breakdowns, travel delays, and technological glitches--the perfect metaphor for a model that seems to be going backward on the one metric the challenge actually uses to keep score. + +## Why Text Diffusion Was Worth Trying + +Autoregressive language models do one thing with almost tyrannical consistency: they predict the next token, then the next, then the next, each one contingent on the whole left context and inaccessible from the right. This is a fantastically strong way to model text distributionally, but it is also sequential by construction. Diffusion models tempt us with another picture. Instead of writing the sentence one token at a time, they begin from corruption and iteratively denoise, revising many positions at once, sometimes all of them. In images this has already redrawn the map. + +In text it has been harder, but not for lack of serious attempts: we took some inspiration from e.g. [Diffusion-LM](https://arxiv.org/abs/2205.14217) which made the case that diffusion objectives could be useful for controllable text generation. Some [later papers](https://arxiv.org/abs/2410.18514) [suggested](https://arxiv.org/abs/2602.15014) that text diffusion is not merely a curiosity, but something that may become more competitive with the right scaling laws, formulations, and systems. + +A diffusion language model should, in principle generate multiple tokens in parallel, revise its own mistakes, handle infill and other any-order generation tasks naturally, expose an explicit speed-vs-quality tradeoff through the number of denoising rounds, and perhaps, if the stars are properly aligned, do all this much faster than ordinary left-to-right decoding. + +That is also roughly how [Inception’s Mercury announcement](https://www.inceptionlabs.ai/blog/introducing-mercury) presents the idea: a coarse-to-fine model that modifies multiple tokens in parallel, can serve as a drop-in replacement for Transformer-based LLM infrastructure, and gets much of its practical force not only from modeling but from the surrounding systems stack. (That last clause matters a lot.) + +## Why It Probably Does Not Work Here + +The short version is that Parameter Golf is a peculiarly hostile habitat for text diffusion. + +This challenge rewards compression quality on FineWeb through exact `val_bpb`, under an extremely tight artifact budget and a brutal training wall clock. This environment is almost offensively well-suited to small autoregressive models. AR spends nearly all of its capacity on one job: estimating the next-token distribution as faithfully as possible. A compact diffusion model, by contrast, has to learn something more baroque. It must model text well enough to know what should go in a corrupted position, while also learning how to denoise under a corruption process, while also staying stable under iterative refinement, while also paying for the fact that its natural interface is no longer the exact objective by which the challenge ranks submissions. + +This becomes worse, not better, when the model is tiny. A large diffusion model may have enough slack capacity to learn both language statistics and denoising dynamics and maybe even some graceful error correction. A small model, trained briefly, tends instead to discover a grimly economical compromise: guess common tokens, repeat punctuation, repair locally if possible, and otherwise retreat into the high-frequency regions of the distribution. Which is, to be fair, a distributional strategy of a sort. It just is not the sort you want. + +There is also the systems issue. Inception's public framing is not merely "diffusion is better." It is "diffusion plus an optimized inference engine plus batching plus kernels plus scale is fast enough to matter." We did not reproduce that stack here. We reproduced, on purpose, the modeling gesture: parallel coarse-to-fine text denoising inside the challenge’s compact Transformer setup. That makes this a useful scientific negative result rather than a failed product launch. It isolates the part we could test. + +## What We Actually Tried + +### Phase 1: naive hybrid text diffusion + +We began with a modest premise: keep the ordinary causal LM machinery, then add text-diffusion-style corruption objectives and see whether some measured amount of denoising helps without wrecking BPB. + +**Early 180-second ladder:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `td_control` | plain causal LM | 2.0027 | +| `td_span15` | 15% span corruption | 2.1725 | +| `td_span30` | 30% span corruption | 2.1831 | +| `td_prog15` | progressive corruption to 15% | 2.1752 | +| `td_prog30` | progressive corruption to 30% | 2.1893 | + +The immediate lesson was not subtle. Even gentle diffusion-style corruption moved BPB in the wrong direction. That did not kill the project, but it did force a conceptual fork. If we optimized only for BPB, the best "text diffusion" model would simply be the one with the least text diffusion in it, which is not what we wanted to chase here. + +### Phase 2: late auxiliary losses and corruption sweeps + +We then tried to make diffusion less destructive by deferring or demoting it. The denoising objective was pushed later in training, or weakened, or made more selective. + +**Late auxiliary ladder:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `td_control` | plain causal LM | 1.9997 | +| `td_late02` | 2% late diffusion weight | 2.0437 | +| `td_late05` | 5% late diffusion weight | 2.0485 | +| `td_late05p` | 5% late diffusion, progressive | 2.0489 | +| `td_late10` | 10% late diffusion weight | 2.0489 | + +**Corruption sweep:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `td_control` | plain causal LM | 2.0035 | +| `td_cons05` | consistency-style late objective | 2.0477 | +| `td_hyb05` | 5% hybrid corruption | 2.0515 | +| `td_hyb05m` | 5% hybrid, mask-heavier | 2.0503 | +| `td_unif05` | 5% uniform corruption | 2.0601 | +| `td_unif10` | 10% uniform corruption | 2.0690 | + +These runs were important precisely because they sort of worked, in the narrow sense that they did not completely explode. But that turned out to be the problem. They were not diffusion-native enough to be interesting. They preserved more of the AR model because they were, in spirit, still AR models with a denoising side hustle. + +### Phase 3: make diffusion explicit, then see what survives + +At that point the project changed from "can we smuggle in a little diffusion without hurting BPB" to "what is the most informative compact text-diffusion submission we can make, even if the BPB is worse." + +That led to the Mercury-style branch: a direct denoising mode, mixed continuation/infill masking, self-conditioning, progressive hybrid corruption, and parallel refinement metrics logged on purpose. If the model was going to lose on the main challenge metric, it should at least lose in a way that teaches something specific. + +**Initial Mercury screens, 180 seconds on 1xH100:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `mercury_mask25` | pure masked denoising | 12.9566 | +| `mercury_hybrid25` | hybrid corruption, 25% | 3.5492 | +| `mercury_uniform50` | uniform corruption, 50% | 3.5424 | +| `mercury_hybrid50p` | progressive hybrid to 50% | 3.5412 | +| `mercury_auxlate` | AR-heavy late auxiliary fallback | 2.0131 | + +This is where the shape of the problem finally became clear. The AR-heavy fallback was numerically healthier but aesthetically boring. The diffusion-native Mercury variants were much worse on BPB, but at least they behaved like diffusion models rather than as causal models wearing a fake mustache. + +We then tried to improve the Mercury path without betraying it: + +- align training masks with continuation and infill tasks, +- add self-conditioning, +- keep a small clean-language prior, +- reduce the corruption ceiling, +- bias somewhat more toward continuation, +- and, because one always hopes the obvious next thing will be the thing, try explicit 2-step unrolled training. + +Those produced the following sequence. + +**Self-conditioning and task-alignment screens, 180 seconds:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `mercury_uniform50` | mixed tasks, no strongest self-conditioning | 3.5056 | +| `mercury_hybrid50p` | mixed tasks | 3.4987 | +| `mercury_uniform50_suffixsc` | continuation-only, stronger self-conditioning | 3.5043 | +| `mercury_hybrid50p_mixsc` | mixed tasks, stronger self-conditioning | 3.4980 | +| `mercury_auxlate` | AR-heavy fallback | 2.0150 | + +**Longer 600-second Mercury screen:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `mercury_hybrid50p_mixsc` | progressive hybrid 25% to 50% | 2.2261 | +| `mercury_hybrid37p5_mixsc` | progressive hybrid 25% to 37.5% | 2.2168 | +| `mercury_hybrid50p_cont90` | stronger continuation bias | 2.2199 | + +**Fine local search around the best recipe, 180 seconds:** +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `mercury_hybrid35_mixsc` | progressive hybrid 25% to 35%, mixed tasks | 3.4973 | +| `mercury_hybrid37p5_cont85` | 37.5% ceiling, continuation 0.85 | 3.4980 | +| `mercury_hybrid3125_mixsc` | 31.25% ceiling, mixed tasks | 3.4982 | +| `mercury_hybrid35_cont85` | 35% ceiling, continuation 0.85 | 3.4983 | +| `mercury_hybrid37p5_mixsc` | 37.5% ceiling, mixed tasks | 3.4984 | + +And the explicit **two-step formulation**, which is worth recording because it failed so cleanly: + +| Variant | val_bpb | +| --- | ---: | +| `mercury_hybrid37p5_2step` | 6.8082 | +| `mercury_hybrid37p5_2step_cont90` | 6.7054 | +| retry `mercury_hybrid37p5_2step` | 7.0219 | +| retry `mercury_hybrid37p5_2step_cont90` | 6.9992 | + +This was not merely slow. It was unstable. The runs reached only about 25 steps in 180 seconds and produced catastrophically bad BPB almost immediately. In other words, the obvious "make it more diffusion-like by literally unrolling more denoising" move was, in this setting, precisely the wrong move. + +For completeness, we also explored anchored/block diffusion. + +**Anchored/block ladder, 180 seconds:** + +| Variant | Description | val_bpb | +| --- | --- | ---: | +| `ab_control` | plain causal LM | 1.9970 | +| `ab_anchor32late` | late-onset anchored block | 2.3869 | +| `ab_anchor32` | anchored block | 12.2302 | +| `ab_anchor64` | larger anchored block | 12.7226 | +| `ab_block32` | raw block diffusion | 12.9688 | + +Anchors helped relative to naked block diffusion. Delaying the objective helped much more. None of it was healthy enough to become the submission. + +## How We Landed On The Final Recipe + +The final submission recipe is `mercury_hybrid35_mixsc`, and it is here because it satisfied the one criterion that became more important than vanity: it made diffusion visible without making the whole model nonsensical. + +In concrete terms, it keeps: + +- direct Mercury-style denoising as the main auxiliary interface, +- progressive hybrid corruption from 25% to 35%, +- mixed continuation and infill tasks, +- 75% continuation bias, +- self-conditioning with 75% commit fraction, +- and a small clean-language prior to keep the denoiser from drifting completely into local noise repair. + +We did not choose it because it had the absolute best number from every exploratory run. We chose it because it was the best compromise between two competing obligations: + +- it had to be diffusion-native enough that calling it "text diffusion" was not a euphemism, +- and it had to preserve enough language modeling competence that the model remained legibly a language model rather than an expensive punctuation fountain. + +This is, in a way, the entire retrograde story. The more diffusion we added in the naive sense, the worse BPB became. The more we retreated back toward AR, the less interesting the submission became. The final recipe is the point on that curve where the model is still recognizably about diffusion, yet not so broken that the whole experiment collapses into pure parody. + +## Final 8xH100 Result + +**Final 8xH100 SXM runs used seeds `1337`, `42`, and `2026`.** +*The submitted `train_gpt.py` defaults to this recipe* +| Seed | Final val_loss | Final val_bpb | Artifact bytes | Stop step | +| --- | ---: | ---: | ---: | ---: | +| 1337 | 2.45349105 | 1.45309560 | 15,677,283 | 4628 | +| 42 | 2.46577237 | 1.46036929 | 15,531,183 | 4912 | +| 2026 | 2.45803749 | 1.45578825 | 15,500,938 | 4926 | +| Mean | 2.45910030 | 1.45641771 | - | - | +| Std | 0.00620926 | 0.00367747 | - | - | + +This is much worse than the official naive AR baseline. That sentence should not be hidden in a footnote or apologetically coughed into a sleeve. It is the primary result. In this challenge setting, compact text diffusion simply does not beat compact autoregression on `val_bpb`. + +All three artifacts fit under the decimal `16,000,000` byte cap. The largest logged total was `15,677,283` bytes. + +## What The Model Is Good At, If "Good" Is Used Carefully + +Our experiment exposes diffusion-native behavior clearly enough to inspect. + +**Three-seed mean `parallel_eval` results from the 8x logs:** + +| Task | Refinement steps | Token accuracy | Tokens/sec | +| --- | ---: | ---: | ---: | +| continuation | 1 | 0.0348 | 25,123.81 | +| continuation | 2 | 0.0345 | 36,784.34 | +| continuation | 4 | 0.0348 | 25,990.89 | +| continuation | 8 | 0.0355 | 13,005.42 | +| infill | 1 | 0.0377 | 104,971.51 | +| infill | 2 | 0.0384 | 52,357.51 | +| infill | 4 | 0.0410 | 26,075.30 | +| infill | 8 | 0.0404 | 12,922.12 | + +**These accuracies are awful.** They are also informative. They tell us that the model did learn the interface. It can denoise in parallel. It can infill. It can trade additional refinement steps for different speed and slightly different accuracy. What it did not learn was how to make that interface semantically robust under this budget. But it did reveal where some of the tradeoff was happening. + + +## Matched AR vs Mercury Decode Benchmark + +To make the speed-quality tradeoff legible, I also ran a matched 1xH100 decode benchmark using the actual seed `2026` 8x checkpoint and the official naive baseline checkpoint. + +Setup: + +- 32 validation examples +- 128-token prefix +- 64-token continuation target +- 64-token infill span, with a visible 64-token suffix +- same tokenizer, same validation source, same hardware class for the matched comparison + +Highlights from `decode_benchmark.md`: + +- AR continuation throughput: `1518.79` tokens/sec +- Mercury continuation, 1 step: `0.0400` token accuracy at `33315.36` tokens/sec, `21.94x` AR throughput +- Mercury continuation, 2 steps: `0.0400` token accuracy at `52423.93` tokens/sec, `34.52x` AR throughput +- Mercury infill, 1 step: `0.0400` token accuracy at `147729.24` tokens/sec, `97.27x` AR continuation throughput + +The model is able to be breathtakingly fast, spectacularly parallel, but very, very incorrect. The diffusion-native interface is real, but (at this scale) the diffusion-native model is not very good at all. + +> The raw examples in `decode_benchmark.md` are included because aggregate metrics flatter degeneration. In several samples the model collapses into `the`, punctuation, or repeated fragments. The token accuracy is therefore not evidence of meaningful long-span understanding in the ordinary human sense. It is evidence that the model can sometimes land near the high-frequency skeleton of the target while moving very quickly. + +## Reproduction + +The submitted `train_gpt.py` defaults to the final `mercury_hybrid35_mixsc` recipe. On the official RunPod image with cached SP1024 FineWeb data: + +```bash +DATA_PATH=./data/datasets/fineweb10B_sp1024 \ +TOKENIZER_PATH=./data/tokenizers/fineweb_1024_bpe.model \ +VOCAB_SIZE=1024 \ +SEED=1337 \ +torchrun --standalone --nproc_per_node=8 train_gpt.py +``` + +Repeat with `SEED=42` and `SEED=2026` to reproduce the three-seed set. Included logs: + +- `train_seed1337.log` +- `train_seed42.log` +- `train_seed2026.log` + +The script writes: + +- `final_model.pt` +- `final_model.int8.ptz` +- final `val_loss` and `val_bpb` +- post-roundtrip exact BPB +- continuation and infill `parallel_eval` metrics + +## Compliance + +- This is a non-record submission under `records/track_non_record_16mb`. +- `train_gpt.py` is self-contained and runnable from this folder. +- No network calls or external side information are used during training or validation. +- Validation uses the full FineWeb validation split through the standard Parameter Golf BPB path. +- The compressed artifact is `int8+zlib`, and all logged artifacts are under `16,000,000` bytes. +- Training fits in the 10-minute 8xH100 SXM budget used for the challenge. +- Evaluation fits in the separate evaluation budget. diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.json b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.json new file mode 100644 index 0000000000..30f9d5f2cd --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.json @@ -0,0 +1,147 @@ +{ + "metadata": { + "baseline_script": "/workspace/parameter-golf/records/track_10min_16mb/2026-03-17_NaiveBaseline/train_gpt.py", + "baseline_checkpoint": "/workspace/parameter-golf/records/track_non_record_16mb/2026-04-09_MercuryStyleCompactTextDiffusion/benchmarks/real_8x_seed2026_vs_baseline_20260410_140157/ar_baseline/final_model.pt", + "mercury_script": "/workspace/parameter-golf/records/track_non_record_16mb/2026-04-09_MercuryStyleCompactTextDiffusion/train_mercury_compact_diffusion.py", + "mercury_checkpoint": "/workspace/parameter-golf/records/track_non_record_16mb/2026-04-09_MercuryStyleCompactTextDiffusion/final_model.pt", + "data_path": "/tmp/parameter-golf-data/datasets/fineweb10B_sp1024", + "tokenizer_path": "/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model", + "device": "cuda", + "num_examples": 32, + "prefix_len": 128, + "continuation_len": 64, + "infill_span_len": 64, + "infill_suffix_len": 64, + "benchmark_steps": [ + 1, + 2, + 4, + 8 + ], + "showcase_indices": [ + 192664, + 34663, + 165861, + 152617 + ] + }, + "ar_continuation": { + "token_acc": 0.021484375, + "exact_seq_acc": 0.0, + "tok_per_sec": 1518.7874378194003, + "batch_latency_ms": 1348.444126546383, + "single_latency_ms": 510.45899260789156, + "tokens": 2048.0 + }, + "mercury_continuation": { + "1": { + "token_acc": 0.0400390625, + "exact_seq_acc": 0.0, + "tok_per_sec": 33315.36467238215, + "batch_latency_ms": 61.47313769906759, + "single_latency_ms": 7.682473957538605, + "tokens": 2048.0 + }, + "2": { + "token_acc": 0.0400390625, + "exact_seq_acc": 0.0, + "tok_per_sec": 52423.930452321954, + "batch_latency_ms": 39.0661284327507, + "single_latency_ms": 15.342708677053452, + "tokens": 2048.0 + }, + "4": { + "token_acc": 0.03955078125, + "exact_seq_acc": 0.0, + "tok_per_sec": 43382.67441657036, + "batch_latency_ms": 47.20778577029705, + "single_latency_ms": 30.572386272251606, + "tokens": 2048.0 + }, + "8": { + "token_acc": 0.03759765625, + "exact_seq_acc": 0.0, + "tok_per_sec": 21693.84538256514, + "batch_latency_ms": 94.40465550869703, + "single_latency_ms": 61.32346969097853, + "tokens": 2048.0 + } + }, + "mercury_infill": { + "1": { + "token_acc": 0.0400390625, + "exact_seq_acc": 0.0, + "tok_per_sec": 147729.23657320702, + "batch_latency_ms": 13.863200321793556, + "single_latency_ms": 7.631411962211132, + "tokens": 2048.0 + }, + "2": { + "token_acc": 0.0400390625, + "exact_seq_acc": 0.0, + "tok_per_sec": 86371.16188730345, + "batch_latency_ms": 23.711618036031723, + "single_latency_ms": 15.290173701941967, + "tokens": 2048.0 + }, + "4": { + "token_acc": 0.03955078125, + "exact_seq_acc": 0.0, + "tok_per_sec": 43263.386592025425, + "batch_latency_ms": 47.33794927597046, + "single_latency_ms": 30.280465446412563, + "tokens": 2048.0 + }, + "8": { + "token_acc": 0.03759765625, + "exact_seq_acc": 0.0, + "tok_per_sec": 21754.902379810777, + "batch_latency_ms": 94.13970075547695, + "single_latency_ms": 44.69598866999149, + "tokens": 2048.0 + } + }, + "comparisons": { + "continuation_speedup_vs_ar": { + "1": 21.935501863390904, + "2": 34.516963432084765, + "4": 28.564019780712073, + "8": 14.283661322424479 + }, + "infill_speedup_vs_ar_continuation": { + "1": 97.26788153140727, + "2": 56.868498998984926, + "4": 28.485478293223736, + "8": 14.323862469553598 + } + }, + "examples": [ + { + "prefix": "customers and building brand awareness through social media initiatives *Performing website analyses utilizing Google Analytics *Participating in planning, executing and analyzing email marketing campaigns - Web Design *Designing web imagery used on Mason Companies websites, display advert", + "continuation_target": "ising and landing pages *Designing imagery and HTML for email marketing campaigns *Participating in various projects such as launching new mobile sit", + "ar_continuation": "ising, and marketing - Web Design *Producing web design ideas for websites, websites, and websites - Web Design *Producing web design ideas for we", + "mercury_continuation_step1": "isment,,,,,,,,,,,,,,inginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginging,inginging", + "infill_target": "ising and landing pages *Designing imagery and HTML for email marketing campaigns *Participating in various projects such as launching new mobile sit", + "infill_suffix": "es and websites *Researching and implementing emerging web design technologies and processes - IT Developer *Java J2EE development *Ecommerce Development on Mason", + "mercury_infill_step1": "isment,,,,,,,,,,,,,,inginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginging,inginging" + }, + { + "prefix": "and are responsible for compliance with all applicable laws. You may not access, download, use or export the Information on this web site in violation of U.S. export laws or regulations, or in violation of any applicable local laws or regulations. Webroot Inc. (\"Webroot\") is committed to protecting the intellectual property rights of third parties, and", + "continuation_target": "Webroot requests that its customers and community members do the same. Webroot has no responsibility for content on other websites that you may find or access when using Webroot's produ", + "ar_continuation": "to protecting the privacy of third parties. Webroot is not responsible for the privacy of third parties. Webroot is not responsible for the privacy of third part", + "mercury_continuation_step1": "theinging the the,,,ss,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,", + "infill_target": "Webroot requests that its customers and community members do the same. Webroot has no responsibility for content on other websites that you may find or access when using Webroot's produ", + "infill_suffix": "cts or services, and such content may be protected by copyright and the intellectual property laws of the United States and/or other countries. Without prior notice and at any time, We", + "mercury_infill_step1": "theinging the the,,,ss,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,," + }, + { + "prefix": "os predicted will emerge without deliberate and intentional actions to support them. And the extent to which they can be shaped to further societal goals will depend on constructive dialogue between governments and citizens themselves. Ultimately, this new publication aims to contribute to this dialogue, so that both developing and developed countries", + "continuation_target": "are more likely to leap into better futures. Text co-authored with Tom Steinberg, originally cross-posted from the World Bank\u2019s Governance for Development", + "ar_continuation": "can be more effectively dialogued and delivered to the public. The publication is aimed at the public and private sectors, and is based on the publication\u2019s publication d", + "mercury_continuation_step1": "will the the to the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the", + "infill_target": "are more likely to leap into better futures. Text co-authored with Tom Steinberg, originally cross-posted from the World Bank\u2019s Governance for Development", + "infill_suffix": "blog. You can also read another article about this report in Apolitical here. While I\u2019m at it: if you work in public service and care about making government work better, I highly recommend", + "mercury_infill_step1": "will the the to the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the" + } + ] +} \ No newline at end of file diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.md b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.md new file mode 100644 index 0000000000..5da6238fec --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/decode_benchmark.md @@ -0,0 +1,64 @@ +# AR vs Mercury Decode Benchmark + +## Setup +- Baseline checkpoint: `/workspace/parameter-golf/records/track_non_record_16mb/2026-04-09_MercuryStyleCompactTextDiffusion/benchmarks/real_8x_seed2026_vs_baseline_20260410_140157/ar_baseline/final_model.pt` +- Mercury checkpoint: `/workspace/parameter-golf/records/track_non_record_16mb/2026-04-09_MercuryStyleCompactTextDiffusion/final_model.pt` +- Device: `cuda` +- Examples: `32` +- Continuation task: prefix `128` tokens, predict next `64` tokens +- Infill task: prefix `128` tokens, infill `64` tokens with suffix `64` visible + +## Highlights + +- AR continuation throughput: `1518.79` tok/s for `64`-token greedy decode. +- Best Mercury continuation accuracy in this run: `0.0400` at `1` refinement step(s). +- Fastest Mercury continuation setting: `52423.93` tok/s at `2` step(s), which is `34.52x` AR continuation throughput. +- Fastest Mercury infill setting: `147729.24` tok/s at `1` step(s), which is `97.27x` AR continuation throughput on the same hardware. + +## Continuation + +| Model | Mode | Token Acc | Exact Seq | Tok/s | Speedup vs AR | Batch Latency ms | Single-example Latency ms | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| AR baseline | greedy | 0.0215 | 0.0000 | 1518.79 | 1.00x | 1348.44 | 510.46 | +| Mercury | 1 refinement step(s) | 0.0400 | 0.0000 | 33315.36 | 21.94x | 61.47 | 7.68 | +| Mercury | 2 refinement step(s) | 0.0400 | 0.0000 | 52423.93 | 34.52x | 39.07 | 15.34 | +| Mercury | 4 refinement step(s) | 0.0396 | 0.0000 | 43382.67 | 28.56x | 47.21 | 30.57 | +| Mercury | 8 refinement step(s) | 0.0376 | 0.0000 | 21693.85 | 14.28x | 94.40 | 61.32 | + +## Infill + +| Model | Mode | Token Acc | Exact Seq | Tok/s | Speedup vs AR Continuation | Batch Latency ms | Single-example Latency ms | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Mercury | 1 refinement step(s) | 0.0400 | 0.0000 | 147729.24 | 97.27x | 13.86 | 7.63 | +| Mercury | 2 refinement step(s) | 0.0400 | 0.0000 | 86371.16 | 56.87x | 23.71 | 15.29 | +| Mercury | 4 refinement step(s) | 0.0396 | 0.0000 | 43263.39 | 28.49x | 47.34 | 30.28 | +| Mercury | 8 refinement step(s) | 0.0376 | 0.0000 | 21754.90 | 14.32x | 94.14 | 44.70 | + +## Example Outputs + +### Example 1 +- Prefix: customers and building brand awareness through social media initiatives *Performing website analyses utilizing Google Analytics *Participating in planning, executing and analyzing email marketing campaigns - Web Design *Designing web imagery used on Mason Companies websites, display advert +- Continuation target: ising and landing pages *Designing imagery and HTML for email marketing campaigns *Participating in various projects such as launching new mobile sit +- AR continuation: ising, and marketing - Web Design *Producing web design ideas for websites, websites, and websites - Web Design *Producing web design ideas for we +- Mercury continuation (1 step): isment,,,,,,,,,,,,,,inginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginging,inginging +- Infill target: ising and landing pages *Designing imagery and HTML for email marketing campaigns *Participating in various projects such as launching new mobile sit +- Infill suffix: es and websites *Researching and implementing emerging web design technologies and processes - IT Developer *Java J2EE development *Ecommerce Development on Mason +- Mercury infill (1 step): isment,,,,,,,,,,,,,,inginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginginging,inginging + +### Example 2 +- Prefix: and are responsible for compliance with all applicable laws. You may not access, download, use or export the Information on this web site in violation of U.S. export laws or regulations, or in violation of any applicable local laws or regulations. Webroot Inc. ("Webroot") is committed to protecting the intellectual property rights of third parties, and +- Continuation target: Webroot requests that its customers and community members do the same. Webroot has no responsibility for content on other websites that you may find or access when using Webroot's produ +- AR continuation: to protecting the privacy of third parties. Webroot is not responsible for the privacy of third parties. Webroot is not responsible for the privacy of third part +- Mercury continuation (1 step): theinging the the,,,ss,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, +- Infill target: Webroot requests that its customers and community members do the same. Webroot has no responsibility for content on other websites that you may find or access when using Webroot's produ +- Infill suffix: cts or services, and such content may be protected by copyright and the intellectual property laws of the United States and/or other countries. Without prior notice and at any time, We +- Mercury infill (1 step): theinging the the,,,ss,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, + +### Example 3 +- Prefix: os predicted will emerge without deliberate and intentional actions to support them. And the extent to which they can be shaped to further societal goals will depend on constructive dialogue between governments and citizens themselves. Ultimately, this new publication aims to contribute to this dialogue, so that both developing and developed countries +- Continuation target: are more likely to leap into better futures. Text co-authored with Tom Steinberg, originally cross-posted from the World Bank’s Governance for Development +- AR continuation: can be more effectively dialogued and delivered to the public. The publication is aimed at the public and private sectors, and is based on the publication’s publication d +- Mercury continuation (1 step): will the the to the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the +- Infill target: are more likely to leap into better futures. Text co-authored with Tom Steinberg, originally cross-posted from the World Bank’s Governance for Development +- Infill suffix: blog. You can also read another article about this report in Apolitical here. While I’m at it: if you work in public service and care about making government work better, I highly recommend +- Mercury infill (1 step): will the the to the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/submission.json b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/submission.json new file mode 100644 index 0000000000..150f9eafff --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/submission.json @@ -0,0 +1,89 @@ +{ + "author": "Simon Marcus", + "github_id": "simon-marcus", + "name": "Mercury in Retrograde", + "blurb": "Non-record text-diffusion submission: a compact Mercury-style denoising model with hybrid corruption, self-conditioning, continuation/infill parallel refinement metrics, and an intentionally documented negative result.", + "date": "2026-04-11", + "track": "non_record_16mb", + "non_record": true, + "val_loss": 2.45910030, + "val_bpb": 1.45641771, + "val_loss_std": 0.00620926, + "val_bpb_std": 0.00367747, + "bytes_total": 15677283, + "bytes_code": 83831, + "bytes_model_int8_zlib_max": 15593452, + "hardware": "8xH100 80GB SXM", + "seeds": [1337, 42, 2026], + "seed_results": { + "1337": { + "val_loss": 2.45349105, + "val_bpb": 1.45309560, + "artifact_bytes": 15677283, + "model_int8_zlib_bytes": 15593452, + "steps": 4628, + "step_avg_ms": 129.64 + }, + "42": { + "val_loss": 2.46577237, + "val_bpb": 1.46036929, + "artifact_bytes": 15531183, + "model_int8_zlib_bytes": 15447352, + "steps": 4912, + "step_avg_ms": 122.14 + }, + "2026": { + "val_loss": 2.45803749, + "val_bpb": 1.45578825, + "artifact_bytes": 15500938, + "model_int8_zlib_bytes": 15417107, + "steps": 4926, + "step_avg_ms": 121.82 + } + }, + "technique_summary": "Mercury-style compact text diffusion: progressive 25-35% hybrid corruption, mixed continuation/infill denoising, self-conditioning, small clean CE prior, and parallel refinement evaluation.", + "diffusion_recipe": { + "DIFFUSION_TRAIN_MODE": "mercury", + "DIFFUSION_CORRUPTION_MODE": "hybrid", + "DIFFUSION_HYBRID_MASK_PROB": 0.5, + "DIFFUSION_LOSS_WEIGHT": 1.0, + "DIFFUSION_PROGRESSIVE": 1, + "DIFFUSION_MIN_MASK_RATIO": 0.25, + "DIFFUSION_MAX_MASK_RATIO": 0.35, + "DIFFUSION_SPAN_LEN": 64, + "DIFFUSION_SPANS_PER_SEQ": 8, + "MERCURY_TASK_MODE": "mixed", + "MERCURY_CONTINUATION_PROB": 0.75, + "MERCURY_CLEAN_LOSS_WEIGHT": 0.10, + "MERCURY_SELF_CONDITION_PROB": 1.0, + "MERCURY_SELF_CONDITION_FRACTION": 0.75 + }, + "parallel_eval_mean": { + "continuation": { + "steps_1": {"token_acc": 0.0348, "tok_per_sec": 25123.81}, + "steps_2": {"token_acc": 0.0345, "tok_per_sec": 36784.34}, + "steps_4": {"token_acc": 0.0348, "tok_per_sec": 25990.89}, + "steps_8": {"token_acc": 0.0355, "tok_per_sec": 13005.42} + }, + "infill": { + "steps_1": {"token_acc": 0.0377, "tok_per_sec": 104971.51}, + "steps_2": {"token_acc": 0.0384, "tok_per_sec": 52357.51}, + "steps_4": {"token_acc": 0.0410, "tok_per_sec": 26075.30}, + "steps_8": {"token_acc": 0.0404, "tok_per_sec": 12922.12} + } + }, + "matched_decode_benchmark": { + "hardware": "1xH100", + "checkpoint_seed": 2026, + "examples": 32, + "prefix_tokens": 128, + "target_tokens": 64, + "ar_continuation_tok_per_sec": 1518.79, + "mercury_continuation_step1_token_acc": 0.0400, + "mercury_continuation_step1_tok_per_sec": 33315.36, + "mercury_continuation_step1_speedup_vs_ar": 21.94, + "mercury_infill_step1_token_acc": 0.0400, + "mercury_infill_step1_tok_per_sec": 147729.24, + "mercury_infill_step1_speedup_vs_ar_continuation": 97.27 + } +} diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_gpt.py b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_gpt.py new file mode 100644 index 0000000000..703aa5d3f3 --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_gpt.py @@ -0,0 +1,1849 @@ +""" +Mercury-style compact diffusion submission candidate. + +This fork keeps exact Parameter Golf BPB evaluation and export path, but it +adds a diffusion-native training mode and extra reporting for parallel +coarse-to-fine refinement. The goal is a non-record submission that is +interesting specifically because it acts like a text diffusion model rather +than an AR model with a tiny auxiliary loss. +""" + +from __future__ import annotations + +import copy +import glob +import io +import math +import os +import random +import subprocess +import sys +import time +import uuid +import zlib +from pathlib import Path + +import numpy as np +import torch +import torch.distributed as dist +import torch.nn.functional as F +from torch import Tensor, nn +from torch.nn.parallel import DistributedDataParallel as DDP + +# ----------------------------- +# HYPERPARAMETERS +# ----------------------------- +# Default Simple Baseline run: +# - 9 transformer blocks at width 512 +# - 8 attention heads with 4 KV heads (GQA) and 2x MLP expansion +# - vocab size 1024, sequence length 1024, tied embeddings +# - 524,288 train tokens per step for 20,000 iterations with a ~10 minute cap + +class Hyperparameters: + # Data paths are shard globs produced by the existing preprocessing pipeline. + data_path = os.environ.get("DATA_PATH", "./data/datasets/fineweb10B_sp1024") + train_files = os.path.join(data_path, "fineweb_train_*.bin") + val_files = os.path.join(data_path, "fineweb_val_*.bin") + tokenizer_path = os.environ.get("TOKENIZER_PATH", "./data/tokenizers/fineweb_1024_bpe.model") + tokenizer_meta_path = os.environ.get("TOKENIZER_META_PATH", "") + tokenizer_meta_validate = bool(int(os.environ.get("TOKENIZER_META_VALIDATE", "0"))) + run_id = os.environ.get("RUN_ID", str(uuid.uuid4())) + seed = int(os.environ.get("SEED", 1337)) + + # Validation cadence and batch size. Validation always uses the full fineweb_val split. + val_batch_size = int(os.environ.get("VAL_BATCH_SIZE", 524_288)) + val_loss_every = int(os.environ.get("VAL_LOSS_EVERY", 1000)) + train_log_every = int(os.environ.get("TRAIN_LOG_EVERY", 200)) + + # Training length. + iterations = int(os.environ.get("ITERATIONS", 20000)) + warmdown_iters = int(os.environ.get("WARMDOWN_ITERS", 1200)) + warmup_steps = int(os.environ.get("WARMUP_STEPS", 20)) + train_batch_tokens = int(os.environ.get("TRAIN_BATCH_TOKENS", 524_288)) + train_seq_len = int(os.environ.get("TRAIN_SEQ_LEN", 1024)) + max_wallclock_seconds = float(os.environ.get("MAX_WALLCLOCK_SECONDS", 600.0)) + qk_gain_init = float(os.environ.get("QK_GAIN_INIT", 1.5)) + + # Model shape. + vocab_size = int(os.environ.get("VOCAB_SIZE", 1024)) + num_layers = int(os.environ.get("NUM_LAYERS", 9)) + num_kv_heads = int(os.environ.get("NUM_KV_HEADS", 4)) + model_dim = int(os.environ.get("MODEL_DIM", 512)) + num_heads = int(os.environ.get("NUM_HEADS", 8)) + mlp_mult = int(os.environ.get("MLP_MULT", 2)) + tie_embeddings = bool(int(os.environ.get("TIE_EMBEDDINGS", "1"))) + rope_base = float(os.environ.get("ROPE_BASE", 10000.0)) + logit_softcap = float(os.environ.get("LOGIT_SOFTCAP", 30.0)) + + # Optimizer hyperparameters. + embed_lr = float(os.environ.get("EMBED_LR", 0.6)) + head_lr = float(os.environ.get("HEAD_LR", 0.008)) + tied_embed_lr = float(os.environ.get("TIED_EMBED_LR", 0.05)) + tied_embed_init_std = float(os.environ.get("TIED_EMBED_INIT_STD", 0.005)) + matrix_lr = float(os.environ.get("MATRIX_LR", 0.04)) + scalar_lr = float(os.environ.get("SCALAR_LR", 0.04)) + muon_momentum = float(os.environ.get("MUON_MOMENTUM", 0.95)) + muon_backend_steps = int(os.environ.get("MUON_BACKEND_STEPS", 5)) + muon_momentum_warmup_start = float(os.environ.get("MUON_MOMENTUM_WARMUP_START", 0.85)) + muon_momentum_warmup_steps = int(os.environ.get("MUON_MOMENTUM_WARMUP_STEPS", 500)) + beta1 = float(os.environ.get("BETA1", 0.9)) + beta2 = float(os.environ.get("BETA2", 0.95)) + adam_eps = float(os.environ.get("ADAM_EPS", 1e-8)) + grad_clip_norm = float(os.environ.get("GRAD_CLIP_NORM", 0.0)) + use_compile = bool(int(os.environ.get("USE_COMPILE", "0"))) + + # Text-diffusion hybrid knobs. + diffusion_enabled = bool(int(os.environ.get("DIFFUSION_ENABLED", "1"))) + diffusion_train_mode = os.environ.get("DIFFUSION_TRAIN_MODE", "mercury").strip().lower() + diffusion_corruption_mode = os.environ.get("DIFFUSION_CORRUPTION_MODE", "hybrid").strip().lower() + diffusion_hybrid_mask_prob = float(os.environ.get("DIFFUSION_HYBRID_MASK_PROB", 0.5)) + diffusion_loss_weight = float(os.environ.get("DIFFUSION_LOSS_WEIGHT", 1.0)) + diffusion_kd_temperature = float(os.environ.get("DIFFUSION_KD_TEMPERATURE", 1.0)) + diffusion_span_len = int(os.environ.get("DIFFUSION_SPAN_LEN", 64)) + diffusion_spans_per_seq = int(os.environ.get("DIFFUSION_SPANS_PER_SEQ", 8)) + diffusion_min_mask_ratio = float(os.environ.get("DIFFUSION_MIN_MASK_RATIO", 0.25)) + diffusion_max_mask_ratio = float(os.environ.get("DIFFUSION_MAX_MASK_RATIO", 0.35)) + diffusion_progressive = bool(int(os.environ.get("DIFFUSION_PROGRESSIVE", "1"))) + diffusion_warmstart_ce_only_steps = int(os.environ.get("DIFFUSION_WARMSTART_CE_ONLY_STEPS", 100)) + diffusion_warmstart_ce_only_seconds = float(os.environ.get("DIFFUSION_WARMSTART_CE_ONLY_SECONDS", 0.0)) + diffusion_warmstart_ce_only_fraction = float(os.environ.get("DIFFUSION_WARMSTART_CE_ONLY_FRACTION", 0.0)) + mercury_task_mode = os.environ.get("MERCURY_TASK_MODE", "mixed").strip().lower() + mercury_continuation_prob = float(os.environ.get("MERCURY_CONTINUATION_PROB", 0.75)) + mercury_clean_loss_weight = float(os.environ.get("MERCURY_CLEAN_LOSS_WEIGHT", 0.10)) + mercury_self_condition_prob = float(os.environ.get("MERCURY_SELF_CONDITION_PROB", 1.0)) + mercury_self_condition_fraction = float(os.environ.get("MERCURY_SELF_CONDITION_FRACTION", 0.75)) + mercury_unroll_steps = int(os.environ.get("MERCURY_UNROLL_STEPS", 1)) + mercury_first_pass_loss_weight = float(os.environ.get("MERCURY_FIRST_PASS_LOSS_WEIGHT", 0.5)) + mercury_second_pass_loss_weight = float(os.environ.get("MERCURY_SECOND_PASS_LOSS_WEIGHT", 1.0)) + mercury_commit_strategy = os.environ.get("MERCURY_COMMIT_STRATEGY", "confidence").strip().lower() + report_parallel_metrics = bool(int(os.environ.get("REPORT_PARALLEL_METRICS", "1"))) + parallel_eval_examples = int(os.environ.get("PARALLEL_EVAL_EXAMPLES", 16)) + parallel_eval_prefix_len = int(os.environ.get("PARALLEL_EVAL_PREFIX_LEN", 128)) + parallel_eval_suffix_len = int(os.environ.get("PARALLEL_EVAL_SUFFIX_LEN", 64)) + parallel_eval_span_len = int(os.environ.get("PARALLEL_EVAL_SPAN_LEN", 64)) + parallel_eval_steps = os.environ.get("PARALLEL_EVAL_STEPS", "1,2,4,8") + +# ----------------------------- +# MUON OPTIMIZER +# ----------------------------- +# +# As borrowed from modded-nanogpt +# Background on Muon: https://kellerjordan.github.io/posts/muon/ + +def zeropower_via_newtonschulz5(G: Tensor, steps: int = 10, eps: float = 1e-7) -> Tensor: + # Orthogonalize a 2D update matrix with a fast Newton-Schulz iteration. + # Muon uses this to normalize matrix-shaped gradients before applying them. + a, b, c = (3.4445, -4.7750, 2.0315) + X = G.bfloat16() + X /= X.norm() + eps + transposed = G.size(0) > G.size(1) + if transposed: + X = X.T + for _ in range(steps): + A = X @ X.T + B = b * A + c * A @ A + X = a * X + B @ X + return X.T if transposed else X + + +class Muon(torch.optim.Optimizer): + def __init__(self, params, lr: float, momentum: float, backend_steps: int, nesterov: bool = True): + super().__init__( + params, + dict(lr=lr, momentum=momentum, backend_steps=backend_steps, nesterov=nesterov), + ) + + @torch.no_grad() + def step(self, closure=None): + loss = None + if closure is not None: + with torch.enable_grad(): + loss = closure() + + distributed = dist.is_available() and dist.is_initialized() + world_size = dist.get_world_size() if distributed else 1 + rank = dist.get_rank() if distributed else 0 + + for group in self.param_groups: + params = group["params"] + if not params: + continue + lr = group["lr"] + momentum = group["momentum"] + backend_steps = group["backend_steps"] + nesterov = group["nesterov"] + + total_params = sum(int(p.numel()) for p in params) + updates_flat = torch.zeros(total_params, device=params[0].device, dtype=torch.bfloat16) + + curr = 0 + for i, p in enumerate(params): + if i % world_size == rank and p.grad is not None: + g = p.grad + state = self.state[p] + if "momentum_buffer" not in state: + state["momentum_buffer"] = torch.zeros_like(g) + buf = state["momentum_buffer"] + buf.mul_(momentum).add_(g) + if nesterov: + g = g.add(buf, alpha=momentum) + g = zeropower_via_newtonschulz5(g, steps=backend_steps) + # Scale correction from Muon reference implementations. + g *= max(1, g.size(0) / g.size(1)) ** 0.5 + updates_flat[curr : curr + p.numel()] = g.reshape(-1) + curr += p.numel() + + if distributed: + dist.all_reduce(updates_flat, op=dist.ReduceOp.SUM) + + curr = 0 + for p in params: + g = updates_flat[curr : curr + p.numel()].view_as(p).to(dtype=p.dtype) + p.add_(g, alpha=-lr) + curr += p.numel() + + return loss + + +# ----------------------------- +# TOKENIZER-AGNOSTIC EVALUATION SETUP +# ----------------------------- +# +# It's common for small models have a large fraction of their parameters be embeddings, since the 2 * d_model * d_vocab vectors can be gigantic. +# Instead of locking the tokenizer, we let you bring your own and calculate our validation metrics on the average compression of the validation set. +# We calculate BPB (bits-per-byte) instead of validation loss, so we need methods to count the number of bits per token in the tokenizer. +# Note: Submissions that edit the tokenizer will be examined more carefully, since screwing this up might unjustly improve your score. + +TOKENIZER_META_FORMAT_VERSION = 1 +TOKENIZER_META_SUFFIX = ".meta.npz" + + +def _derive_tokenizer_meta_path(tokenizer_path: str) -> Path: + tokenizer = Path(tokenizer_path) + if tokenizer.suffix == ".model": + return tokenizer.with_suffix(TOKENIZER_META_SUFFIX) + return tokenizer.with_name(tokenizer.name + TOKENIZER_META_SUFFIX) + + +def _torch_luts_from_numpy( + base_bytes_np: np.ndarray, + has_leading_space_np: np.ndarray, + is_boundary_token_np: np.ndarray, + *, + device: torch.device, +) -> tuple[Tensor, Tensor, Tensor]: + return ( + torch.tensor(base_bytes_np, dtype=torch.int16, device=device), + torch.tensor(has_leading_space_np, dtype=torch.bool, device=device), + torch.tensor(is_boundary_token_np, dtype=torch.bool, device=device), + ) + + +def build_sentencepiece_luts_np(sp, vocab_size: int) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + sp_vocab_size = int(sp.vocab_size()) + table_size = max(sp_vocab_size, vocab_size) + base_bytes_np = np.zeros((table_size,), dtype=np.int16) + has_leading_space_np = np.zeros((table_size,), dtype=np.bool_) + is_boundary_token_np = np.ones((table_size,), dtype=np.bool_) + for token_id in range(sp_vocab_size): + if sp.is_control(token_id) or sp.is_unknown(token_id) or sp.is_unused(token_id): + continue + is_boundary_token_np[token_id] = False + if sp.is_byte(token_id): + base_bytes_np[token_id] = 1 + continue + piece = sp.id_to_piece(token_id) + if piece.startswith("▁"): + has_leading_space_np[token_id] = True + piece = piece[1:] + base_bytes_np[token_id] = len(piece.encode("utf-8")) + return base_bytes_np, has_leading_space_np, is_boundary_token_np + + +def build_sentencepiece_luts(sp, vocab_size: int, device: torch.device) -> tuple[Tensor, Tensor, Tensor]: + return _torch_luts_from_numpy(*build_sentencepiece_luts_np(sp, vocab_size), device=device) + + +def load_tokenizer_meta_luts_np(meta_path: Path, vocab_size: int) -> tuple[np.ndarray, np.ndarray, np.ndarray, dict[str, object]]: + with np.load(meta_path, allow_pickle=False) as data: + format_version = int(data["format_version"]) + if format_version != TOKENIZER_META_FORMAT_VERSION: + raise ValueError( + f"Unsupported tokenizer metadata format_version={format_version} in {meta_path}; " + f"expected {TOKENIZER_META_FORMAT_VERSION}" + ) + meta_vocab_size = int(data["vocab_size"]) + table_size = max(meta_vocab_size, vocab_size) + base_bytes_np = np.zeros((table_size,), dtype=np.int16) + has_leading_space_np = np.zeros((table_size,), dtype=np.bool_) + is_boundary_token_np = np.ones((table_size,), dtype=np.bool_) + base_src = np.asarray(data["base_bytes"], dtype=np.int16) + lead_src = np.asarray(data["has_leading_space"], dtype=np.bool_) + boundary_src = np.asarray(data["is_boundary_token"], dtype=np.bool_) + if not (len(base_src) == len(lead_src) == len(boundary_src) == meta_vocab_size): + raise ValueError(f"Tokenizer metadata arrays in {meta_path} do not match vocab_size={meta_vocab_size}") + base_bytes_np[:meta_vocab_size] = base_src + has_leading_space_np[:meta_vocab_size] = lead_src + is_boundary_token_np[:meta_vocab_size] = boundary_src + metadata = { + "format_version": format_version, + "tokenizer_kind": str(data["tokenizer_kind"].item()), + "source_model_name": str(data["source_model_name"].item()), + "vocab_size": meta_vocab_size, + } + return base_bytes_np, has_leading_space_np, is_boundary_token_np, metadata + + +def load_tokenizer_luts( + *, + tokenizer_path: str, + tokenizer_meta_path: str, + vocab_size: int, + device: torch.device, + validate_meta: bool, +) -> tuple[tuple[Tensor, Tensor, Tensor], dict[str, object]]: + explicit_meta = Path(tokenizer_meta_path).expanduser() if tokenizer_meta_path else None + meta_path = explicit_meta if explicit_meta is not None else _derive_tokenizer_meta_path(tokenizer_path) + meta_path = meta_path.resolve() + tokenizer = Path(tokenizer_path).expanduser().resolve() + if meta_path.is_file(): + base_bytes_np, has_leading_space_np, is_boundary_token_np, metadata = load_tokenizer_meta_luts_np( + meta_path, vocab_size + ) + if validate_meta: + sp_luts = load_tokenizer_luts_from_sentencepiece_np(tokenizer, vocab_size) + if not ( + np.array_equal(base_bytes_np, sp_luts[0]) + and np.array_equal(has_leading_space_np, sp_luts[1]) + and np.array_equal(is_boundary_token_np, sp_luts[2]) + ): + raise ValueError( + f"Tokenizer metadata parity check failed for meta_path={meta_path} tokenizer_path={tokenizer}" + ) + metadata.update( + { + "meta_path": str(meta_path), + "tokenizer_path": str(tokenizer), + "source": "meta", + } + ) + return _torch_luts_from_numpy(base_bytes_np, has_leading_space_np, is_boundary_token_np, device=device), metadata + if explicit_meta is not None: + raise FileNotFoundError(f"TOKENIZER_META_PATH does not exist: {meta_path}") + luts = load_tokenizer_luts_from_sentencepiece_np(tokenizer, vocab_size) + metadata = { + "format_version": None, + "tokenizer_kind": "sentencepiece", + "source_model_name": tokenizer.name, + "vocab_size": max(vocab_size, len(luts[0])), + "meta_path": str(meta_path), + "tokenizer_path": str(tokenizer), + "source": "sentencepiece-fallback", + } + return _torch_luts_from_numpy(*luts, device=device), metadata + + +def load_tokenizer_luts_from_sentencepiece_np( + tokenizer_path: Path, vocab_size: int +) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + try: + import sentencepiece as spm + except ImportError as exc: + raise RuntimeError( + f"Tokenizer metadata not found at { _derive_tokenizer_meta_path(str(tokenizer_path)) } and " + "sentencepiece is unavailable for compatibility fallback" + ) from exc + sp = spm.SentencePieceProcessor(model_file=str(tokenizer_path)) + if int(sp.vocab_size()) != vocab_size: + raise ValueError( + f"VOCAB_SIZE={vocab_size} does not match tokenizer vocab_size={int(sp.vocab_size())}" + ) + return build_sentencepiece_luts_np(sp, vocab_size) + + +def load_validation_tokens(pattern: str, seq_len: int) -> Tensor: + files = [Path(p) for p in sorted(glob.glob(pattern))] + if not files: + raise FileNotFoundError(f"No files found for pattern: {pattern}") + # The export pipeline writes the fixed first-50k-doc validation set to fineweb_val_*. + tokens = torch.cat([load_data_shard(file) for file in files]).contiguous() + usable = ((tokens.numel() - 1) // seq_len) * seq_len + if usable <= 0: + raise ValueError(f"Validation split is too short for TRAIN_SEQ_LEN={seq_len}") + return tokens[: usable + 1] + + +def eval_val( + args: Hyperparameters, + model: nn.Module, + rank: int, + world_size: int, + device: torch.device, + grad_accum_steps: int, + val_tokens: Tensor, + base_bytes_lut: Tensor, + has_leading_space_lut: Tensor, + is_boundary_token_lut: Tensor, +) -> tuple[float, float]: + # Validation computes two metrics: + # - val_loss: token cross-entropy (natural log) + # - val_bpb: tokenizer-agnostic compression metric used by the challenge + local_batch_tokens = args.val_batch_size // (world_size * grad_accum_steps) + if local_batch_tokens < args.train_seq_len: + raise ValueError( + "VAL_BATCH_SIZE must provide at least one sequence per rank; " + f"got VAL_BATCH_SIZE={args.val_batch_size}, WORLD_SIZE={world_size}, " + f"GRAD_ACCUM_STEPS={grad_accum_steps}, TRAIN_SEQ_LEN={args.train_seq_len}" + ) + local_batch_seqs = local_batch_tokens // args.train_seq_len + total_seqs = (val_tokens.numel() - 1) // args.train_seq_len + seq_start = (total_seqs * rank) // world_size + seq_end = (total_seqs * (rank + 1)) // world_size + val_loss_sum = torch.zeros((), device=device, dtype=torch.float64) + val_token_count = torch.zeros((), device=device, dtype=torch.float64) + val_byte_count = torch.zeros((), device=device, dtype=torch.float64) + + model.eval() + with torch.inference_mode(): + for batch_seq_start in range(seq_start, seq_end, local_batch_seqs): + batch_seq_end = min(batch_seq_start + local_batch_seqs, seq_end) + raw_start = batch_seq_start * args.train_seq_len + raw_end = batch_seq_end * args.train_seq_len + 1 + local = val_tokens[raw_start:raw_end].to(device=device, dtype=torch.int64, non_blocking=True) + x = local[:-1].reshape(-1, args.train_seq_len) + y = local[1:].reshape(-1, args.train_seq_len) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + batch_loss = model(x, y).detach() + batch_token_count = float(y.numel()) + val_loss_sum += batch_loss.to(torch.float64) * batch_token_count + val_token_count += batch_token_count + prev_ids = x.reshape(-1) + tgt_ids = y.reshape(-1) + token_bytes = base_bytes_lut[tgt_ids].to(dtype=torch.int16) + token_bytes += (has_leading_space_lut[tgt_ids] & ~is_boundary_token_lut[prev_ids]).to(dtype=torch.int16) + val_byte_count += token_bytes.to(torch.float64).sum() + + if dist.is_available() and dist.is_initialized(): + dist.all_reduce(val_loss_sum, op=dist.ReduceOp.SUM) + dist.all_reduce(val_token_count, op=dist.ReduceOp.SUM) + dist.all_reduce(val_byte_count, op=dist.ReduceOp.SUM) + + val_loss = val_loss_sum / val_token_count + bits_per_token = val_loss.item() / math.log(2.0) + tokens_per_byte = val_token_count.item() / val_byte_count.item() + model.train() + return float(val_loss.item()), float(bits_per_token * tokens_per_byte) + +# ----------------------------- +# POST-TRAINING QUANTIZATION +# ----------------------------- +# +# It's silly to export our model, which is trained in bf16 and fp32, at that same precision. +# Instead, we get approximately the same model (with a small hit) by quantizing the model to int8 & zlib compressing. +# We can then decompress the model and run in higher precision for evaluation, after closing in under the size limit. + +CONTROL_TENSOR_NAME_PATTERNS = tuple( + pattern + for pattern in os.environ.get( + "CONTROL_TENSOR_NAME_PATTERNS", + "attn_scale,attn_scales,mlp_scale,mlp_scales,resid_mix,resid_mixes,q_gain,skip_weight,skip_weights", + ).split(",") + if pattern +) +INT8_KEEP_FLOAT_FP32_NAME_PATTERNS = tuple( + pattern + for pattern in os.environ.get( + "INT8_KEEP_FLOAT_FP32_NAME_PATTERNS", + ",".join(CONTROL_TENSOR_NAME_PATTERNS), + ).split(",") + if pattern +) +INT8_KEEP_FLOAT_MAX_NUMEL = 65_536 +INT8_KEEP_FLOAT_STORE_DTYPE = torch.float16 +INT8_PER_ROW_SCALE_DTYPE = torch.float16 +INT8_CLIP_PERCENTILE = 99.99984 +INT8_CLIP_Q = INT8_CLIP_PERCENTILE / 100.0 + +def tensor_nbytes(t: Tensor) -> int: + return int(t.numel()) * int(t.element_size()) + +def keep_float_tensor(name: str, t: Tensor, passthrough_orig_dtypes: dict[str, str]) -> Tensor: + if any(pattern in name for pattern in INT8_KEEP_FLOAT_FP32_NAME_PATTERNS): + return t.float().contiguous() + if t.dtype in {torch.float32, torch.bfloat16}: + passthrough_orig_dtypes[name] = str(t.dtype).removeprefix("torch.") + return t.to(dtype=INT8_KEEP_FLOAT_STORE_DTYPE).contiguous() + return t + +def quantize_float_tensor(t: Tensor) -> tuple[Tensor, Tensor]: + t32 = t.float() + if t32.ndim == 2: + # Matrices get one scale per row, which usually tracks output-channel + # ranges much better than a single tensor-wide scale. + clip_abs = ( + torch.quantile(t32.abs(), INT8_CLIP_Q, dim=1) + if t32.numel() + else torch.empty((t32.shape[0],), dtype=torch.float32) + ) + clipped = torch.maximum(torch.minimum(t32, clip_abs[:, None]), -clip_abs[:, None]) + scale = (clip_abs / 127.0).clamp_min(1.0 / 127.0) + q = torch.clamp(torch.round(clipped / scale[:, None]), -127, 127).to(torch.int8).contiguous() + return q, scale.to(dtype=INT8_PER_ROW_SCALE_DTYPE).contiguous() + + # Vectors / scalars use a simpler per-tensor scale. + clip_abs = float(torch.quantile(t32.abs().flatten(), INT8_CLIP_Q).item()) if t32.numel() else 0.0 + scale = torch.tensor(clip_abs / 127.0 if clip_abs > 0 else 1.0, dtype=torch.float32) + q = torch.clamp(torch.round(torch.clamp(t32, -clip_abs, clip_abs) / scale), -127, 127).to(torch.int8).contiguous() + return q, scale + +def quantize_state_dict_int8(state_dict: dict[str, Tensor]): + # Single supported clean-script export format: + # - per-row int8 for 2D float tensors + # - per-tensor int8 for other float tensors + # - exact passthrough for non-floats + # - passthrough for small float tensors, stored as fp16 to save bytes + quantized: dict[str, Tensor] = {} + scales: dict[str, Tensor] = {} + dtypes: dict[str, str] = {} + passthrough: dict[str, Tensor] = {} + passthrough_orig_dtypes: dict[str, str] = {} + qmeta: dict[str, dict[str, object]] = {} + stats = dict.fromkeys( + ("param_count", "num_tensors", "num_float_tensors", "num_nonfloat_tensors", "baseline_tensor_bytes", "int8_payload_bytes"), + 0, + ) + + for name, tensor in state_dict.items(): + t = tensor.detach().to("cpu").contiguous() + stats["param_count"] += int(t.numel()) + stats["num_tensors"] += 1 + stats["baseline_tensor_bytes"] += tensor_nbytes(t) + + if not t.is_floating_point(): + stats["num_nonfloat_tensors"] += 1 + passthrough[name] = t + stats["int8_payload_bytes"] += tensor_nbytes(t) + continue + + # Small float tensors are cheap enough to keep directly. We still downcast + # fp32/bf16 passthrough tensors to fp16 so metadata does not dominate size. + if t.numel() <= INT8_KEEP_FLOAT_MAX_NUMEL: + kept = keep_float_tensor(name, t, passthrough_orig_dtypes) + passthrough[name] = kept + stats["int8_payload_bytes"] += tensor_nbytes(kept) + continue + + stats["num_float_tensors"] += 1 + q, s = quantize_float_tensor(t) + if s.ndim > 0: + qmeta[name] = {"scheme": "per_row", "axis": 0} + quantized[name] = q + scales[name] = s + dtypes[name] = str(t.dtype).removeprefix("torch.") + stats["int8_payload_bytes"] += tensor_nbytes(q) + tensor_nbytes(s) + + obj: dict[str, object] = { + "__quant_format__": "int8_clean_per_row_v1", + "quantized": quantized, + "scales": scales, + "dtypes": dtypes, + "passthrough": passthrough, + } + if qmeta: + obj["qmeta"] = qmeta + if passthrough_orig_dtypes: + obj["passthrough_orig_dtypes"] = passthrough_orig_dtypes + return obj, stats + +def dequantize_state_dict_int8(obj: dict[str, object]) -> dict[str, Tensor]: + out: dict[str, Tensor] = {} + qmeta = obj.get("qmeta", {}) + passthrough_orig_dtypes = obj.get("passthrough_orig_dtypes", {}) + for name, q in obj["quantized"].items(): + dtype = getattr(torch, obj["dtypes"][name]) + s = obj["scales"][name] + if qmeta.get(name, {}).get("scheme") == "per_row" or s.ndim > 0: + s = s.to(dtype=torch.float32) + # Broadcast the saved row scale back across trailing dimensions. + out[name] = (q.float() * s.view(q.shape[0], *([1] * (q.ndim - 1)))).to(dtype=dtype).contiguous() + else: + scale = float(s.item()) + out[name] = (q.float() * scale).to(dtype=dtype).contiguous() + for name, t in obj["passthrough"].items(): + # Restore small tensors, undoing the temporary fp16 storage cast if needed. + out_t = t.detach().to("cpu").contiguous() + orig_dtype = passthrough_orig_dtypes.get(name) + if isinstance(orig_dtype, str): + out_t = out_t.to(dtype=getattr(torch, orig_dtype)).contiguous() + out[name] = out_t + return out + + +# ----------------------------- +# DATA LOADING +# ----------------------------- + +def load_data_shard(file: Path) -> Tensor: + header_bytes = 256 * np.dtype(" None: + self.file_idx = (self.file_idx + 1) % len(self.files) + self.tokens = load_data_shard(self.files[self.file_idx]) + self.pos = 0 + + def take(self, n: int) -> Tensor: + chunks: list[Tensor] = [] + remaining = n + while remaining > 0: + avail = self.tokens.numel() - self.pos + if avail <= 0: + self._advance_file() + continue + k = min(remaining, avail) + chunks.append(self.tokens[self.pos : self.pos + k]) + self.pos += k + remaining -= k + return chunks[0] if len(chunks) == 1 else torch.cat(chunks) + + +class DistributedTokenLoader: + # Each call consumes a contiguous chunk from the shared token stream, then slices out + # one disjoint span per rank. The extra "+1" token lets us build (x, y) by shifting. + def __init__(self, pattern: str, rank: int, world_size: int, device: torch.device): + self.rank = rank + self.world_size = world_size + self.device = device + self.stream = TokenStream(pattern) + + def next_batch(self, global_tokens: int, seq_len: int, grad_accum_steps: int) -> tuple[Tensor, Tensor]: + local_tokens = global_tokens // (self.world_size * grad_accum_steps) + per_rank_span = local_tokens + 1 + chunk = self.stream.take(per_rank_span * self.world_size) + start = self.rank * per_rank_span + local = chunk[start : start + per_rank_span].to(dtype=torch.int64) + x = local[:-1].reshape(-1, seq_len) + y = local[1:].reshape(-1, seq_len) + return x.to(self.device, non_blocking=True), y.to(self.device, non_blocking=True) + + +def sample_diffusion_target_mask(args: Hyperparameters, batch_size: int, seq_len: int, step: int, device: torch.device) -> Tensor: + if args.diffusion_span_len <= 0: + raise ValueError(f"DIFFUSION_SPAN_LEN must be positive, got {args.diffusion_span_len}") + if args.diffusion_min_mask_ratio <= 0.0 or args.diffusion_max_mask_ratio <= 0.0: + raise ValueError("Diffusion mask ratios must be positive") + if args.diffusion_min_mask_ratio > args.diffusion_max_mask_ratio: + raise ValueError("DIFFUSION_MIN_MASK_RATIO must be <= DIFFUSION_MAX_MASK_RATIO") + + progress = min(step / max(args.iterations, 1), 1.0) if args.diffusion_progressive else 0.0 + target_ratio = (1.0 - progress) * args.diffusion_min_mask_ratio + progress * args.diffusion_max_mask_ratio + mask_tokens = max(1, min(int(round(seq_len * target_ratio)), seq_len)) + span_len = min(args.diffusion_span_len, seq_len) + num_spans = max(args.diffusion_spans_per_seq, math.ceil(mask_tokens / span_len)) + mask = torch.zeros((batch_size, seq_len), device=device, dtype=torch.bool) + + for batch_idx in range(batch_size): + tokens_left = mask_tokens + for _ in range(num_spans): + if tokens_left <= 0: + break + current_span = min(span_len, tokens_left) + max_start = seq_len - current_span + start = int(torch.randint(max_start + 1, (), device=device).item()) + mask[batch_idx, start : start + current_span] = True + tokens_left -= current_span + return mask + + +def sample_mercury_target_mask( + args: Hyperparameters, + batch_size: int, + seq_len: int, + step: int, + device: torch.device, +) -> Tensor: + if not (0.0 <= args.mercury_continuation_prob <= 1.0): + raise ValueError(f"MERCURY_CONTINUATION_PROB must be in [0, 1], got {args.mercury_continuation_prob}") + if args.mercury_task_mode not in {"mixed", "continuation", "infill", "random"}: + raise ValueError(f"Unsupported MERCURY_TASK_MODE={args.mercury_task_mode!r}") + if args.mercury_task_mode == "random": + return sample_diffusion_target_mask(args, batch_size, seq_len, step, device) + + progress = min(step / max(args.iterations, 1), 1.0) if args.diffusion_progressive else 0.0 + target_ratio = (1.0 - progress) * args.diffusion_min_mask_ratio + progress * args.diffusion_max_mask_ratio + mask_tokens = max(1, min(int(round(seq_len * target_ratio)), seq_len - 2)) + mask = torch.zeros((batch_size, seq_len), device=device, dtype=torch.bool) + + for batch_idx in range(batch_size): + if args.mercury_task_mode == "continuation": + task = "continuation" + elif args.mercury_task_mode == "infill": + task = "infill" + else: + task = "continuation" if torch.rand((), device=device).item() < args.mercury_continuation_prob else "infill" + + if task == "continuation": + mask[batch_idx, seq_len - mask_tokens :] = True + continue + + max_start = seq_len - mask_tokens - 1 + min_start = 1 + start = min_start if max_start <= min_start else int(torch.randint(min_start, max_start + 1, (), device=device).item()) + mask[batch_idx, start : start + mask_tokens] = True + return mask + + +def build_diffusion_corruption( + args: Hyperparameters, + input_ids: Tensor, + step: int, +) -> tuple[Tensor, Tensor, Tensor]: + if args.diffusion_train_mode == "mercury": + target_mask = sample_mercury_target_mask(args, input_ids.size(0), input_ids.size(1), step, input_ids.device) + else: + target_mask = sample_diffusion_target_mask(args, input_ids.size(0), input_ids.size(1), step, input_ids.device) + mask_token_mask = torch.zeros_like(target_mask) + corrupted_ids = input_ids.clone() + + if not target_mask.any(): + return corrupted_ids, target_mask, mask_token_mask + + if args.diffusion_corruption_mode == "mask": + mask_token_mask = target_mask + return corrupted_ids, target_mask, mask_token_mask + + random_ids = torch.randint(args.vocab_size, input_ids.shape, device=input_ids.device, dtype=input_ids.dtype) + random_ids = torch.where(random_ids == input_ids, (random_ids + 1) % args.vocab_size, random_ids) + + if args.diffusion_corruption_mode == "uniform": + corrupted_ids = torch.where(target_mask, random_ids, corrupted_ids) + return corrupted_ids, target_mask, mask_token_mask + + if args.diffusion_corruption_mode == "hybrid": + if not (0.0 <= args.diffusion_hybrid_mask_prob <= 1.0): + raise ValueError( + f"DIFFUSION_HYBRID_MASK_PROB must be in [0, 1], got {args.diffusion_hybrid_mask_prob}" + ) + choose_mask = torch.rand(input_ids.shape, device=input_ids.device) < args.diffusion_hybrid_mask_prob + mask_token_mask = target_mask & choose_mask + uniform_mask = target_mask & ~choose_mask + corrupted_ids = torch.where(uniform_mask, random_ids, corrupted_ids) + return corrupted_ids, target_mask, mask_token_mask + + raise ValueError(f"Unsupported DIFFUSION_CORRUPTION_MODE={args.diffusion_corruption_mode!r}") + + +def masked_cross_entropy(logits: Tensor, targets: Tensor, mask: Tensor) -> Tensor: + flat_mask = mask.reshape(-1).float() + losses = F.cross_entropy( + logits.reshape(-1, logits.size(-1)).float(), + targets.reshape(-1), + reduction="none", + ) + denom = flat_mask.sum().clamp_min(1.0) + return (losses * flat_mask).sum() / denom + + +def masked_kl_divergence(student_logits: Tensor, teacher_logits: Tensor, mask: Tensor, temperature: float) -> Tensor: + if temperature <= 0.0: + raise ValueError(f"DIFFUSION_KD_TEMPERATURE must be positive, got {temperature}") + flat_mask = mask.reshape(-1).float() + student = student_logits.reshape(-1, student_logits.size(-1)).float() + teacher = teacher_logits.reshape(-1, teacher_logits.size(-1)).float().detach() + student_log_probs = F.log_softmax(student / temperature, dim=-1) + teacher_probs = F.softmax(teacher / temperature, dim=-1) + per_token = F.kl_div(student_log_probs, teacher_probs, reduction="none").sum(dim=-1) + denom = flat_mask.sum().clamp_min(1.0) + return (per_token * flat_mask).sum() / denom * (temperature * temperature) + + +def parse_parallel_eval_steps(steps_csv: str) -> list[int]: + steps: list[int] = [] + for raw in steps_csv.split(","): + raw = raw.strip() + if not raw: + continue + value = int(raw) + if value <= 0: + raise ValueError(f"PARALLEL_EVAL_STEPS values must be positive, got {value}") + steps.append(value) + if not steps: + raise ValueError("PARALLEL_EVAL_STEPS must contain at least one positive integer") + return sorted(set(steps)) + + +@torch.no_grad() +def mercury_commit_predictions( + model: GPT, + corrupted_ids: Tensor, + target_mask: Tensor, + mask_token_mask: Tensor, + *, + commit_fraction: float, + strategy: str, +) -> tuple[Tensor, Tensor, Tensor]: + if not (0.0 <= commit_fraction <= 1.0): + raise ValueError(f"commit_fraction must be in [0, 1], got {commit_fraction}") + if strategy not in {"confidence", "random"}: + raise ValueError(f"Unsupported MERCURY_COMMIT_STRATEGY={strategy!r}") + if not target_mask.any() or commit_fraction <= 0.0: + return corrupted_ids, target_mask, mask_token_mask + + hidden = model.encode_hidden(corrupted_ids, mask_token_mask=mask_token_mask) + logits = model.token_logits(hidden) + probs = F.softmax(logits.float(), dim=-1) + pred_ids = probs.argmax(dim=-1) + pred_conf = probs.max(dim=-1).values + + refined_ids = corrupted_ids.clone() + remaining_mask = target_mask.clone() + remaining_mask_tokens = mask_token_mask.clone() + + for batch_idx in range(corrupted_ids.size(0)): + active = torch.nonzero(target_mask[batch_idx], as_tuple=False).flatten() + if active.numel() == 0: + continue + if commit_fraction >= 1.0: + chosen = active + else: + k = max(1, min(active.numel(), math.ceil(active.numel() * commit_fraction))) + if strategy == "confidence": + conf = pred_conf[batch_idx, active] + _, order = torch.topk(conf, k=k, largest=True, sorted=False) + chosen = active[order] + else: + perm = torch.randperm(active.numel(), device=active.device)[:k] + chosen = active[perm] + refined_ids[batch_idx, chosen] = pred_ids[batch_idx, chosen] + remaining_mask[batch_idx, chosen] = False + remaining_mask_tokens[batch_idx, chosen] = False + + return refined_ids, remaining_mask, remaining_mask_tokens + +# ----------------------------- +# TRANSFORMER MODULES +# ----------------------------- + +class RMSNorm(nn.Module): + def __init__(self, eps: float | None = None): + super().__init__() + self.eps = eps + + def forward(self, x: Tensor) -> Tensor: + return F.rms_norm(x, (x.size(-1),), eps=self.eps) + + +class CastedLinear(nn.Linear): + # Keep weights in fp32 for optimizer/state quality, cast at matmul time for bf16 compute. + def forward(self, x: Tensor) -> Tensor: + bias = self.bias.to(x.dtype) if self.bias is not None else None + return F.linear(x, self.weight.to(x.dtype), bias) + + +def restore_low_dim_params_to_fp32(module: nn.Module) -> None: + # Keep small/control parameters in fp32 even when the model body runs in bf16. + with torch.no_grad(): + for name, param in module.named_parameters(): + if (param.ndim < 2 or any(pattern in name for pattern in CONTROL_TENSOR_NAME_PATTERNS)) and param.dtype != torch.float32: + param.data = param.data.float() + + +class Rotary(nn.Module): + # Caches cos/sin tables per sequence length on the current device. + def __init__(self, dim: int, base: float = 10000.0): + super().__init__() + inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2, dtype=torch.float32) / dim)) + self.register_buffer("inv_freq", inv_freq, persistent=False) + self._seq_len_cached = 0 + self._cos_cached: Tensor | None = None + self._sin_cached: Tensor | None = None + + def forward(self, seq_len: int, device: torch.device, dtype: torch.dtype) -> tuple[Tensor, Tensor]: + if ( + self._cos_cached is None + or self._sin_cached is None + or self._seq_len_cached != seq_len + or self._cos_cached.device != device + ): + t = torch.arange(seq_len, device=device, dtype=self.inv_freq.dtype) + freqs = torch.outer(t, self.inv_freq.to(device)) + self._cos_cached = freqs.cos()[None, None, :, :] + self._sin_cached = freqs.sin()[None, None, :, :] + self._seq_len_cached = seq_len + return self._cos_cached.to(dtype=dtype), self._sin_cached.to(dtype=dtype) + + +def apply_rotary_emb(x: Tensor, cos: Tensor, sin: Tensor) -> Tensor: + half = x.size(-1) // 2 + x1, x2 = x[..., :half], x[..., half:] + return torch.cat((x1 * cos + x2 * sin, x1 * (-sin) + x2 * cos), dim=-1) + + +class CausalSelfAttention(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + num_kv_heads: int, + rope_base: float, + qk_gain_init: float, + ): + super().__init__() + if dim % num_heads != 0: + raise ValueError("model_dim must be divisible by num_heads") + if num_heads % num_kv_heads != 0: + raise ValueError("num_heads must be divisible by num_kv_heads") + self.num_heads = num_heads + self.num_kv_heads = num_kv_heads + self.head_dim = dim // num_heads + if self.head_dim % 2 != 0: + raise ValueError("head_dim must be even for RoPE") + kv_dim = self.num_kv_heads * self.head_dim + self.c_q = CastedLinear(dim, dim, bias=False) + self.c_k = CastedLinear(dim, kv_dim, bias=False) + self.c_v = CastedLinear(dim, kv_dim, bias=False) + self.proj = CastedLinear(dim, dim, bias=False) + self.proj._zero_init = True + self.q_gain = nn.Parameter(torch.full((num_heads,), qk_gain_init, dtype=torch.float32)) + self.rotary = Rotary(self.head_dim, base=rope_base) + + def forward(self, x: Tensor) -> Tensor: + bsz, seqlen, dim = x.shape + q = self.c_q(x).reshape(bsz, seqlen, self.num_heads, self.head_dim).transpose(1, 2) + k = self.c_k(x).reshape(bsz, seqlen, self.num_kv_heads, self.head_dim).transpose(1, 2) + v = self.c_v(x).reshape(bsz, seqlen, self.num_kv_heads, self.head_dim).transpose(1, 2) + q = F.rms_norm(q, (q.size(-1),)) + k = F.rms_norm(k, (k.size(-1),)) + cos, sin = self.rotary(seqlen, x.device, q.dtype) + q = apply_rotary_emb(q, cos, sin) + k = apply_rotary_emb(k, cos, sin) + q = q * self.q_gain.to(dtype=q.dtype)[None, :, None, None] + y = F.scaled_dot_product_attention( + q, + k, + v, + attn_mask=None, + is_causal=True, + enable_gqa=(self.num_kv_heads != self.num_heads), + ) + y = y.transpose(1, 2).contiguous().reshape(bsz, seqlen, dim) + return self.proj(y) + + +class MLP(nn.Module): + # relu^2 MLP from the original modded-nanogpt setup + def __init__(self, dim: int, mlp_mult: int): + super().__init__() + hidden = mlp_mult * dim + self.fc = CastedLinear(dim, hidden, bias=False) + self.proj = CastedLinear(hidden, dim, bias=False) + self.proj._zero_init = True + + def forward(self, x: Tensor) -> Tensor: + x = torch.relu(self.fc(x)) + return self.proj(x.square()) + + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + num_kv_heads: int, + mlp_mult: int, + rope_base: float, + qk_gain_init: float, + ): + super().__init__() + self.attn_norm = RMSNorm() + self.mlp_norm = RMSNorm() + self.attn = CausalSelfAttention(dim, num_heads, num_kv_heads, rope_base, qk_gain_init) + self.mlp = MLP(dim, mlp_mult) + self.attn_scale = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + self.mlp_scale = nn.Parameter(torch.ones(dim, dtype=torch.float32)) + self.resid_mix = nn.Parameter(torch.stack((torch.ones(dim), torch.zeros(dim))).float()) + + def forward(self, x: Tensor, x0: Tensor) -> Tensor: + mix = self.resid_mix.to(dtype=x.dtype) + x = mix[0][None, None, :] * x + mix[1][None, None, :] * x0 + attn_out = self.attn(self.attn_norm(x)) + x = x + self.attn_scale.to(dtype=x.dtype)[None, None, :] * attn_out + x = x + self.mlp_scale.to(dtype=x.dtype)[None, None, :] * self.mlp(self.mlp_norm(x)) + return x + + +class GPT(nn.Module): + def __init__( + self, + vocab_size: int, + num_layers: int, + model_dim: int, + num_heads: int, + num_kv_heads: int, + mlp_mult: int, + tie_embeddings: bool, + tied_embed_init_std: float, + logit_softcap: float, + rope_base: float, + qk_gain_init: float, + ): + super().__init__() + if logit_softcap <= 0.0: + raise ValueError(f"logit_softcap must be positive, got {logit_softcap}") + self.tie_embeddings = tie_embeddings + self.tied_embed_init_std = tied_embed_init_std + self.logit_softcap = logit_softcap + self.tok_emb = nn.Embedding(vocab_size, model_dim) + self.mask_emb = nn.Parameter(torch.zeros(model_dim, dtype=torch.float32)) + self.num_encoder_layers = num_layers // 2 + self.num_decoder_layers = num_layers - self.num_encoder_layers + self.num_skip_weights = min(self.num_encoder_layers, self.num_decoder_layers) + self.skip_weights = nn.Parameter(torch.ones(self.num_skip_weights, model_dim, dtype=torch.float32)) + self.blocks = nn.ModuleList( + [ + Block( + model_dim, + num_heads, + num_kv_heads, + mlp_mult, + rope_base, + qk_gain_init, + ) + for i in range(num_layers) + ] + ) + self.final_norm = RMSNorm() + self.lm_head = None if tie_embeddings else CastedLinear(model_dim, vocab_size, bias=False) + if self.lm_head is not None: + self.lm_head._zero_init = True + self._init_weights() + + def _init_weights(self) -> None: + if self.tie_embeddings: + nn.init.normal_(self.tok_emb.weight, mean=0.0, std=self.tied_embed_init_std) + nn.init.normal_(self.mask_emb, mean=0.0, std=self.tied_embed_init_std) + for module in self.modules(): + if isinstance(module, nn.Linear) and getattr(module, "_zero_init", False): + nn.init.zeros_(module.weight) + + def encode_hidden(self, input_ids: Tensor, mask_token_mask: Tensor | None = None) -> Tensor: + x = self.tok_emb(input_ids) + if mask_token_mask is not None: + x = torch.where(mask_token_mask[..., None], self.mask_emb.to(dtype=x.dtype)[None, None, :], x) + x = F.rms_norm(x, (x.size(-1),)) + x0 = x + skips: list[Tensor] = [] + + # First half stores skips; second half reuses them in reverse order. + for i in range(self.num_encoder_layers): + x = self.blocks[i](x, x0) + skips.append(x) + for i in range(self.num_decoder_layers): + if skips: + x = x + self.skip_weights[i].to(dtype=x.dtype)[None, None, :] * skips.pop() + x = self.blocks[self.num_encoder_layers + i](x, x0) + return self.final_norm(x) + + def token_logits(self, hidden: Tensor) -> Tensor: + x = hidden.reshape(-1, hidden.size(-1)) + if self.tie_embeddings: + logits_proj = F.linear(x, self.tok_emb.weight) + else: + if self.lm_head is None: + raise RuntimeError("lm_head is required when tie_embeddings=False") + logits_proj = self.lm_head(x) + logits = self.logit_softcap * torch.tanh(logits_proj / self.logit_softcap) + return logits.reshape(hidden.size(0), hidden.size(1), -1) + + def forward( + self, + input_ids: Tensor, + target_ids: Tensor, + diffusion_input_ids: Tensor | None = None, + diffusion_target_mask: Tensor | None = None, + diffusion_mask_token_mask: Tensor | None = None, + diffusion_train_mode: str = "aux", + diffusion_loss_weight: float = 0.0, + diffusion_kd_temperature: float = 1.0, + mercury_clean_loss_weight: float = 0.05, + return_metrics: bool = False, + ) -> Tensor | tuple[Tensor, Tensor, Tensor, Tensor]: + if diffusion_train_mode not in {"aux", "block", "consistency", "mercury"}: + raise ValueError(f"Unsupported DIFFUSION_TRAIN_MODE={diffusion_train_mode!r}") + + hidden = self.encode_hidden(input_ids) + logits = self.token_logits(hidden) + clean_loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)).float(), target_ids.reshape(-1), reduction="mean") + + diffusion_loss = clean_loss.new_zeros(()) + mask_ratio = clean_loss.new_zeros(()) + total_loss = clean_loss + if diffusion_target_mask is not None and diffusion_loss_weight > 0.0 and diffusion_train_mode == "block": + mask_present = diffusion_target_mask.float().amax() + corrupted_ids = diffusion_input_ids if diffusion_input_ids is not None else input_ids + masked_hidden = self.encode_hidden(corrupted_ids, mask_token_mask=diffusion_mask_token_mask) + masked_logits = self.token_logits(masked_hidden) + block_loss = masked_cross_entropy(masked_logits, target_ids, diffusion_target_mask) + mask_ratio = diffusion_target_mask.float().mean() + diffusion_loss = block_loss.detach() + clean_loss = torch.lerp(clean_loss.detach(), block_loss.detach(), mask_present) + total_loss = torch.lerp(total_loss, block_loss, mask_present) + elif diffusion_target_mask is not None and diffusion_loss_weight > 0.0 and diffusion_train_mode == "mercury": + mask_present = diffusion_target_mask.float().amax() + corrupted_ids = diffusion_input_ids if diffusion_input_ids is not None else input_ids + masked_hidden = self.encode_hidden(corrupted_ids, mask_token_mask=diffusion_mask_token_mask) + masked_logits = self.token_logits(masked_hidden) + diffusion_loss = masked_cross_entropy(masked_logits, input_ids, diffusion_target_mask) + mask_ratio = diffusion_target_mask.float().mean() + mercury_total_loss = diffusion_loss_weight * diffusion_loss + mercury_clean_loss_weight * clean_loss + total_loss = torch.lerp(total_loss, mercury_total_loss, mask_present) + elif diffusion_target_mask is not None and diffusion_loss_weight > 0.0 and diffusion_train_mode == "consistency": + corrupted_ids = diffusion_input_ids if diffusion_input_ids is not None else input_ids + masked_hidden = self.encode_hidden(corrupted_ids, mask_token_mask=diffusion_mask_token_mask) + masked_logits = self.token_logits(masked_hidden) + diffusion_loss = masked_kl_divergence( + masked_logits, + logits, + diffusion_target_mask, + diffusion_kd_temperature, + ) + mask_ratio = diffusion_target_mask.float().mean() + total_loss = clean_loss + diffusion_loss_weight * diffusion_loss + elif diffusion_target_mask is not None and diffusion_loss_weight > 0.0: + corrupted_ids = diffusion_input_ids if diffusion_input_ids is not None else input_ids + masked_hidden = self.encode_hidden(corrupted_ids, mask_token_mask=diffusion_mask_token_mask) + masked_logits = self.token_logits(masked_hidden) + diffusion_loss = masked_cross_entropy(masked_logits, input_ids, diffusion_target_mask) + mask_ratio = diffusion_target_mask.float().mean() + total_loss = clean_loss + diffusion_loss_weight * diffusion_loss + + # Keep mask_emb in the autograd graph even during CE-only warmup/control runs + # so DDP does not flag it as an unused parameter. + total_loss = total_loss + self.mask_emb[:1].float().sum() * 0.0 + + if return_metrics: + return total_loss, clean_loss.detach(), diffusion_loss.detach(), mask_ratio.detach() + return total_loss + + +def iterative_parallel_refine( + model: GPT, + seed_ids: Tensor, + target_mask: Tensor, + num_steps: int, +) -> Tensor: + if num_steps <= 0: + raise ValueError(f"num_steps must be positive, got {num_steps}") + + refined_ids = seed_ids.clone() + remaining_mask = target_mask.clone() + if not remaining_mask.any(): + return refined_ids + + for step_idx in range(num_steps): + hidden = model.encode_hidden(refined_ids, mask_token_mask=remaining_mask) + logits = model.token_logits(hidden) + probs = F.softmax(logits.float(), dim=-1) + pred_ids = probs.argmax(dim=-1) + pred_conf = probs.max(dim=-1).values + + steps_left = num_steps - step_idx + for batch_idx in range(refined_ids.size(0)): + active = torch.nonzero(remaining_mask[batch_idx], as_tuple=False).flatten() + if active.numel() == 0: + continue + if steps_left == 1: + chosen = active + else: + fill_count = max(1, math.ceil(active.numel() / steps_left)) + conf = pred_conf[batch_idx, active] + _, order = torch.topk(conf, k=min(fill_count, active.numel()), largest=True, sorted=False) + chosen = active[order] + refined_ids[batch_idx, chosen] = pred_ids[batch_idx, chosen] + remaining_mask[batch_idx, chosen] = False + return refined_ids + + +def sample_parallel_eval_batch( + val_tokens: Tensor, + *, + batch_size: int, + seq_len: int, + prefix_len: int, + suffix_len: int, + span_len: int, + device: torch.device, +) -> tuple[Tensor, Tensor, Tensor]: + if prefix_len + suffix_len > seq_len: + raise ValueError( + f"prefix_len + suffix_len must be <= seq_len, got {prefix_len}+{suffix_len}>{seq_len}" + ) + if prefix_len <= 0 or suffix_len <= 0 or span_len <= 0: + raise ValueError("Parallel eval lengths must be positive") + total_seqs = (val_tokens.numel() - 1) // seq_len + if total_seqs < batch_size: + raise ValueError(f"Need at least {batch_size} validation sequences, found {total_seqs}") + + seq_ids = torch.randperm(total_seqs)[:batch_size] + base = [] + for seq_id in seq_ids.tolist(): + start = seq_id * seq_len + base.append(val_tokens[start : start + seq_len].to(device=device, dtype=torch.int64, non_blocking=True)) + clean = torch.stack(base, dim=0) + + continuation_mask = torch.zeros((batch_size, seq_len), device=device, dtype=torch.bool) + continuation_mask[:, prefix_len : prefix_len + suffix_len] = True + + infill_mask = torch.zeros((batch_size, seq_len), device=device, dtype=torch.bool) + max_span_start = seq_len - span_len + min_span_start = min(prefix_len, max_span_start) + for batch_idx in range(batch_size): + span_start = int(torch.randint(min_span_start, max_span_start + 1, (), device=device).item()) + infill_mask[batch_idx, span_start : span_start + span_len] = True + + return clean, continuation_mask, infill_mask + + +def eval_parallel_metrics( + args: Hyperparameters, + model: GPT, + val_tokens: Tensor, + device: torch.device, +) -> list[str]: + batch_size = min(args.parallel_eval_examples, 32) + steps_list = parse_parallel_eval_steps(args.parallel_eval_steps) + clean, continuation_mask, infill_mask = sample_parallel_eval_batch( + val_tokens, + batch_size=batch_size, + seq_len=args.train_seq_len, + prefix_len=args.parallel_eval_prefix_len, + suffix_len=args.parallel_eval_suffix_len, + span_len=args.parallel_eval_span_len, + device=device, + ) + log_lines: list[str] = [] + + model.eval() + with torch.inference_mode(): + for steps in steps_list: + for task_name, task_mask in (("continuation", continuation_mask), ("infill", infill_mask)): + seed_ids = clean.clone() + seed_ids[task_mask] = 0 + torch.cuda.synchronize() + t0 = time.perf_counter() + refined = iterative_parallel_refine(model, seed_ids, task_mask, steps) + torch.cuda.synchronize() + elapsed = max(time.perf_counter() - t0, 1e-9) + + masked_tokens = task_mask.sum().item() + token_acc = (refined[task_mask] == clean[task_mask]).float().mean().item() + exact_seq = (refined[task_mask] == clean[task_mask]).reshape(batch_size, -1).all(dim=1).float().mean().item() + tok_per_sec = masked_tokens / elapsed + log_lines.append( + f"parallel_eval task:{task_name} steps:{steps} " + f"token_acc:{token_acc:.4f} exact_seq:{exact_seq:.4f} " + f"tokens:{masked_tokens} tok_per_sec:{tok_per_sec:.2f}" + ) + model.train() + return log_lines + + +# ----------------------------- +# TRAINING +# ----------------------------- + +def main() -> None: + global zeropower_via_newtonschulz5 + + code = Path(__file__).read_text(encoding="utf-8") + args = Hyperparameters() + if args.use_compile: + zeropower_via_newtonschulz5 = torch.compile(zeropower_via_newtonschulz5) + + # ----------------------------- + # DISTRIBUTED + CUDA SETUP + # ----------------------------- + + distributed = "RANK" in os.environ and "WORLD_SIZE" in os.environ + rank = int(os.environ.get("RANK", "0")) + world_size = int(os.environ.get("WORLD_SIZE", "1")) + local_rank = int(os.environ.get("LOCAL_RANK", "0")) + if world_size <= 0: + raise ValueError(f"WORLD_SIZE must be positive, got {world_size}") + if 8 % world_size != 0: + raise ValueError(f"WORLD_SIZE={world_size} must divide 8 so grad_accum_steps stays integral") + grad_accum_steps = 8 // world_size + grad_scale = 1.0 / grad_accum_steps + if not torch.cuda.is_available(): + raise RuntimeError("CUDA is required") + device = torch.device("cuda", local_rank) + torch.cuda.set_device(device) + if distributed: + dist.init_process_group(backend="nccl", device_id=device) + dist.barrier() + master_process = rank == 0 + + # Fast math knobs + torch.backends.cuda.matmul.allow_tf32 = True + torch.backends.cudnn.allow_tf32 = True + from torch.backends.cuda import enable_cudnn_sdp, enable_flash_sdp, enable_math_sdp, enable_mem_efficient_sdp + + enable_cudnn_sdp(False) + enable_flash_sdp(True) + enable_mem_efficient_sdp(False) + enable_math_sdp(False) + + logfile = None + if master_process: + os.makedirs("logs", exist_ok=True) + logfile = f"logs/{args.run_id}.txt" + print(logfile) + + def log0(msg: str, console: bool = True) -> None: + if not master_process: + return + if console: + print(msg) + if logfile is not None: + with open(logfile, "a", encoding="utf-8") as f: + print(msg, file=f) + + log0(code, console=False) + log0("=" * 100, console=False) + log0(f"Running Python {sys.version}", console=False) + log0(f"Running PyTorch {torch.__version__}", console=False) + log0( + subprocess.run(["nvidia-smi"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=False).stdout, + console=False, + ) + log0("=" * 100, console=False) + + # ----------------------------- + # TOKENIZER + VALIDATION METRIC SETUP + # ----------------------------- + + random.seed(args.seed) + np.random.seed(args.seed) + torch.manual_seed(args.seed) + torch.cuda.manual_seed_all(args.seed) + + dataset_dir = Path(args.data_path).resolve() + actual_train_files = len(list(dataset_dir.glob("fineweb_train_*.bin"))) + val_tokens = load_validation_tokens(args.val_files, args.train_seq_len) + (base_bytes_lut, has_leading_space_lut, is_boundary_token_lut), tokenizer_meta = load_tokenizer_luts( + tokenizer_path=args.tokenizer_path, + tokenizer_meta_path=args.tokenizer_meta_path, + vocab_size=args.vocab_size, + device=device, + validate_meta=args.tokenizer_meta_validate, + ) + meta_path = tokenizer_meta.get("meta_path") + log0( + "val_bpb:enabled " + f"tokenizer_kind={tokenizer_meta['tokenizer_kind']} " + f"tokenizer_path={tokenizer_meta['tokenizer_path']} " + f"tokenizer_source={tokenizer_meta['source']} " + f"tokenizer_meta_path={meta_path}" + ) + log0(f"train_loader:dataset:{dataset_dir.name} train_shards:{actual_train_files}") + log0(f"val_loader:shards pattern={args.val_files} tokens:{val_tokens.numel() - 1}") + + # ----------------------------- + # MODEL + OPTIMIZER SETUP + # ----------------------------- + + base_model = GPT( + vocab_size=args.vocab_size, + num_layers=args.num_layers, + model_dim=args.model_dim, + num_heads=args.num_heads, + num_kv_heads=args.num_kv_heads, + mlp_mult=args.mlp_mult, + tie_embeddings=args.tie_embeddings, + tied_embed_init_std=args.tied_embed_init_std, + logit_softcap=args.logit_softcap, + rope_base=args.rope_base, + qk_gain_init=args.qk_gain_init, + ).to(device).bfloat16() + for module in base_model.modules(): + if isinstance(module, CastedLinear): + module.float() + restore_low_dim_params_to_fp32(base_model) + compiled_model = torch.compile(base_model, dynamic=False, fullgraph=True) if args.use_compile else base_model + model: nn.Module = DDP(compiled_model, device_ids=[local_rank], broadcast_buffers=False) if distributed else compiled_model + + # Optimizer split: + # - token embedding (Adam) uses EMBED_LR + # - untied lm_head (Adam) uses HEAD_LR + # - matrix params in transformer blocks use MATRIX_LR via Muon + # - vectors/scalars use SCALAR_LR via Adam + block_named_params = list(base_model.blocks.named_parameters()) + matrix_params = [ + p + for name, p in block_named_params + if p.ndim == 2 and not any(pattern in name for pattern in CONTROL_TENSOR_NAME_PATTERNS) + ] + scalar_params = [ + p + for name, p in block_named_params + if p.ndim < 2 or any(pattern in name for pattern in CONTROL_TENSOR_NAME_PATTERNS) + ] + if base_model.skip_weights.numel() > 0: + scalar_params.append(base_model.skip_weights) + scalar_params.append(base_model.mask_emb) + token_lr = args.tied_embed_lr if args.tie_embeddings else args.embed_lr + optimizer_tok = torch.optim.Adam( + [{"params": [base_model.tok_emb.weight], "lr": token_lr, "base_lr": token_lr}], + betas=(args.beta1, args.beta2), + eps=args.adam_eps, + fused=True, + ) + optimizer_muon = Muon( + matrix_params, + lr=args.matrix_lr, + momentum=args.muon_momentum, + backend_steps=args.muon_backend_steps, + ) + for group in optimizer_muon.param_groups: + group["base_lr"] = args.matrix_lr + optimizer_scalar = torch.optim.Adam( + [{"params": scalar_params, "lr": args.scalar_lr, "base_lr": args.scalar_lr}], + betas=(args.beta1, args.beta2), + eps=args.adam_eps, + fused=True, + ) + optimizers: list[torch.optim.Optimizer] = [optimizer_tok, optimizer_muon, optimizer_scalar] + if base_model.lm_head is not None: + optimizer_head = torch.optim.Adam( + [{"params": [base_model.lm_head.weight], "lr": args.head_lr, "base_lr": args.head_lr}], + betas=(args.beta1, args.beta2), + eps=args.adam_eps, + fused=True, + ) + optimizers.insert(1, optimizer_head) + + n_params = sum(p.numel() for p in base_model.parameters()) + log0(f"model_params:{n_params}") + log0(f"world_size:{world_size} grad_accum_steps:{grad_accum_steps}") + log0("sdp_backends:cudnn=False flash=True mem_efficient=False math=False") + log0(f"attention_mode:gqa num_heads:{args.num_heads} num_kv_heads:{args.num_kv_heads}") + log0( + f"tie_embeddings:{args.tie_embeddings} embed_lr:{token_lr} " + f"head_lr:{args.head_lr if base_model.lm_head is not None else 0.0} " + f"matrix_lr:{args.matrix_lr} scalar_lr:{args.scalar_lr}" + ) + log0( + f"train_batch_tokens:{args.train_batch_tokens} train_seq_len:{args.train_seq_len} " + f"iterations:{args.iterations} warmup_steps:{args.warmup_steps} " + f"max_wallclock_seconds:{args.max_wallclock_seconds:.3f}" + ) + log0(f"seed:{args.seed}") + log0( + f"diffusion_enabled:{args.diffusion_enabled} diffusion_train_mode:{args.diffusion_train_mode} " + f"corruption_mode:{args.diffusion_corruption_mode} hybrid_mask_prob:{args.diffusion_hybrid_mask_prob:.3f} " + f"diffusion_loss_weight:{args.diffusion_loss_weight} kd_temperature:{args.diffusion_kd_temperature} " + f"span_len:{args.diffusion_span_len} spans_per_seq:{args.diffusion_spans_per_seq} " + f"mask_ratio:{args.diffusion_min_mask_ratio:.3f}->{args.diffusion_max_mask_ratio:.3f} " + f"progressive:{args.diffusion_progressive} warmstart_ce_only_steps:{args.diffusion_warmstart_ce_only_steps} " + f"warmstart_ce_only_seconds:{args.diffusion_warmstart_ce_only_seconds:.1f} " + f"warmstart_ce_only_fraction:{args.diffusion_warmstart_ce_only_fraction:.3f} " + f"use_compile:{args.use_compile}" + ) + log0( + f"mercury_task_mode:{args.mercury_task_mode} mercury_continuation_prob:{args.mercury_continuation_prob:.3f} " + f"mercury_clean_loss_weight:{args.mercury_clean_loss_weight:.3f} " + f"mercury_self_condition_prob:{args.mercury_self_condition_prob:.3f} " + f"mercury_self_condition_fraction:{args.mercury_self_condition_fraction:.3f} " + f"mercury_unroll_steps:{args.mercury_unroll_steps} " + f"mercury_first_pass_loss_weight:{args.mercury_first_pass_loss_weight:.3f} " + f"mercury_second_pass_loss_weight:{args.mercury_second_pass_loss_weight:.3f} " + f"mercury_commit_strategy:{args.mercury_commit_strategy}" + ) + log0( + f"report_parallel_metrics:{args.report_parallel_metrics} " + f"parallel_eval_examples:{args.parallel_eval_examples} " + f"parallel_eval_prefix_len:{args.parallel_eval_prefix_len} " + f"parallel_eval_suffix_len:{args.parallel_eval_suffix_len} " + f"parallel_eval_span_len:{args.parallel_eval_span_len} " + f"parallel_eval_steps:{args.parallel_eval_steps}" + ) + + # ----------------------------- + # DATA LOADER & MODEL WARMUP + # ----------------------------- + + train_loader = DistributedTokenLoader(args.train_files, rank, world_size, device) + + def zero_grad_all() -> None: + for opt in optimizers: + opt.zero_grad(set_to_none=True) + + max_wallclock_ms = 1000.0 * args.max_wallclock_seconds if args.max_wallclock_seconds > 0 else None + + def lr_mul(step: int, elapsed_ms: float) -> float: + if args.warmdown_iters <= 0: + return 1.0 + if max_wallclock_ms is None: + warmdown_start = max(args.iterations - args.warmdown_iters, 0) + return max((args.iterations - step) / max(args.warmdown_iters, 1), 0.0) if warmdown_start <= step < args.iterations else 1.0 + step_ms = elapsed_ms / max(step, 1) + warmdown_ms = args.warmdown_iters * step_ms + remaining_ms = max(max_wallclock_ms - elapsed_ms, 0.0) + return remaining_ms / max(warmdown_ms, 1e-9) if remaining_ms <= warmdown_ms else 1.0 + + def diffusion_is_active(step: int, elapsed_ms: float) -> bool: + if not args.diffusion_enabled: + return False + if step < args.diffusion_warmstart_ce_only_steps: + return False + if args.diffusion_warmstart_ce_only_seconds > 0.0 and elapsed_ms < 1000.0 * args.diffusion_warmstart_ce_only_seconds: + return False + if ( + args.diffusion_warmstart_ce_only_fraction > 0.0 + and max_wallclock_ms is not None + and elapsed_ms < args.diffusion_warmstart_ce_only_fraction * max_wallclock_ms + ): + return False + return True + + # Warmup primes the compiled forward/backward/optimizer paths, then we restore the + # initial weights/optimizer state so measured training starts from the true init. + if args.warmup_steps > 0: + initial_model_state = {name: tensor.detach().cpu().clone() for name, tensor in base_model.state_dict().items()} + initial_optimizer_states = [copy.deepcopy(opt.state_dict()) for opt in optimizers] + model.train() + for warmup_step in range(args.warmup_steps): + zero_grad_all() + for micro_step in range(grad_accum_steps): + if distributed: + model.require_backward_grad_sync = micro_step == grad_accum_steps - 1 + x, y = train_loader.next_batch(args.train_batch_tokens, args.train_seq_len, grad_accum_steps) + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + warmup_loss = model(x, y) + (warmup_loss * grad_scale).backward() + for opt in optimizers: + opt.step() + zero_grad_all() + if args.warmup_steps <= 20 or (warmup_step + 1) % 10 == 0 or warmup_step + 1 == args.warmup_steps: + log0(f"warmup_step:{warmup_step + 1}/{args.warmup_steps}") + base_model.load_state_dict(initial_model_state, strict=True) + for opt, state in zip(optimizers, initial_optimizer_states, strict=True): + opt.load_state_dict(state) + zero_grad_all() + if distributed: + model.require_backward_grad_sync = True + train_loader = DistributedTokenLoader(args.train_files, rank, world_size, device) + + # ----------------------------- + # MAIN TRAINING LOOP + # ----------------------------- + + training_time_ms = 0.0 + stop_after_step: int | None = None + torch.cuda.synchronize() + t0 = time.perf_counter() + + step = 0 + while True: + last_step = step == args.iterations or (stop_after_step is not None and step >= stop_after_step) + + should_validate = last_step or (args.val_loss_every > 0 and step % args.val_loss_every == 0) + if should_validate: + torch.cuda.synchronize() + training_time_ms += 1000.0 * (time.perf_counter() - t0) + val_loss, val_bpb = eval_val( + args, + model, + rank, + world_size, + device, + grad_accum_steps, + val_tokens, + base_bytes_lut, + has_leading_space_lut, + is_boundary_token_lut, + ) + log0( + f"step:{step}/{args.iterations} val_loss:{val_loss:.4f} val_bpb:{val_bpb:.4f} " + f"train_time:{training_time_ms:.0f}ms step_avg:{training_time_ms / max(step, 1):.2f}ms" + ) + torch.cuda.synchronize() + t0 = time.perf_counter() + + if last_step: + if stop_after_step is not None and step < args.iterations: + log0( + f"stopping_early: wallclock_cap train_time:{training_time_ms:.0f}ms " + f"step:{step}/{args.iterations}" + ) + break + + elapsed_ms = training_time_ms + 1000.0 * (time.perf_counter() - t0) + scale = lr_mul(step, elapsed_ms) + zero_grad_all() + train_loss = torch.zeros((), device=device) + train_clean_loss = torch.zeros((), device=device) + train_diffusion_loss = torch.zeros((), device=device) + train_mask_ratio = torch.zeros((), device=device) + diffusion_weight = args.diffusion_loss_weight if diffusion_is_active(step, elapsed_ms) else 0.0 + for micro_step in range(grad_accum_steps): + if distributed: + model.require_backward_grad_sync = micro_step == grad_accum_steps - 1 + x, y = train_loader.next_batch(args.train_batch_tokens, args.train_seq_len, grad_accum_steps) + diffusion_input_ids = None + diffusion_target_mask = None + diffusion_mask_token_mask = None + if diffusion_weight > 0.0: + diffusion_input_ids, diffusion_target_mask, diffusion_mask_token_mask = build_diffusion_corruption( + args, x, step + ) + if ( + args.diffusion_train_mode == "mercury" + and args.mercury_unroll_steps <= 1 + and args.mercury_self_condition_prob > 0.0 + and diffusion_target_mask is not None + and diffusion_target_mask.any() + and torch.rand((), device=device).item() < args.mercury_self_condition_prob + ): + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + with torch.no_grad(): + sc_hidden = base_model.encode_hidden( + diffusion_input_ids, + mask_token_mask=diffusion_mask_token_mask, + ) + sc_logits = base_model.token_logits(sc_hidden) + sc_pred_ids = sc_logits.argmax(dim=-1) + replace_mask = diffusion_target_mask & ( + torch.rand(diffusion_target_mask.shape, device=device) < args.mercury_self_condition_fraction + ) + diffusion_input_ids = torch.where(replace_mask, sc_pred_ids, diffusion_input_ids) + diffusion_mask_token_mask = diffusion_mask_token_mask & ~replace_mask + if ( + diffusion_weight > 0.0 + and args.diffusion_train_mode == "mercury" + and args.mercury_unroll_steps > 1 + and diffusion_target_mask is not None + and diffusion_target_mask.any() + ): + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + first_loss_raw, clean_loss, first_diffusion_loss, mask_ratio = model( + x, + y, + diffusion_input_ids=diffusion_input_ids, + diffusion_target_mask=diffusion_target_mask, + diffusion_mask_token_mask=diffusion_mask_token_mask, + diffusion_train_mode=args.diffusion_train_mode, + diffusion_loss_weight=diffusion_weight, + diffusion_kd_temperature=args.diffusion_kd_temperature, + mercury_clean_loss_weight=0.0, + return_metrics=True, + ) + clean_prior_loss = model(x, y) + refined_input_ids, remaining_target_mask, refined_mask_token_mask = mercury_commit_predictions( + base_model, + diffusion_input_ids, + diffusion_target_mask, + diffusion_mask_token_mask, + commit_fraction=args.mercury_self_condition_fraction, + strategy=args.mercury_commit_strategy, + ) + if remaining_target_mask.any(): + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + second_loss_raw, _, second_diffusion_loss, second_mask_ratio = model( + x, + y, + diffusion_input_ids=refined_input_ids, + diffusion_target_mask=remaining_target_mask, + diffusion_mask_token_mask=refined_mask_token_mask, + diffusion_train_mode=args.diffusion_train_mode, + diffusion_loss_weight=diffusion_weight, + diffusion_kd_temperature=args.diffusion_kd_temperature, + mercury_clean_loss_weight=0.0, + return_metrics=True, + ) + total_weight = ( + args.mercury_first_pass_loss_weight + + args.mercury_second_pass_loss_weight + + args.mercury_clean_loss_weight + ) + loss = ( + args.mercury_first_pass_loss_weight * first_loss_raw + + args.mercury_second_pass_loss_weight * second_loss_raw + + args.mercury_clean_loss_weight * clean_prior_loss + ) / max(total_weight, 1e-9) + diffusion_loss = ( + args.mercury_first_pass_loss_weight * first_diffusion_loss + + args.mercury_second_pass_loss_weight * second_diffusion_loss + ) / max(args.mercury_first_pass_loss_weight + args.mercury_second_pass_loss_weight, 1e-9) + mask_ratio = second_mask_ratio + else: + total_weight = args.mercury_first_pass_loss_weight + args.mercury_clean_loss_weight + loss = ( + args.mercury_first_pass_loss_weight * first_loss_raw + + args.mercury_clean_loss_weight * clean_prior_loss + ) / max(total_weight, 1e-9) + diffusion_loss = first_diffusion_loss + clean_loss = clean_prior_loss.detach() + else: + with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=True): + loss, clean_loss, diffusion_loss, mask_ratio = model( + x, + y, + diffusion_input_ids=diffusion_input_ids, + diffusion_target_mask=diffusion_target_mask, + diffusion_mask_token_mask=diffusion_mask_token_mask, + diffusion_train_mode=args.diffusion_train_mode, + diffusion_loss_weight=diffusion_weight, + diffusion_kd_temperature=args.diffusion_kd_temperature, + mercury_clean_loss_weight=args.mercury_clean_loss_weight, + return_metrics=True, + ) + train_loss += loss.detach() + train_clean_loss += clean_loss + train_diffusion_loss += diffusion_loss + train_mask_ratio += mask_ratio + (loss * grad_scale).backward() + train_loss /= grad_accum_steps + train_clean_loss /= grad_accum_steps + train_diffusion_loss /= grad_accum_steps + train_mask_ratio /= grad_accum_steps + + frac = min(step / args.muon_momentum_warmup_steps, 1.0) if args.muon_momentum_warmup_steps > 0 else 1.0 + muon_momentum = (1 - frac) * args.muon_momentum_warmup_start + frac * args.muon_momentum + for group in optimizer_muon.param_groups: + group["momentum"] = muon_momentum + + for opt in optimizers: + for group in opt.param_groups: + group["lr"] = group["base_lr"] * scale + + if args.grad_clip_norm > 0: + torch.nn.utils.clip_grad_norm_(base_model.parameters(), args.grad_clip_norm) + for opt in optimizers: + opt.step() + zero_grad_all() + + step += 1 + approx_training_time_ms = training_time_ms + 1000.0 * (time.perf_counter() - t0) + should_log_train = ( + args.train_log_every > 0 + and (step <= 10 or step % args.train_log_every == 0 or stop_after_step is not None) + ) + if should_log_train: + log0( + f"step:{step}/{args.iterations} train_loss:{train_loss.item():.4f} " + f"clean_loss:{train_clean_loss.item():.4f} diffusion_loss:{train_diffusion_loss.item():.4f} " + f"mask_ratio:{train_mask_ratio.item():.4f} " + f"train_time:{approx_training_time_ms:.0f}ms step_avg:{approx_training_time_ms / step:.2f}ms" + ) + + # Needed to sync whether we've reached the wallclock cap. + reached_cap = max_wallclock_ms is not None and approx_training_time_ms >= max_wallclock_ms + if distributed and max_wallclock_ms is not None: + reached_cap_tensor = torch.tensor(int(reached_cap), device=device) + dist.all_reduce(reached_cap_tensor, op=dist.ReduceOp.MAX) + reached_cap = bool(reached_cap_tensor.item()) + if stop_after_step is None and reached_cap: + stop_after_step = step + + log0( + f"peak memory allocated: {torch.cuda.max_memory_allocated() // 1024 // 1024} MiB " + f"reserved: {torch.cuda.max_memory_reserved() // 1024 // 1024} MiB" + ) + + # ----------------------------- + # SERIALIZATION + ROUNDTRIP VALIDATION + # ----------------------------- + # Save the raw state (useful for debugging/loading in PyTorch directly), then always produce + # the compressed int8+zlib artifact and validate the round-tripped weights. + + if master_process: + torch.save(base_model.state_dict(), "final_model.pt") + model_bytes = os.path.getsize("final_model.pt") + code_bytes = len(code.encode("utf-8")) + log0(f"Serialized model: {model_bytes} bytes") + log0(f"Code size: {code_bytes} bytes") + log0(f"Total submission size: {model_bytes + code_bytes} bytes") + + quant_obj, quant_stats = quantize_state_dict_int8(base_model.state_dict()) + quant_buf = io.BytesIO() + torch.save(quant_obj, quant_buf) + quant_raw = quant_buf.getvalue() + quant_blob = zlib.compress(quant_raw, level=9) + quant_raw_bytes = len(quant_raw) + if master_process: + with open("final_model.int8.ptz", "wb") as f: + f.write(quant_blob) + quant_file_bytes = os.path.getsize("final_model.int8.ptz") + code_bytes = len(code.encode("utf-8")) + ratio = quant_stats["baseline_tensor_bytes"] / max(quant_stats["int8_payload_bytes"], 1) + log0( + f"Serialized model int8+zlib: {quant_file_bytes} bytes " + f"(payload:{quant_stats['int8_payload_bytes']} raw_torch:{quant_raw_bytes} payload_ratio:{ratio:.2f}x)" + ) + log0(f"Total submission size int8+zlib: {quant_file_bytes + code_bytes} bytes") + + if distributed: + dist.barrier() + with open("final_model.int8.ptz", "rb") as f: + quant_blob_disk = f.read() + quant_state = torch.load(io.BytesIO(zlib.decompress(quant_blob_disk)), map_location="cpu") + base_model.load_state_dict(dequantize_state_dict_int8(quant_state), strict=True) + torch.cuda.synchronize() + t_qeval = time.perf_counter() + q_val_loss, q_val_bpb = eval_val( + args, + model, + rank, + world_size, + device, + grad_accum_steps, + val_tokens, + base_bytes_lut, + has_leading_space_lut, + is_boundary_token_lut, + ) + torch.cuda.synchronize() + log0( + f"final_int8_zlib_roundtrip val_loss:{q_val_loss:.4f} val_bpb:{q_val_bpb:.4f} " + f"eval_time:{1000.0 * (time.perf_counter() - t_qeval):.0f}ms" + ) + log0(f"final_int8_zlib_roundtrip_exact val_loss:{q_val_loss:.8f} val_bpb:{q_val_bpb:.8f}") + + if args.report_parallel_metrics: + if distributed: + dist.barrier() + if master_process: + for line in eval_parallel_metrics(args, base_model, val_tokens, device): + log0(line) + + if distributed: + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed1337.log b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed1337.log new file mode 100644 index 0000000000..317912372f --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed1337.log @@ -0,0 +1,256 @@ +running mercury_compact_mercury_hybrid35_mixsc_seed1337_20260409_195929 with python3 + DATA_PATH=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024 + TOKENIZER_PATH=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model + SCREEN_SECONDS=600 + VARIANT=mercury_hybrid35_mixsc + +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +logs/mercury_compact_mercury_hybrid35_mixsc_seed1337_20260409_195929.txt +val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model tokenizer_source=sentencepiece-fallback tokenizer_meta_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.meta.npz +train_loader:dataset:fineweb10B_sp1024 train_shards:80 +val_loader:shards pattern=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024/fineweb_val_*.bin tokens:62021632 +model_params:17060424 +world_size:8 grad_accum_steps:1 +sdp_backends:cudnn=False flash=True mem_efficient=False math=False +attention_mode:gqa num_heads:8 num_kv_heads:4 +tie_embeddings:True embed_lr:0.05 head_lr:0.0 matrix_lr:0.04 scalar_lr:0.04 +train_batch_tokens:524288 train_seq_len:1024 iterations:20000 warmup_steps:20 max_wallclock_seconds:600.000 +seed:1337 +diffusion_enabled:True diffusion_train_mode:mercury corruption_mode:hybrid hybrid_mask_prob:0.500 diffusion_loss_weight:1.0 kd_temperature:1.0 span_len:64 spans_per_seq:8 mask_ratio:0.250->0.350 progressive:True warmstart_ce_only_steps:0 warmstart_ce_only_seconds:0.0 warmstart_ce_only_fraction:0.000 use_compile:True +mercury_task_mode:mixed mercury_continuation_prob:0.750 mercury_clean_loss_weight:0.100 mercury_self_condition_prob:1.000 mercury_self_condition_fraction:0.750 mercury_unroll_steps:1 mercury_first_pass_loss_weight:0.500 mercury_second_pass_loss_weight:1.000 mercury_commit_strategy:confidence +report_parallel_metrics:True parallel_eval_examples:16 parallel_eval_prefix_len:128 parallel_eval_suffix_len:64 parallel_eval_span_len:64 parallel_eval_steps:1,2,4,8 +warmup_step:1/20 +warmup_step:2/20 +warmup_step:3/20 +warmup_step:4/20 +warmup_step:5/20 +warmup_step:6/20 +warmup_step:7/20 +warmup_step:8/20 +warmup_step:9/20 +warmup_step:10/20 +warmup_step:11/20 +warmup_step:12/20 +warmup_step:13/20 +warmup_step:14/20 +warmup_step:15/20 +warmup_step:16/20 +warmup_step:17/20 +warmup_step:18/20 +warmup_step:19/20 +warmup_step:20/20 +step:1/20000 train_loss:7.6380 clean_loss:6.9370 diffusion_loss:6.9443 mask_ratio:0.2500 train_time:46827ms step_avg:46826.59ms +step:2/20000 train_loss:21.3483 clean_loss:16.2987 diffusion_loss:19.7184 mask_ratio:0.2500 train_time:48564ms step_avg:24282.23ms +step:3/20000 train_loss:21.2896 clean_loss:16.6692 diffusion_loss:19.6226 mask_ratio:0.2500 train_time:48693ms step_avg:16231.15ms +step:4/20000 train_loss:20.9059 clean_loss:16.6155 diffusion_loss:19.2443 mask_ratio:0.2500 train_time:48839ms step_avg:12209.84ms +step:5/20000 train_loss:20.3231 clean_loss:16.3452 diffusion_loss:18.6886 mask_ratio:0.2500 train_time:48989ms step_avg:9797.80ms +step:6/20000 train_loss:19.5648 clean_loss:16.1843 diffusion_loss:17.9464 mask_ratio:0.2500 train_time:49260ms step_avg:8209.94ms +step:7/20000 train_loss:18.6702 clean_loss:15.5251 diffusion_loss:17.1177 mask_ratio:0.2500 train_time:49391ms step_avg:7055.92ms +step:8/20000 train_loss:18.9738 clean_loss:15.4367 diffusion_loss:17.4301 mask_ratio:0.2500 train_time:49535ms step_avg:6191.89ms +step:9/20000 train_loss:18.1528 clean_loss:14.5433 diffusion_loss:16.6985 mask_ratio:0.2500 train_time:49666ms step_avg:5518.46ms +step:10/20000 train_loss:16.9928 clean_loss:13.4755 diffusion_loss:15.6453 mask_ratio:0.2500 train_time:49796ms step_avg:4979.64ms +step:25/20000 train_loss:7.0924 clean_loss:6.4313 diffusion_loss:6.4493 mask_ratio:0.2500 train_time:51595ms step_avg:2063.81ms +step:50/20000 train_loss:6.7231 clean_loss:6.0352 diffusion_loss:6.1195 mask_ratio:0.2500 train_time:54574ms step_avg:1091.49ms +step:75/20000 train_loss:6.5692 clean_loss:5.8413 diffusion_loss:5.9851 mask_ratio:0.2500 train_time:57541ms step_avg:767.21ms +step:100/20000 train_loss:6.5315 clean_loss:5.1415 diffusion_loss:6.0173 mask_ratio:0.2510 train_time:60510ms step_avg:605.10ms +step:125/20000 train_loss:6.4464 clean_loss:4.4572 diffusion_loss:6.0007 mask_ratio:0.2510 train_time:63479ms step_avg:507.83ms +step:150/20000 train_loss:6.3875 clean_loss:4.1539 diffusion_loss:5.9721 mask_ratio:0.2510 train_time:66446ms step_avg:442.97ms +step:175/20000 train_loss:6.3391 clean_loss:3.8982 diffusion_loss:5.9493 mask_ratio:0.2510 train_time:69414ms step_avg:396.65ms +step:200/20000 train_loss:6.3609 clean_loss:3.6559 diffusion_loss:5.9953 mask_ratio:0.2510 train_time:72447ms step_avg:362.23ms +step:225/20000 train_loss:6.3487 clean_loss:3.6441 diffusion_loss:5.9843 mask_ratio:0.2510 train_time:75403ms step_avg:335.12ms +step:250/20000 train_loss:6.3245 clean_loss:3.5140 diffusion_loss:5.9731 mask_ratio:0.2510 train_time:78359ms step_avg:313.44ms +step:275/20000 train_loss:6.2999 clean_loss:3.4686 diffusion_loss:5.9530 mask_ratio:0.2510 train_time:81314ms step_avg:295.69ms +step:300/20000 train_loss:6.1069 clean_loss:3.2863 diffusion_loss:5.7783 mask_ratio:0.2520 train_time:84286ms step_avg:280.95ms +step:325/20000 train_loss:6.2802 clean_loss:3.3382 diffusion_loss:5.9464 mask_ratio:0.2520 train_time:87254ms step_avg:268.47ms +step:350/20000 train_loss:6.2273 clean_loss:3.3454 diffusion_loss:5.8927 mask_ratio:0.2520 train_time:90205ms step_avg:257.73ms +step:375/20000 train_loss:6.1973 clean_loss:3.2628 diffusion_loss:5.8711 mask_ratio:0.2520 train_time:93160ms step_avg:248.43ms +step:400/20000 train_loss:6.1959 clean_loss:2.9714 diffusion_loss:5.8988 mask_ratio:0.2520 train_time:96187ms step_avg:240.47ms +step:425/20000 train_loss:6.2290 clean_loss:3.1602 diffusion_loss:5.9130 mask_ratio:0.2520 train_time:99160ms step_avg:233.32ms +step:450/20000 train_loss:6.2003 clean_loss:3.1157 diffusion_loss:5.8887 mask_ratio:0.2520 train_time:102121ms step_avg:226.93ms +step:475/20000 train_loss:6.2233 clean_loss:3.0462 diffusion_loss:5.9187 mask_ratio:0.2520 train_time:105077ms step_avg:221.21ms +step:500/20000 train_loss:6.2469 clean_loss:3.0495 diffusion_loss:5.9420 mask_ratio:0.2529 train_time:108027ms step_avg:216.05ms +step:525/20000 train_loss:6.2479 clean_loss:3.0706 diffusion_loss:5.9408 mask_ratio:0.2529 train_time:110978ms step_avg:211.39ms +step:550/20000 train_loss:6.1335 clean_loss:2.9448 diffusion_loss:5.8390 mask_ratio:0.2529 train_time:113929ms step_avg:207.14ms +step:575/20000 train_loss:6.1586 clean_loss:3.0426 diffusion_loss:5.8543 mask_ratio:0.2529 train_time:116953ms step_avg:203.40ms +step:600/20000 train_loss:6.2041 clean_loss:3.0682 diffusion_loss:5.8973 mask_ratio:0.2529 train_time:119905ms step_avg:199.84ms +step:625/20000 train_loss:6.2233 clean_loss:2.9934 diffusion_loss:5.9239 mask_ratio:0.2529 train_time:122860ms step_avg:196.58ms +step:650/20000 train_loss:6.2043 clean_loss:2.8687 diffusion_loss:5.9174 mask_ratio:0.2529 train_time:125810ms step_avg:193.55ms +step:675/20000 train_loss:6.1782 clean_loss:2.8297 diffusion_loss:5.8952 mask_ratio:0.2529 train_time:128760ms step_avg:190.76ms +step:700/20000 train_loss:6.1588 clean_loss:2.9268 diffusion_loss:5.8661 mask_ratio:0.2539 train_time:131714ms step_avg:188.16ms +step:725/20000 train_loss:6.1559 clean_loss:2.9120 diffusion_loss:5.8647 mask_ratio:0.2539 train_time:134664ms step_avg:185.74ms +step:750/20000 train_loss:6.1812 clean_loss:2.7631 diffusion_loss:5.9049 mask_ratio:0.2539 train_time:137617ms step_avg:183.49ms +step:775/20000 train_loss:6.1511 clean_loss:2.8167 diffusion_loss:5.8695 mask_ratio:0.2539 train_time:140635ms step_avg:181.46ms +step:800/20000 train_loss:6.1412 clean_loss:2.7668 diffusion_loss:5.8646 mask_ratio:0.2539 train_time:143583ms step_avg:179.48ms +step:825/20000 train_loss:6.1892 clean_loss:2.8971 diffusion_loss:5.8995 mask_ratio:0.2539 train_time:146548ms step_avg:177.63ms +step:850/20000 train_loss:6.0801 clean_loss:3.1624 diffusion_loss:5.7639 mask_ratio:0.2539 train_time:149508ms step_avg:175.89ms +step:875/20000 train_loss:6.1660 clean_loss:2.8010 diffusion_loss:5.8859 mask_ratio:0.2539 train_time:152504ms step_avg:174.29ms +step:900/20000 train_loss:6.0930 clean_loss:2.8241 diffusion_loss:5.8106 mask_ratio:0.2549 train_time:155465ms step_avg:172.74ms +step:925/20000 train_loss:5.2574 clean_loss:2.8103 diffusion_loss:4.9764 mask_ratio:0.2549 train_time:158416ms step_avg:171.26ms +step:950/20000 train_loss:6.1966 clean_loss:2.8796 diffusion_loss:5.9086 mask_ratio:0.2549 train_time:161366ms step_avg:169.86ms +step:975/20000 train_loss:5.9467 clean_loss:2.6858 diffusion_loss:5.6782 mask_ratio:0.2549 train_time:164388ms step_avg:168.60ms +step:1000/20000 train_loss:6.1496 clean_loss:2.7949 diffusion_loss:5.8701 mask_ratio:0.2549 train_time:167360ms step_avg:167.36ms +step:1025/20000 train_loss:6.1575 clean_loss:2.8423 diffusion_loss:5.8732 mask_ratio:0.2549 train_time:170310ms step_avg:166.16ms +step:1050/20000 train_loss:6.1658 clean_loss:2.9206 diffusion_loss:5.8737 mask_ratio:0.2549 train_time:173265ms step_avg:165.01ms +step:1075/20000 train_loss:6.1278 clean_loss:2.6919 diffusion_loss:5.8586 mask_ratio:0.2549 train_time:176216ms step_avg:163.92ms +step:1100/20000 train_loss:6.1574 clean_loss:2.7196 diffusion_loss:5.8854 mask_ratio:0.2559 train_time:179170ms step_avg:162.88ms +step:1125/20000 train_loss:6.1751 clean_loss:2.7716 diffusion_loss:5.8980 mask_ratio:0.2559 train_time:182124ms step_avg:161.89ms +step:1150/20000 train_loss:6.1560 clean_loss:2.7045 diffusion_loss:5.8856 mask_ratio:0.2559 train_time:185141ms step_avg:160.99ms +step:1175/20000 train_loss:6.1112 clean_loss:2.6583 diffusion_loss:5.8454 mask_ratio:0.2559 train_time:188093ms step_avg:160.08ms +step:1200/20000 train_loss:6.1758 clean_loss:2.8215 diffusion_loss:5.8937 mask_ratio:0.2559 train_time:191056ms step_avg:159.21ms +step:1225/20000 train_loss:6.1118 clean_loss:2.8354 diffusion_loss:5.8283 mask_ratio:0.2559 train_time:194007ms step_avg:158.37ms +step:1250/20000 train_loss:6.1485 clean_loss:2.6277 diffusion_loss:5.8858 mask_ratio:0.2559 train_time:196959ms step_avg:157.57ms +step:1275/20000 train_loss:6.1187 clean_loss:2.7617 diffusion_loss:5.8425 mask_ratio:0.2568 train_time:199917ms step_avg:156.80ms +step:1300/20000 train_loss:6.2235 clean_loss:2.8036 diffusion_loss:5.9432 mask_ratio:0.2568 train_time:202873ms step_avg:156.06ms +step:1325/20000 train_loss:6.0831 clean_loss:2.7427 diffusion_loss:5.8088 mask_ratio:0.2568 train_time:205829ms step_avg:155.34ms +step:1350/20000 train_loss:6.1922 clean_loss:2.6988 diffusion_loss:5.9223 mask_ratio:0.2568 train_time:208848ms step_avg:154.70ms +step:1375/20000 train_loss:6.1565 clean_loss:2.7745 diffusion_loss:5.8791 mask_ratio:0.2568 train_time:211809ms step_avg:154.04ms +step:1400/20000 train_loss:6.1199 clean_loss:2.8396 diffusion_loss:5.8360 mask_ratio:0.2568 train_time:214763ms step_avg:153.40ms +step:1425/20000 train_loss:6.1649 clean_loss:2.7400 diffusion_loss:5.8909 mask_ratio:0.2568 train_time:217713ms step_avg:152.78ms +step:1450/20000 train_loss:6.0065 clean_loss:2.6898 diffusion_loss:5.7375 mask_ratio:0.2568 train_time:221609ms step_avg:152.83ms +step:1475/20000 train_loss:6.1280 clean_loss:2.7695 diffusion_loss:5.8510 mask_ratio:0.2578 train_time:224837ms step_avg:152.43ms +step:1500/20000 train_loss:6.1137 clean_loss:2.6830 diffusion_loss:5.8454 mask_ratio:0.2578 train_time:227785ms step_avg:151.86ms +step:1525/20000 train_loss:6.1800 clean_loss:2.7881 diffusion_loss:5.9012 mask_ratio:0.2578 train_time:230760ms step_avg:151.32ms +step:1550/20000 train_loss:6.0827 clean_loss:2.5876 diffusion_loss:5.8239 mask_ratio:0.2578 train_time:233788ms step_avg:150.83ms +step:1575/20000 train_loss:6.1405 clean_loss:2.6767 diffusion_loss:5.8728 mask_ratio:0.2578 train_time:237610ms step_avg:150.86ms +step:1600/20000 train_loss:6.0208 clean_loss:2.5139 diffusion_loss:5.7695 mask_ratio:0.2578 train_time:240574ms step_avg:150.36ms +step:1625/20000 train_loss:6.0899 clean_loss:2.6297 diffusion_loss:5.8270 mask_ratio:0.2578 train_time:243524ms step_avg:149.86ms +step:1650/20000 train_loss:6.1281 clean_loss:2.6403 diffusion_loss:5.8641 mask_ratio:0.2578 train_time:246481ms step_avg:149.38ms +step:1675/20000 train_loss:6.1600 clean_loss:2.6574 diffusion_loss:5.8943 mask_ratio:0.2588 train_time:249443ms step_avg:148.92ms +step:1700/20000 train_loss:6.1094 clean_loss:2.5750 diffusion_loss:5.8519 mask_ratio:0.2588 train_time:252392ms step_avg:148.47ms +step:1725/20000 train_loss:6.1482 clean_loss:2.6712 diffusion_loss:5.8811 mask_ratio:0.2588 train_time:255428ms step_avg:148.07ms +step:1750/20000 train_loss:6.1424 clean_loss:2.6452 diffusion_loss:5.8779 mask_ratio:0.2588 train_time:258397ms step_avg:147.66ms +step:1775/20000 train_loss:6.0082 clean_loss:2.6548 diffusion_loss:5.7428 mask_ratio:0.2588 train_time:261359ms step_avg:147.24ms +step:1800/20000 train_loss:6.1328 clean_loss:2.6079 diffusion_loss:5.8720 mask_ratio:0.2588 train_time:264309ms step_avg:146.84ms +step:1825/20000 train_loss:6.1647 clean_loss:2.6742 diffusion_loss:5.8973 mask_ratio:0.2588 train_time:267261ms step_avg:146.44ms +step:1850/20000 train_loss:6.1210 clean_loss:2.6883 diffusion_loss:5.8522 mask_ratio:0.2588 train_time:270212ms step_avg:146.06ms +step:1875/20000 train_loss:6.1122 clean_loss:2.6123 diffusion_loss:5.8509 mask_ratio:0.2598 train_time:273164ms step_avg:145.69ms +step:1900/20000 train_loss:6.0946 clean_loss:2.6147 diffusion_loss:5.8331 mask_ratio:0.2598 train_time:276119ms step_avg:145.33ms +step:1925/20000 train_loss:6.1190 clean_loss:2.6678 diffusion_loss:5.8522 mask_ratio:0.2598 train_time:279134ms step_avg:145.00ms +step:1950/20000 train_loss:6.0837 clean_loss:2.6176 diffusion_loss:5.8219 mask_ratio:0.2598 train_time:282085ms step_avg:144.66ms +step:1975/20000 train_loss:6.1763 clean_loss:2.6986 diffusion_loss:5.9064 mask_ratio:0.2598 train_time:285053ms step_avg:144.33ms +step:2000/20000 train_loss:6.0970 clean_loss:2.6344 diffusion_loss:5.8335 mask_ratio:0.2598 train_time:288002ms step_avg:144.00ms +step:2025/20000 train_loss:6.1491 clean_loss:2.7328 diffusion_loss:5.8758 mask_ratio:0.2598 train_time:290953ms step_avg:143.68ms +step:2050/20000 train_loss:6.1307 clean_loss:2.6445 diffusion_loss:5.8663 mask_ratio:0.2598 train_time:293901ms step_avg:143.37ms +step:2075/20000 train_loss:6.1062 clean_loss:2.5837 diffusion_loss:5.8478 mask_ratio:0.2607 train_time:296856ms step_avg:143.06ms +step:2100/20000 train_loss:6.0983 clean_loss:2.6512 diffusion_loss:5.8332 mask_ratio:0.2607 train_time:299869ms step_avg:142.79ms +step:2125/20000 train_loss:6.1228 clean_loss:2.6719 diffusion_loss:5.8556 mask_ratio:0.2607 train_time:302818ms step_avg:142.50ms +step:2150/20000 train_loss:6.0823 clean_loss:2.5880 diffusion_loss:5.8235 mask_ratio:0.2607 train_time:305778ms step_avg:142.22ms +step:2175/20000 train_loss:6.1276 clean_loss:2.5583 diffusion_loss:5.8717 mask_ratio:0.2607 train_time:308728ms step_avg:141.94ms +step:2200/20000 train_loss:5.9669 clean_loss:2.5133 diffusion_loss:5.7156 mask_ratio:0.2607 train_time:311680ms step_avg:141.67ms +step:2225/20000 train_loss:6.1325 clean_loss:2.5866 diffusion_loss:5.8738 mask_ratio:0.2607 train_time:314647ms step_avg:141.41ms +step:2250/20000 train_loss:6.1196 clean_loss:2.5691 diffusion_loss:5.8627 mask_ratio:0.2617 train_time:317599ms step_avg:141.15ms +step:2275/20000 train_loss:6.1107 clean_loss:2.6697 diffusion_loss:5.8437 mask_ratio:0.2617 train_time:320557ms step_avg:140.90ms +step:2300/20000 train_loss:6.0563 clean_loss:2.7473 diffusion_loss:5.7816 mask_ratio:0.2617 train_time:323571ms step_avg:140.68ms +step:2325/20000 train_loss:6.1050 clean_loss:2.5854 diffusion_loss:5.8465 mask_ratio:0.2617 train_time:326521ms step_avg:140.44ms +step:2350/20000 train_loss:6.1108 clean_loss:2.5787 diffusion_loss:5.8530 mask_ratio:0.2617 train_time:329473ms step_avg:140.20ms +step:2375/20000 train_loss:6.1019 clean_loss:2.5073 diffusion_loss:5.8512 mask_ratio:0.2617 train_time:332422ms step_avg:139.97ms +step:2400/20000 train_loss:6.1030 clean_loss:2.5909 diffusion_loss:5.8439 mask_ratio:0.2617 train_time:335372ms step_avg:139.74ms +step:2425/20000 train_loss:6.1369 clean_loss:2.6330 diffusion_loss:5.8736 mask_ratio:0.2617 train_time:338325ms step_avg:139.52ms +step:2450/20000 train_loss:6.0753 clean_loss:2.5949 diffusion_loss:5.8158 mask_ratio:0.2627 train_time:341276ms step_avg:139.30ms +step:2475/20000 train_loss:6.1009 clean_loss:2.5964 diffusion_loss:5.8412 mask_ratio:0.2627 train_time:344229ms step_avg:139.08ms +step:2500/20000 train_loss:6.0582 clean_loss:2.5180 diffusion_loss:5.8065 mask_ratio:0.2627 train_time:347245ms step_avg:138.90ms +step:2525/20000 train_loss:6.0514 clean_loss:2.5223 diffusion_loss:5.7992 mask_ratio:0.2627 train_time:350197ms step_avg:138.69ms +step:2550/20000 train_loss:6.0924 clean_loss:2.5189 diffusion_loss:5.8405 mask_ratio:0.2627 train_time:353198ms step_avg:138.51ms +step:2575/20000 train_loss:6.0755 clean_loss:2.5975 diffusion_loss:5.8158 mask_ratio:0.2627 train_time:356154ms step_avg:138.31ms +step:2600/20000 train_loss:6.0795 clean_loss:2.7666 diffusion_loss:5.8028 mask_ratio:0.2627 train_time:359106ms step_avg:138.12ms +step:2625/20000 train_loss:5.9968 clean_loss:2.4778 diffusion_loss:5.7490 mask_ratio:0.2627 train_time:362061ms step_avg:137.93ms +step:2650/20000 train_loss:6.0981 clean_loss:2.6419 diffusion_loss:5.8339 mask_ratio:0.2637 train_time:365012ms step_avg:137.74ms +step:2675/20000 train_loss:6.1102 clean_loss:2.6790 diffusion_loss:5.8423 mask_ratio:0.2637 train_time:368030ms step_avg:137.58ms +step:2700/20000 train_loss:6.0955 clean_loss:2.5275 diffusion_loss:5.8428 mask_ratio:0.2637 train_time:370982ms step_avg:137.40ms +step:2725/20000 train_loss:6.0111 clean_loss:2.4955 diffusion_loss:5.7616 mask_ratio:0.2637 train_time:373944ms step_avg:137.23ms +step:2750/20000 train_loss:6.1404 clean_loss:2.7271 diffusion_loss:5.8677 mask_ratio:0.2637 train_time:376895ms step_avg:137.05ms +step:2775/20000 train_loss:6.0775 clean_loss:2.5342 diffusion_loss:5.8241 mask_ratio:0.2637 train_time:379855ms step_avg:136.88ms +step:2800/20000 train_loss:6.1595 clean_loss:2.6338 diffusion_loss:5.8961 mask_ratio:0.2637 train_time:382805ms step_avg:136.72ms +step:2825/20000 train_loss:6.1234 clean_loss:2.7211 diffusion_loss:5.8513 mask_ratio:0.2637 train_time:385757ms step_avg:136.55ms +step:2850/20000 train_loss:6.1124 clean_loss:2.5897 diffusion_loss:5.8534 mask_ratio:0.2646 train_time:388707ms step_avg:136.39ms +step:2875/20000 train_loss:6.0778 clean_loss:2.6830 diffusion_loss:5.8095 mask_ratio:0.2646 train_time:391723ms step_avg:136.25ms +step:2900/20000 train_loss:6.1181 clean_loss:2.5861 diffusion_loss:5.8595 mask_ratio:0.2646 train_time:394674ms step_avg:136.09ms +step:2925/20000 train_loss:6.0893 clean_loss:2.5115 diffusion_loss:5.8382 mask_ratio:0.2646 train_time:397625ms step_avg:135.94ms +step:2950/20000 train_loss:6.0958 clean_loss:2.5890 diffusion_loss:5.8369 mask_ratio:0.2646 train_time:400576ms step_avg:135.79ms +step:2975/20000 train_loss:6.1187 clean_loss:2.5309 diffusion_loss:5.8656 mask_ratio:0.2646 train_time:403531ms step_avg:135.64ms +step:3000/20000 train_loss:6.1148 clean_loss:2.5940 diffusion_loss:5.8554 mask_ratio:0.2646 train_time:406483ms step_avg:135.49ms +step:3025/20000 train_loss:6.0710 clean_loss:2.5694 diffusion_loss:5.8141 mask_ratio:0.2646 train_time:409451ms step_avg:135.36ms +step:3050/20000 train_loss:6.0615 clean_loss:2.5443 diffusion_loss:5.8071 mask_ratio:0.2656 train_time:412400ms step_avg:135.21ms +step:3075/20000 train_loss:6.0686 clean_loss:2.6523 diffusion_loss:5.8034 mask_ratio:0.2656 train_time:415433ms step_avg:135.10ms +step:3100/20000 train_loss:6.0678 clean_loss:2.5636 diffusion_loss:5.8115 mask_ratio:0.2656 train_time:418383ms step_avg:134.96ms +step:3125/20000 train_loss:6.0578 clean_loss:2.4823 diffusion_loss:5.8095 mask_ratio:0.2656 train_time:421354ms step_avg:134.83ms +step:3150/20000 train_loss:6.0745 clean_loss:2.5336 diffusion_loss:5.8211 mask_ratio:0.2656 train_time:424302ms step_avg:134.70ms +step:3175/20000 train_loss:6.0866 clean_loss:2.6582 diffusion_loss:5.8208 mask_ratio:0.2656 train_time:427257ms step_avg:134.57ms +step:3200/20000 train_loss:6.0827 clean_loss:2.5604 diffusion_loss:5.8267 mask_ratio:0.2656 train_time:430207ms step_avg:134.44ms +step:3225/20000 train_loss:6.0947 clean_loss:2.6277 diffusion_loss:5.8319 mask_ratio:0.2666 train_time:433155ms step_avg:134.31ms +step:3250/20000 train_loss:6.0794 clean_loss:2.4748 diffusion_loss:5.8319 mask_ratio:0.2666 train_time:436177ms step_avg:134.21ms +step:3275/20000 train_loss:6.0513 clean_loss:2.6153 diffusion_loss:5.7898 mask_ratio:0.2666 train_time:439130ms step_avg:134.09ms +step:3300/20000 train_loss:6.1396 clean_loss:2.6048 diffusion_loss:5.8791 mask_ratio:0.2666 train_time:442082ms step_avg:133.96ms +step:3325/20000 train_loss:6.0969 clean_loss:2.6135 diffusion_loss:5.8355 mask_ratio:0.2666 train_time:445046ms step_avg:133.85ms +step:3350/20000 train_loss:6.0371 clean_loss:2.4601 diffusion_loss:5.7911 mask_ratio:0.2666 train_time:447997ms step_avg:133.73ms +step:3375/20000 train_loss:6.0421 clean_loss:2.4655 diffusion_loss:5.7955 mask_ratio:0.2666 train_time:450961ms step_avg:133.62ms +step:3400/20000 train_loss:6.1227 clean_loss:2.5325 diffusion_loss:5.8694 mask_ratio:0.2666 train_time:453910ms step_avg:133.50ms +step:3425/20000 train_loss:6.0134 clean_loss:2.4546 diffusion_loss:5.7679 mask_ratio:0.2676 train_time:456865ms step_avg:133.39ms +step:3450/20000 train_loss:6.0602 clean_loss:2.4610 diffusion_loss:5.8141 mask_ratio:0.2676 train_time:459885ms step_avg:133.30ms +step:3475/20000 train_loss:6.1075 clean_loss:2.4833 diffusion_loss:5.8592 mask_ratio:0.2676 train_time:462850ms step_avg:133.19ms +step:3500/20000 train_loss:6.1342 clean_loss:2.6183 diffusion_loss:5.8724 mask_ratio:0.2676 train_time:465809ms step_avg:133.09ms +step:3525/20000 train_loss:6.1683 clean_loss:2.5787 diffusion_loss:5.9104 mask_ratio:0.2676 train_time:468766ms step_avg:132.98ms +step:3550/20000 train_loss:6.0739 clean_loss:2.7296 diffusion_loss:5.8009 mask_ratio:0.2676 train_time:471717ms step_avg:132.88ms +step:3575/20000 train_loss:6.1403 clean_loss:2.5587 diffusion_loss:5.8844 mask_ratio:0.2676 train_time:474669ms step_avg:132.77ms +step:3600/20000 train_loss:6.0839 clean_loss:2.4732 diffusion_loss:5.8365 mask_ratio:0.2676 train_time:477622ms step_avg:132.67ms +step:3625/20000 train_loss:6.1102 clean_loss:2.5050 diffusion_loss:5.8597 mask_ratio:0.2686 train_time:480641ms step_avg:132.59ms +step:3650/20000 train_loss:6.1220 clean_loss:2.5440 diffusion_loss:5.8676 mask_ratio:0.2686 train_time:483590ms step_avg:132.49ms +step:3675/20000 train_loss:6.0928 clean_loss:2.6761 diffusion_loss:5.8252 mask_ratio:0.2686 train_time:486555ms step_avg:132.40ms +step:3700/20000 train_loss:6.0665 clean_loss:2.4919 diffusion_loss:5.8173 mask_ratio:0.2686 train_time:489506ms step_avg:132.30ms +step:3725/20000 train_loss:6.1094 clean_loss:2.5519 diffusion_loss:5.8542 mask_ratio:0.2686 train_time:492455ms step_avg:132.20ms +step:3750/20000 train_loss:6.0509 clean_loss:2.4723 diffusion_loss:5.8037 mask_ratio:0.2686 train_time:495406ms step_avg:132.11ms +step:3775/20000 train_loss:6.0814 clean_loss:2.4174 diffusion_loss:5.8397 mask_ratio:0.2686 train_time:498360ms step_avg:132.02ms +step:3800/20000 train_loss:6.1205 clean_loss:2.5440 diffusion_loss:5.8661 mask_ratio:0.2686 train_time:501311ms step_avg:131.92ms +step:3825/20000 train_loss:6.1131 clean_loss:2.5593 diffusion_loss:5.8572 mask_ratio:0.2695 train_time:504342ms step_avg:131.85ms +step:3850/20000 train_loss:6.0610 clean_loss:2.5070 diffusion_loss:5.8103 mask_ratio:0.2695 train_time:507294ms step_avg:131.76ms +step:3875/20000 train_loss:6.0438 clean_loss:2.4752 diffusion_loss:5.7963 mask_ratio:0.2695 train_time:510583ms step_avg:131.76ms +step:3900/20000 train_loss:5.9054 clean_loss:2.3474 diffusion_loss:5.6706 mask_ratio:0.2695 train_time:513551ms step_avg:131.68ms +step:3925/20000 train_loss:6.1063 clean_loss:2.4848 diffusion_loss:5.8578 mask_ratio:0.2695 train_time:516501ms step_avg:131.59ms +step:3950/20000 train_loss:6.0539 clean_loss:2.4462 diffusion_loss:5.8093 mask_ratio:0.2695 train_time:519458ms step_avg:131.51ms +step:3975/20000 train_loss:6.0246 clean_loss:2.4744 diffusion_loss:5.7772 mask_ratio:0.2695 train_time:522408ms step_avg:131.42ms +step:4000/20000 train_loss:6.0993 clean_loss:2.4690 diffusion_loss:5.8524 mask_ratio:0.2695 train_time:525359ms step_avg:131.34ms +step:4025/20000 train_loss:6.0770 clean_loss:2.5758 diffusion_loss:5.8195 mask_ratio:0.2705 train_time:528382ms step_avg:131.27ms +step:4050/20000 train_loss:6.0426 clean_loss:2.4321 diffusion_loss:5.7994 mask_ratio:0.2705 train_time:531345ms step_avg:131.20ms +step:4075/20000 train_loss:6.0520 clean_loss:2.4031 diffusion_loss:5.8117 mask_ratio:0.2705 train_time:534294ms step_avg:131.12ms +step:4100/20000 train_loss:6.0917 clean_loss:2.4897 diffusion_loss:5.8428 mask_ratio:0.2705 train_time:537256ms step_avg:131.04ms +step:4125/20000 train_loss:6.0626 clean_loss:2.4654 diffusion_loss:5.8161 mask_ratio:0.2705 train_time:540207ms step_avg:130.96ms +step:4150/20000 train_loss:6.1271 clean_loss:2.6153 diffusion_loss:5.8655 mask_ratio:0.2705 train_time:543160ms step_avg:130.88ms +step:4175/20000 train_loss:6.0946 clean_loss:2.4477 diffusion_loss:5.8499 mask_ratio:0.2705 train_time:546110ms step_avg:130.80ms +step:4200/20000 train_loss:6.0898 clean_loss:2.4629 diffusion_loss:5.8435 mask_ratio:0.2705 train_time:549136ms step_avg:130.75ms +step:4225/20000 train_loss:6.1016 clean_loss:2.5656 diffusion_loss:5.8451 mask_ratio:0.2715 train_time:552087ms step_avg:130.67ms +step:4250/20000 train_loss:6.0481 clean_loss:2.4305 diffusion_loss:5.8050 mask_ratio:0.2715 train_time:555060ms step_avg:130.60ms +step:4275/20000 train_loss:6.0369 clean_loss:2.3724 diffusion_loss:5.7996 mask_ratio:0.2715 train_time:558010ms step_avg:130.53ms +step:4300/20000 train_loss:5.9876 clean_loss:2.2966 diffusion_loss:5.7580 mask_ratio:0.2715 train_time:561041ms step_avg:130.47ms +step:4325/20000 train_loss:6.0885 clean_loss:2.4008 diffusion_loss:5.8484 mask_ratio:0.2715 train_time:564014ms step_avg:130.41ms +step:4350/20000 train_loss:6.0771 clean_loss:2.5049 diffusion_loss:5.8266 mask_ratio:0.2715 train_time:566964ms step_avg:130.34ms +step:4375/20000 train_loss:6.0572 clean_loss:2.4387 diffusion_loss:5.8133 mask_ratio:0.2715 train_time:569914ms step_avg:130.27ms +step:4400/20000 train_loss:6.0228 clean_loss:2.4072 diffusion_loss:5.7821 mask_ratio:0.2725 train_time:572934ms step_avg:130.21ms +step:4425/20000 train_loss:6.0569 clean_loss:2.5141 diffusion_loss:5.8055 mask_ratio:0.2725 train_time:575884ms step_avg:130.14ms +step:4450/20000 train_loss:6.0026 clean_loss:2.3485 diffusion_loss:5.7678 mask_ratio:0.2725 train_time:578851ms step_avg:130.08ms +step:4475/20000 train_loss:6.0891 clean_loss:2.4814 diffusion_loss:5.8410 mask_ratio:0.2725 train_time:581800ms step_avg:130.01ms +step:4500/20000 train_loss:6.0895 clean_loss:2.5208 diffusion_loss:5.8374 mask_ratio:0.2725 train_time:584756ms step_avg:129.95ms +step:4525/20000 train_loss:6.1133 clean_loss:2.4158 diffusion_loss:5.8717 mask_ratio:0.2725 train_time:587709ms step_avg:129.88ms +step:4550/20000 train_loss:6.0014 clean_loss:2.3456 diffusion_loss:5.7668 mask_ratio:0.2725 train_time:590659ms step_avg:129.82ms +step:4575/20000 train_loss:6.0242 clean_loss:2.4272 diffusion_loss:5.7814 mask_ratio:0.2725 train_time:593610ms step_avg:129.75ms +step:4600/20000 train_loss:6.0567 clean_loss:2.2800 diffusion_loss:5.8287 mask_ratio:0.2734 train_time:596637ms step_avg:129.70ms +step:4625/20000 train_loss:6.0870 clean_loss:2.5261 diffusion_loss:5.8344 mask_ratio:0.2734 train_time:599588ms step_avg:129.64ms +step:4628/20000 val_loss:2.4437 val_bpb:1.4473 train_time:599993ms step_avg:129.64ms +stopping_early: wallclock_cap train_time:599993ms step:4628/20000 +peak memory allocated: 20212 MiB reserved: 20348 MiB +Serialized model: 67227352 bytes +Code size: 83831 bytes +Total submission size: 67311183 bytes +Serialized model int8+zlib: 15593452 bytes (payload:17179936 raw_torch:17225367 payload_ratio:3.91x) +Total submission size int8+zlib: 15677283 bytes +final_int8_zlib_roundtrip val_loss:2.4535 val_bpb:1.4531 eval_time:1399ms +final_int8_zlib_roundtrip_exact val_loss:2.45349105 val_bpb:1.45309560 +parallel_eval task:continuation steps:1 token_acc:0.0361 exact_seq:0.0000 tokens:1024 tok_per_sec:21703.27 +parallel_eval task:infill steps:1 token_acc:0.0361 exact_seq:0.0000 tokens:1024 tok_per_sec:99681.17 +parallel_eval task:continuation steps:2 token_acc:0.0361 exact_seq:0.0000 tokens:1024 tok_per_sec:31918.15 +parallel_eval task:infill steps:2 token_acc:0.0352 exact_seq:0.0000 tokens:1024 tok_per_sec:49320.11 +parallel_eval task:continuation steps:4 token_acc:0.0342 exact_seq:0.0000 tokens:1024 tok_per_sec:24383.50 +parallel_eval task:infill steps:4 token_acc:0.0361 exact_seq:0.0000 tokens:1024 tok_per_sec:24451.59 +parallel_eval task:continuation steps:8 token_acc:0.0371 exact_seq:0.0000 tokens:1024 tok_per_sec:12154.36 +parallel_eval task:infill steps:8 token_acc:0.0381 exact_seq:0.0000 tokens:1024 tok_per_sec:12029.40 diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed2026.log b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed2026.log new file mode 100644 index 0000000000..38d17815d4 --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed2026.log @@ -0,0 +1,268 @@ +running mercury_compact_mercury_hybrid35_mixsc_seed2026_20260409_195929 with python3 + DATA_PATH=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024 + TOKENIZER_PATH=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model + SCREEN_SECONDS=600 + VARIANT=mercury_hybrid35_mixsc + +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +logs/mercury_compact_mercury_hybrid35_mixsc_seed2026_20260409_195929.txt +val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model tokenizer_source=sentencepiece-fallback tokenizer_meta_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.meta.npz +train_loader:dataset:fineweb10B_sp1024 train_shards:80 +val_loader:shards pattern=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024/fineweb_val_*.bin tokens:62021632 +model_params:17060424 +world_size:8 grad_accum_steps:1 +sdp_backends:cudnn=False flash=True mem_efficient=False math=False +attention_mode:gqa num_heads:8 num_kv_heads:4 +tie_embeddings:True embed_lr:0.05 head_lr:0.0 matrix_lr:0.04 scalar_lr:0.04 +train_batch_tokens:524288 train_seq_len:1024 iterations:20000 warmup_steps:20 max_wallclock_seconds:600.000 +seed:2026 +diffusion_enabled:True diffusion_train_mode:mercury corruption_mode:hybrid hybrid_mask_prob:0.500 diffusion_loss_weight:1.0 kd_temperature:1.0 span_len:64 spans_per_seq:8 mask_ratio:0.250->0.350 progressive:True warmstart_ce_only_steps:0 warmstart_ce_only_seconds:0.0 warmstart_ce_only_fraction:0.000 use_compile:True +mercury_task_mode:mixed mercury_continuation_prob:0.750 mercury_clean_loss_weight:0.100 mercury_self_condition_prob:1.000 mercury_self_condition_fraction:0.750 mercury_unroll_steps:1 mercury_first_pass_loss_weight:0.500 mercury_second_pass_loss_weight:1.000 mercury_commit_strategy:confidence +report_parallel_metrics:True parallel_eval_examples:16 parallel_eval_prefix_len:128 parallel_eval_suffix_len:64 parallel_eval_span_len:64 parallel_eval_steps:1,2,4,8 +warmup_step:1/20 +warmup_step:2/20 +warmup_step:3/20 +warmup_step:4/20 +warmup_step:5/20 +warmup_step:6/20 +warmup_step:7/20 +warmup_step:8/20 +warmup_step:9/20 +warmup_step:10/20 +warmup_step:11/20 +warmup_step:12/20 +warmup_step:13/20 +warmup_step:14/20 +warmup_step:15/20 +warmup_step:16/20 +warmup_step:17/20 +warmup_step:18/20 +warmup_step:19/20 +warmup_step:20/20 +step:1/20000 train_loss:7.6404 clean_loss:6.9371 diffusion_loss:6.9466 mask_ratio:0.2500 train_time:15185ms step_avg:15184.53ms +step:2/20000 train_loss:21.4173 clean_loss:16.3597 diffusion_loss:19.7813 mask_ratio:0.2500 train_time:15334ms step_avg:7667.22ms +step:3/20000 train_loss:20.9870 clean_loss:16.5613 diffusion_loss:19.3309 mask_ratio:0.2500 train_time:15468ms step_avg:5155.99ms +step:4/20000 train_loss:19.7047 clean_loss:16.1464 diffusion_loss:18.0901 mask_ratio:0.2500 train_time:15598ms step_avg:3899.60ms +step:5/20000 train_loss:19.4807 clean_loss:15.4362 diffusion_loss:17.9371 mask_ratio:0.2500 train_time:15741ms step_avg:3148.24ms +step:6/20000 train_loss:18.3217 clean_loss:14.4001 diffusion_loss:16.8816 mask_ratio:0.2500 train_time:15873ms step_avg:2645.54ms +step:7/20000 train_loss:15.9576 clean_loss:12.5594 diffusion_loss:14.7016 mask_ratio:0.2500 train_time:16004ms step_avg:2286.27ms +step:8/20000 train_loss:13.3992 clean_loss:10.9761 diffusion_loss:12.3015 mask_ratio:0.2500 train_time:16139ms step_avg:2017.41ms +step:9/20000 train_loss:10.7176 clean_loss:8.9214 diffusion_loss:9.8255 mask_ratio:0.2500 train_time:16272ms step_avg:1808.05ms +step:10/20000 train_loss:8.5405 clean_loss:7.4307 diffusion_loss:7.7975 mask_ratio:0.2500 train_time:16405ms step_avg:1640.48ms +step:25/20000 train_loss:6.8530 clean_loss:6.2280 diffusion_loss:6.2302 mask_ratio:0.2500 train_time:18199ms step_avg:727.95ms +step:50/20000 train_loss:6.6797 clean_loss:5.9883 diffusion_loss:6.0809 mask_ratio:0.2500 train_time:21173ms step_avg:423.47ms +step:75/20000 train_loss:6.5433 clean_loss:5.8919 diffusion_loss:5.9541 mask_ratio:0.2500 train_time:24122ms step_avg:321.63ms +step:100/20000 train_loss:6.5470 clean_loss:5.4585 diffusion_loss:6.0011 mask_ratio:0.2510 train_time:27071ms step_avg:270.71ms +step:125/20000 train_loss:6.5064 clean_loss:4.9930 diffusion_loss:6.0071 mask_ratio:0.2510 train_time:30036ms step_avg:240.29ms +step:150/20000 train_loss:6.4412 clean_loss:4.6718 diffusion_loss:5.9740 mask_ratio:0.2510 train_time:33008ms step_avg:220.05ms +step:175/20000 train_loss:6.3811 clean_loss:4.3525 diffusion_loss:5.9459 mask_ratio:0.2510 train_time:35981ms step_avg:205.61ms +step:200/20000 train_loss:6.3942 clean_loss:4.0050 diffusion_loss:5.9937 mask_ratio:0.2510 train_time:39022ms step_avg:195.11ms +step:225/20000 train_loss:6.3811 clean_loss:3.9227 diffusion_loss:5.9889 mask_ratio:0.2510 train_time:41978ms step_avg:186.57ms +step:250/20000 train_loss:6.3533 clean_loss:3.7574 diffusion_loss:5.9776 mask_ratio:0.2510 train_time:44949ms step_avg:179.80ms +step:275/20000 train_loss:6.3307 clean_loss:3.6770 diffusion_loss:5.9630 mask_ratio:0.2510 train_time:47909ms step_avg:174.21ms +step:300/20000 train_loss:6.1151 clean_loss:3.4709 diffusion_loss:5.7680 mask_ratio:0.2520 train_time:50864ms step_avg:169.55ms +step:325/20000 train_loss:6.3018 clean_loss:3.5410 diffusion_loss:5.9477 mask_ratio:0.2520 train_time:53812ms step_avg:165.58ms +step:350/20000 train_loss:6.2675 clean_loss:3.5550 diffusion_loss:5.9120 mask_ratio:0.2520 train_time:56761ms step_avg:162.17ms +step:375/20000 train_loss:6.2414 clean_loss:3.4706 diffusion_loss:5.8944 mask_ratio:0.2520 train_time:59708ms step_avg:159.22ms +step:400/20000 train_loss:6.2209 clean_loss:3.2199 diffusion_loss:5.8989 mask_ratio:0.2520 train_time:62717ms step_avg:156.79ms +step:425/20000 train_loss:6.2861 clean_loss:3.3904 diffusion_loss:5.9471 mask_ratio:0.2520 train_time:65692ms step_avg:154.57ms +step:450/20000 train_loss:6.2261 clean_loss:3.3664 diffusion_loss:5.8895 mask_ratio:0.2520 train_time:68669ms step_avg:152.60ms +step:475/20000 train_loss:6.2582 clean_loss:3.2532 diffusion_loss:5.9329 mask_ratio:0.2520 train_time:71625ms step_avg:150.79ms +step:500/20000 train_loss:6.2555 clean_loss:3.2177 diffusion_loss:5.9338 mask_ratio:0.2529 train_time:74574ms step_avg:149.15ms +step:525/20000 train_loss:6.2633 clean_loss:3.2105 diffusion_loss:5.9423 mask_ratio:0.2529 train_time:77523ms step_avg:147.66ms +step:550/20000 train_loss:6.1632 clean_loss:3.0841 diffusion_loss:5.8548 mask_ratio:0.2529 train_time:80472ms step_avg:146.31ms +step:575/20000 train_loss:6.1940 clean_loss:3.1662 diffusion_loss:5.8774 mask_ratio:0.2529 train_time:83477ms step_avg:145.18ms +step:600/20000 train_loss:6.2152 clean_loss:3.1626 diffusion_loss:5.8990 mask_ratio:0.2529 train_time:86423ms step_avg:144.04ms +step:625/20000 train_loss:6.2212 clean_loss:3.0836 diffusion_loss:5.9129 mask_ratio:0.2529 train_time:89370ms step_avg:142.99ms +step:650/20000 train_loss:6.2021 clean_loss:2.9422 diffusion_loss:5.9079 mask_ratio:0.2529 train_time:92320ms step_avg:142.03ms +step:675/20000 train_loss:6.2042 clean_loss:2.9099 diffusion_loss:5.9132 mask_ratio:0.2529 train_time:95268ms step_avg:141.14ms +step:700/20000 train_loss:6.1805 clean_loss:2.9972 diffusion_loss:5.8808 mask_ratio:0.2539 train_time:98215ms step_avg:140.31ms +step:725/20000 train_loss:6.1530 clean_loss:2.9835 diffusion_loss:5.8547 mask_ratio:0.2539 train_time:101165ms step_avg:139.54ms +step:750/20000 train_loss:6.1990 clean_loss:2.8248 diffusion_loss:5.9165 mask_ratio:0.2539 train_time:104112ms step_avg:138.82ms +step:775/20000 train_loss:6.1558 clean_loss:2.8914 diffusion_loss:5.8667 mask_ratio:0.2539 train_time:107146ms step_avg:138.25ms +step:800/20000 train_loss:6.1429 clean_loss:2.8331 diffusion_loss:5.8596 mask_ratio:0.2539 train_time:110120ms step_avg:137.65ms +step:825/20000 train_loss:6.2058 clean_loss:2.9660 diffusion_loss:5.9092 mask_ratio:0.2539 train_time:113092ms step_avg:137.08ms +step:850/20000 train_loss:6.0841 clean_loss:3.2111 diffusion_loss:5.7630 mask_ratio:0.2539 train_time:116084ms step_avg:136.57ms +step:875/20000 train_loss:6.1912 clean_loss:2.8621 diffusion_loss:5.9050 mask_ratio:0.2539 train_time:119071ms step_avg:136.08ms +step:900/20000 train_loss:6.0956 clean_loss:2.8792 diffusion_loss:5.8077 mask_ratio:0.2549 train_time:122045ms step_avg:135.61ms +step:925/20000 train_loss:5.4392 clean_loss:3.0330 diffusion_loss:5.1359 mask_ratio:0.2549 train_time:125013ms step_avg:135.15ms +step:950/20000 train_loss:6.1797 clean_loss:2.9263 diffusion_loss:5.8871 mask_ratio:0.2549 train_time:127958ms step_avg:134.69ms +step:975/20000 train_loss:5.9424 clean_loss:2.7452 diffusion_loss:5.6679 mask_ratio:0.2549 train_time:130964ms step_avg:134.32ms +step:1000/20000 train_loss:6.1639 clean_loss:2.8460 diffusion_loss:5.8793 mask_ratio:0.2549 train_time:133910ms step_avg:133.91ms +step:1025/20000 train_loss:6.1643 clean_loss:2.8897 diffusion_loss:5.8753 mask_ratio:0.2549 train_time:136857ms step_avg:133.52ms +step:1050/20000 train_loss:6.1890 clean_loss:2.9669 diffusion_loss:5.8923 mask_ratio:0.2549 train_time:140025ms step_avg:133.36ms +step:1075/20000 train_loss:6.1316 clean_loss:2.7255 diffusion_loss:5.8591 mask_ratio:0.2549 train_time:143517ms step_avg:133.50ms +step:1100/20000 train_loss:6.1496 clean_loss:2.7725 diffusion_loss:5.8723 mask_ratio:0.2559 train_time:146466ms step_avg:133.15ms +step:1125/20000 train_loss:6.1802 clean_loss:2.8213 diffusion_loss:5.8981 mask_ratio:0.2559 train_time:149413ms step_avg:132.81ms +step:1150/20000 train_loss:6.1687 clean_loss:2.7456 diffusion_loss:5.8941 mask_ratio:0.2559 train_time:152416ms step_avg:132.54ms +step:1175/20000 train_loss:6.1411 clean_loss:2.7038 diffusion_loss:5.8707 mask_ratio:0.2559 train_time:155363ms step_avg:132.22ms +step:1200/20000 train_loss:6.1646 clean_loss:2.8663 diffusion_loss:5.8780 mask_ratio:0.2559 train_time:158308ms step_avg:131.92ms +step:1225/20000 train_loss:6.1292 clean_loss:2.8696 diffusion_loss:5.8422 mask_ratio:0.2559 train_time:161258ms step_avg:131.64ms +step:1250/20000 train_loss:6.1344 clean_loss:2.6736 diffusion_loss:5.8671 mask_ratio:0.2559 train_time:164206ms step_avg:131.37ms +step:1275/20000 train_loss:6.1221 clean_loss:2.7924 diffusion_loss:5.8429 mask_ratio:0.2568 train_time:167158ms step_avg:131.10ms +step:1300/20000 train_loss:6.2476 clean_loss:2.8450 diffusion_loss:5.9631 mask_ratio:0.2568 train_time:170103ms step_avg:130.85ms +step:1325/20000 train_loss:6.1040 clean_loss:2.7794 diffusion_loss:5.8261 mask_ratio:0.2568 train_time:173057ms step_avg:130.61ms +step:1350/20000 train_loss:6.2047 clean_loss:2.7323 diffusion_loss:5.9315 mask_ratio:0.2568 train_time:176062ms step_avg:130.42ms +step:1375/20000 train_loss:6.1680 clean_loss:2.8189 diffusion_loss:5.8861 mask_ratio:0.2568 train_time:179008ms step_avg:130.19ms +step:1400/20000 train_loss:6.1246 clean_loss:2.8760 diffusion_loss:5.8370 mask_ratio:0.2568 train_time:181959ms step_avg:129.97ms +step:1425/20000 train_loss:6.1697 clean_loss:2.7754 diffusion_loss:5.8922 mask_ratio:0.2568 train_time:184916ms step_avg:129.77ms +step:1450/20000 train_loss:5.9889 clean_loss:2.7265 diffusion_loss:5.7162 mask_ratio:0.2568 train_time:187863ms step_avg:129.56ms +step:1475/20000 train_loss:6.1479 clean_loss:2.7938 diffusion_loss:5.8685 mask_ratio:0.2578 train_time:190810ms step_avg:129.36ms +step:1500/20000 train_loss:6.1360 clean_loss:2.7190 diffusion_loss:5.8641 mask_ratio:0.2578 train_time:193760ms step_avg:129.17ms +step:1525/20000 train_loss:6.1707 clean_loss:2.8156 diffusion_loss:5.8891 mask_ratio:0.2578 train_time:196705ms step_avg:128.99ms +step:1550/20000 train_loss:6.0790 clean_loss:2.6268 diffusion_loss:5.8163 mask_ratio:0.2578 train_time:199716ms step_avg:128.85ms +step:1575/20000 train_loss:6.1444 clean_loss:2.7162 diffusion_loss:5.8727 mask_ratio:0.2578 train_time:202660ms step_avg:128.67ms +step:1600/20000 train_loss:6.0406 clean_loss:2.5475 diffusion_loss:5.7858 mask_ratio:0.2578 train_time:205607ms step_avg:128.50ms +step:1625/20000 train_loss:6.0892 clean_loss:2.6556 diffusion_loss:5.8237 mask_ratio:0.2578 train_time:208555ms step_avg:128.34ms +step:1650/20000 train_loss:6.1246 clean_loss:2.6718 diffusion_loss:5.8574 mask_ratio:0.2578 train_time:211504ms step_avg:128.18ms +step:1675/20000 train_loss:6.1482 clean_loss:2.6940 diffusion_loss:5.8788 mask_ratio:0.2588 train_time:214472ms step_avg:128.04ms +step:1700/20000 train_loss:6.0804 clean_loss:2.6032 diffusion_loss:5.8201 mask_ratio:0.2588 train_time:217421ms step_avg:127.89ms +step:1725/20000 train_loss:6.1337 clean_loss:2.7013 diffusion_loss:5.8635 mask_ratio:0.2588 train_time:220427ms step_avg:127.78ms +step:1750/20000 train_loss:6.1514 clean_loss:2.6801 diffusion_loss:5.8833 mask_ratio:0.2588 train_time:223374ms step_avg:127.64ms +step:1775/20000 train_loss:6.0079 clean_loss:2.6927 diffusion_loss:5.7386 mask_ratio:0.2588 train_time:226320ms step_avg:127.50ms +step:1800/20000 train_loss:6.1141 clean_loss:2.6423 diffusion_loss:5.8499 mask_ratio:0.2588 train_time:229265ms step_avg:127.37ms +step:1825/20000 train_loss:6.1626 clean_loss:2.7075 diffusion_loss:5.8919 mask_ratio:0.2588 train_time:232210ms step_avg:127.24ms +step:1850/20000 train_loss:6.1365 clean_loss:2.7205 diffusion_loss:5.8644 mask_ratio:0.2588 train_time:235164ms step_avg:127.12ms +step:1875/20000 train_loss:6.1252 clean_loss:2.6411 diffusion_loss:5.8611 mask_ratio:0.2598 train_time:238109ms step_avg:126.99ms +step:1900/20000 train_loss:6.1026 clean_loss:2.6520 diffusion_loss:5.8374 mask_ratio:0.2598 train_time:241058ms step_avg:126.87ms +step:1925/20000 train_loss:6.1223 clean_loss:2.7157 diffusion_loss:5.8507 mask_ratio:0.2598 train_time:244066ms step_avg:126.79ms +step:1950/20000 train_loss:6.1010 clean_loss:2.6495 diffusion_loss:5.8361 mask_ratio:0.2598 train_time:247013ms step_avg:126.67ms +step:1975/20000 train_loss:6.1653 clean_loss:2.7328 diffusion_loss:5.8920 mask_ratio:0.2598 train_time:249960ms step_avg:126.56ms +step:2000/20000 train_loss:6.1031 clean_loss:2.6597 diffusion_loss:5.8371 mask_ratio:0.2598 train_time:252908ms step_avg:126.45ms +step:2025/20000 train_loss:6.1289 clean_loss:2.7638 diffusion_loss:5.8525 mask_ratio:0.2598 train_time:255854ms step_avg:126.35ms +step:2050/20000 train_loss:6.1304 clean_loss:2.6807 diffusion_loss:5.8623 mask_ratio:0.2598 train_time:258805ms step_avg:126.25ms +step:2075/20000 train_loss:6.1150 clean_loss:2.6081 diffusion_loss:5.8542 mask_ratio:0.2607 train_time:261758ms step_avg:126.15ms +step:2100/20000 train_loss:6.1380 clean_loss:2.6797 diffusion_loss:5.8701 mask_ratio:0.2607 train_time:264759ms step_avg:126.08ms +step:2125/20000 train_loss:6.1043 clean_loss:2.7092 diffusion_loss:5.8334 mask_ratio:0.2607 train_time:267704ms step_avg:125.98ms +step:2150/20000 train_loss:6.0845 clean_loss:2.6126 diffusion_loss:5.8232 mask_ratio:0.2607 train_time:270655ms step_avg:125.89ms +step:2175/20000 train_loss:6.1366 clean_loss:2.5929 diffusion_loss:5.8773 mask_ratio:0.2607 train_time:273602ms step_avg:125.79ms +step:2200/20000 train_loss:5.9594 clean_loss:2.5410 diffusion_loss:5.7053 mask_ratio:0.2607 train_time:276558ms step_avg:125.71ms +step:2225/20000 train_loss:6.1467 clean_loss:2.6159 diffusion_loss:5.8851 mask_ratio:0.2607 train_time:279502ms step_avg:125.62ms +step:2250/20000 train_loss:6.0994 clean_loss:2.5966 diffusion_loss:5.8397 mask_ratio:0.2617 train_time:282458ms step_avg:125.54ms +step:2275/20000 train_loss:6.0888 clean_loss:2.6975 diffusion_loss:5.8191 mask_ratio:0.2617 train_time:285404ms step_avg:125.45ms +step:2300/20000 train_loss:6.0494 clean_loss:2.7713 diffusion_loss:5.7723 mask_ratio:0.2617 train_time:288459ms step_avg:125.42ms +step:2325/20000 train_loss:6.1030 clean_loss:2.6178 diffusion_loss:5.8412 mask_ratio:0.2617 train_time:291412ms step_avg:125.34ms +step:2350/20000 train_loss:6.1241 clean_loss:2.6066 diffusion_loss:5.8634 mask_ratio:0.2617 train_time:294366ms step_avg:125.26ms +step:2375/20000 train_loss:6.1064 clean_loss:2.5312 diffusion_loss:5.8533 mask_ratio:0.2617 train_time:297322ms step_avg:125.19ms +step:2400/20000 train_loss:6.1096 clean_loss:2.6189 diffusion_loss:5.8477 mask_ratio:0.2617 train_time:300279ms step_avg:125.12ms +step:2425/20000 train_loss:6.1295 clean_loss:2.6561 diffusion_loss:5.8639 mask_ratio:0.2617 train_time:303237ms step_avg:125.05ms +step:2450/20000 train_loss:6.0502 clean_loss:2.6119 diffusion_loss:5.7890 mask_ratio:0.2627 train_time:306190ms step_avg:124.98ms +step:2475/20000 train_loss:6.0917 clean_loss:2.6155 diffusion_loss:5.8301 mask_ratio:0.2627 train_time:309163ms step_avg:124.91ms +step:2500/20000 train_loss:6.0732 clean_loss:2.5456 diffusion_loss:5.8186 mask_ratio:0.2627 train_time:312183ms step_avg:124.87ms +step:2525/20000 train_loss:6.0489 clean_loss:2.5452 diffusion_loss:5.7944 mask_ratio:0.2627 train_time:315148ms step_avg:124.81ms +step:2550/20000 train_loss:6.1042 clean_loss:2.5361 diffusion_loss:5.8506 mask_ratio:0.2627 train_time:318104ms step_avg:124.75ms +step:2575/20000 train_loss:6.0873 clean_loss:2.6250 diffusion_loss:5.8248 mask_ratio:0.2627 train_time:321055ms step_avg:124.68ms +step:2600/20000 train_loss:6.0807 clean_loss:2.7895 diffusion_loss:5.8018 mask_ratio:0.2627 train_time:324003ms step_avg:124.62ms +step:2625/20000 train_loss:5.9979 clean_loss:2.4986 diffusion_loss:5.7480 mask_ratio:0.2627 train_time:326959ms step_avg:124.56ms +step:2650/20000 train_loss:6.0820 clean_loss:2.6617 diffusion_loss:5.8158 mask_ratio:0.2637 train_time:329905ms step_avg:124.49ms +step:2675/20000 train_loss:6.1155 clean_loss:2.7072 diffusion_loss:5.8448 mask_ratio:0.2637 train_time:332932ms step_avg:124.46ms +step:2700/20000 train_loss:6.0973 clean_loss:2.5496 diffusion_loss:5.8424 mask_ratio:0.2637 train_time:335879ms step_avg:124.40ms +step:2725/20000 train_loss:6.0138 clean_loss:2.5171 diffusion_loss:5.7621 mask_ratio:0.2637 train_time:338824ms step_avg:124.34ms +step:2750/20000 train_loss:6.1389 clean_loss:2.7488 diffusion_loss:5.8641 mask_ratio:0.2637 train_time:341771ms step_avg:124.28ms +step:2775/20000 train_loss:6.0732 clean_loss:2.5664 diffusion_loss:5.8165 mask_ratio:0.2637 train_time:344715ms step_avg:124.22ms +step:2800/20000 train_loss:6.1579 clean_loss:2.6566 diffusion_loss:5.8923 mask_ratio:0.2637 train_time:347663ms step_avg:124.17ms +step:2825/20000 train_loss:6.1319 clean_loss:2.7491 diffusion_loss:5.8570 mask_ratio:0.2637 train_time:350611ms step_avg:124.11ms +step:2850/20000 train_loss:6.1208 clean_loss:2.6116 diffusion_loss:5.8596 mask_ratio:0.2646 train_time:353558ms step_avg:124.06ms +step:2875/20000 train_loss:6.0795 clean_loss:2.7099 diffusion_loss:5.8085 mask_ratio:0.2646 train_time:356577ms step_avg:124.03ms +step:2900/20000 train_loss:6.1425 clean_loss:2.5920 diffusion_loss:5.8833 mask_ratio:0.2646 train_time:359523ms step_avg:123.97ms +step:2925/20000 train_loss:6.0852 clean_loss:2.5295 diffusion_loss:5.8322 mask_ratio:0.2646 train_time:362468ms step_avg:123.92ms +step:2950/20000 train_loss:6.1025 clean_loss:2.6109 diffusion_loss:5.8414 mask_ratio:0.2646 train_time:365417ms step_avg:123.87ms +step:2975/20000 train_loss:6.1180 clean_loss:2.5537 diffusion_loss:5.8626 mask_ratio:0.2646 train_time:368363ms step_avg:123.82ms +step:3000/20000 train_loss:6.1049 clean_loss:2.6141 diffusion_loss:5.8434 mask_ratio:0.2646 train_time:371309ms step_avg:123.77ms +step:3025/20000 train_loss:6.0797 clean_loss:2.5895 diffusion_loss:5.8208 mask_ratio:0.2646 train_time:374259ms step_avg:123.72ms +step:3050/20000 train_loss:6.0854 clean_loss:2.5621 diffusion_loss:5.8292 mask_ratio:0.2656 train_time:377204ms step_avg:123.67ms +step:3075/20000 train_loss:6.0843 clean_loss:2.6724 diffusion_loss:5.8170 mask_ratio:0.2656 train_time:380215ms step_avg:123.65ms +step:3100/20000 train_loss:6.0761 clean_loss:2.5766 diffusion_loss:5.8185 mask_ratio:0.2656 train_time:383161ms step_avg:123.60ms +step:3125/20000 train_loss:6.0493 clean_loss:2.5103 diffusion_loss:5.7983 mask_ratio:0.2656 train_time:386108ms step_avg:123.55ms +step:3150/20000 train_loss:6.0826 clean_loss:2.5506 diffusion_loss:5.8275 mask_ratio:0.2656 train_time:389056ms step_avg:123.51ms +step:3175/20000 train_loss:6.0870 clean_loss:2.6839 diffusion_loss:5.8186 mask_ratio:0.2656 train_time:392005ms step_avg:123.47ms +step:3200/20000 train_loss:6.0826 clean_loss:2.5773 diffusion_loss:5.8249 mask_ratio:0.2656 train_time:394960ms step_avg:123.43ms +step:3225/20000 train_loss:6.0892 clean_loss:2.6556 diffusion_loss:5.8237 mask_ratio:0.2666 train_time:397905ms step_avg:123.38ms +step:3250/20000 train_loss:6.0856 clean_loss:2.5016 diffusion_loss:5.8355 mask_ratio:0.2666 train_time:400919ms step_avg:123.36ms +step:3275/20000 train_loss:6.0651 clean_loss:2.6356 diffusion_loss:5.8015 mask_ratio:0.2666 train_time:403867ms step_avg:123.32ms +step:3300/20000 train_loss:6.1281 clean_loss:2.6331 diffusion_loss:5.8648 mask_ratio:0.2666 train_time:406814ms step_avg:123.28ms +step:3325/20000 train_loss:6.1041 clean_loss:2.6417 diffusion_loss:5.8399 mask_ratio:0.2666 train_time:409789ms step_avg:123.24ms +step:3350/20000 train_loss:6.0407 clean_loss:2.4871 diffusion_loss:5.7920 mask_ratio:0.2666 train_time:412829ms step_avg:123.23ms +step:3375/20000 train_loss:6.0310 clean_loss:2.4897 diffusion_loss:5.7821 mask_ratio:0.2666 train_time:415777ms step_avg:123.19ms +step:3400/20000 train_loss:6.0922 clean_loss:2.5572 diffusion_loss:5.8365 mask_ratio:0.2666 train_time:418725ms step_avg:123.15ms +step:3425/20000 train_loss:6.0045 clean_loss:2.4788 diffusion_loss:5.7566 mask_ratio:0.2676 train_time:421669ms step_avg:123.12ms +step:3450/20000 train_loss:6.0552 clean_loss:2.4876 diffusion_loss:5.8064 mask_ratio:0.2676 train_time:424675ms step_avg:123.09ms +step:3475/20000 train_loss:6.1252 clean_loss:2.5139 diffusion_loss:5.8738 mask_ratio:0.2676 train_time:427623ms step_avg:123.06ms +step:3500/20000 train_loss:6.1365 clean_loss:2.6503 diffusion_loss:5.8714 mask_ratio:0.2676 train_time:430572ms step_avg:123.02ms +step:3525/20000 train_loss:6.1114 clean_loss:2.6157 diffusion_loss:5.8498 mask_ratio:0.2676 train_time:433518ms step_avg:122.98ms +step:3550/20000 train_loss:6.0893 clean_loss:2.7648 diffusion_loss:5.8128 mask_ratio:0.2676 train_time:436464ms step_avg:122.95ms +step:3575/20000 train_loss:6.1505 clean_loss:2.5939 diffusion_loss:5.8911 mask_ratio:0.2676 train_time:439413ms step_avg:122.91ms +step:3600/20000 train_loss:6.1044 clean_loss:2.5098 diffusion_loss:5.8535 mask_ratio:0.2676 train_time:442362ms step_avg:122.88ms +step:3625/20000 train_loss:6.1354 clean_loss:2.5414 diffusion_loss:5.8813 mask_ratio:0.2686 train_time:445381ms step_avg:122.86ms +step:3650/20000 train_loss:6.1395 clean_loss:2.5837 diffusion_loss:5.8811 mask_ratio:0.2686 train_time:448326ms step_avg:122.83ms +step:3675/20000 train_loss:6.0907 clean_loss:2.7118 diffusion_loss:5.8195 mask_ratio:0.2686 train_time:451272ms step_avg:122.80ms +step:3700/20000 train_loss:6.0619 clean_loss:2.5353 diffusion_loss:5.8084 mask_ratio:0.2686 train_time:454218ms step_avg:122.76ms +step:3725/20000 train_loss:6.1112 clean_loss:2.5858 diffusion_loss:5.8526 mask_ratio:0.2686 train_time:457165ms step_avg:122.73ms +step:3750/20000 train_loss:6.0688 clean_loss:2.5176 diffusion_loss:5.8170 mask_ratio:0.2686 train_time:460111ms step_avg:122.70ms +step:3775/20000 train_loss:6.0770 clean_loss:2.4622 diffusion_loss:5.8307 mask_ratio:0.2686 train_time:463058ms step_avg:122.66ms +step:3800/20000 train_loss:6.1099 clean_loss:2.5882 diffusion_loss:5.8511 mask_ratio:0.2686 train_time:466011ms step_avg:122.63ms +step:3825/20000 train_loss:6.1325 clean_loss:2.5928 diffusion_loss:5.8732 mask_ratio:0.2695 train_time:469015ms step_avg:122.62ms +step:3850/20000 train_loss:6.0919 clean_loss:2.5526 diffusion_loss:5.8366 mask_ratio:0.2695 train_time:471963ms step_avg:122.59ms +step:3875/20000 train_loss:6.0547 clean_loss:2.5246 diffusion_loss:5.8022 mask_ratio:0.2695 train_time:474908ms step_avg:122.56ms +step:3900/20000 train_loss:5.9270 clean_loss:2.3949 diffusion_loss:5.6875 mask_ratio:0.2695 train_time:477864ms step_avg:122.53ms +step:3925/20000 train_loss:6.1050 clean_loss:2.5368 diffusion_loss:5.8513 mask_ratio:0.2695 train_time:480809ms step_avg:122.50ms +step:3950/20000 train_loss:6.0728 clean_loss:2.4905 diffusion_loss:5.8238 mask_ratio:0.2695 train_time:483758ms step_avg:122.47ms +step:3975/20000 train_loss:6.0432 clean_loss:2.5194 diffusion_loss:5.7912 mask_ratio:0.2695 train_time:486705ms step_avg:122.44ms +step:4000/20000 train_loss:6.1079 clean_loss:2.5133 diffusion_loss:5.8565 mask_ratio:0.2695 train_time:489658ms step_avg:122.41ms +step:4025/20000 train_loss:6.0852 clean_loss:2.6139 diffusion_loss:5.8238 mask_ratio:0.2705 train_time:493005ms step_avg:122.49ms +step:4050/20000 train_loss:6.0686 clean_loss:2.4753 diffusion_loss:5.8210 mask_ratio:0.2705 train_time:496431ms step_avg:122.58ms +step:4075/20000 train_loss:6.0484 clean_loss:2.4469 diffusion_loss:5.8037 mask_ratio:0.2705 train_time:499376ms step_avg:122.55ms +step:4100/20000 train_loss:6.1326 clean_loss:2.5350 diffusion_loss:5.8791 mask_ratio:0.2705 train_time:502322ms step_avg:122.52ms +step:4125/20000 train_loss:6.0813 clean_loss:2.5074 diffusion_loss:5.8306 mask_ratio:0.2705 train_time:505268ms step_avg:122.49ms +step:4150/20000 train_loss:6.1326 clean_loss:2.6558 diffusion_loss:5.8671 mask_ratio:0.2705 train_time:508216ms step_avg:122.46ms +step:4175/20000 train_loss:6.0905 clean_loss:2.4873 diffusion_loss:5.8418 mask_ratio:0.2705 train_time:511176ms step_avg:122.44ms +step:4200/20000 train_loss:6.0972 clean_loss:2.5083 diffusion_loss:5.8463 mask_ratio:0.2705 train_time:514198ms step_avg:122.43ms +step:4225/20000 train_loss:6.1184 clean_loss:2.5978 diffusion_loss:5.8587 mask_ratio:0.2715 train_time:517173ms step_avg:122.41ms +step:4250/20000 train_loss:6.0489 clean_loss:2.4704 diffusion_loss:5.8018 mask_ratio:0.2715 train_time:520117ms step_avg:122.38ms +step:4275/20000 train_loss:6.0226 clean_loss:2.4103 diffusion_loss:5.7816 mask_ratio:0.2715 train_time:523065ms step_avg:122.35ms +step:4300/20000 train_loss:6.0240 clean_loss:2.3357 diffusion_loss:5.7905 mask_ratio:0.2715 train_time:526010ms step_avg:122.33ms +step:4325/20000 train_loss:6.0912 clean_loss:2.4381 diffusion_loss:5.8474 mask_ratio:0.2715 train_time:528962ms step_avg:122.30ms +step:4350/20000 train_loss:6.0970 clean_loss:2.5498 diffusion_loss:5.8420 mask_ratio:0.2715 train_time:531908ms step_avg:122.28ms +step:4375/20000 train_loss:6.0590 clean_loss:2.4703 diffusion_loss:5.8119 mask_ratio:0.2715 train_time:534857ms step_avg:122.25ms +step:4400/20000 train_loss:6.0438 clean_loss:2.4538 diffusion_loss:5.7984 mask_ratio:0.2725 train_time:537858ms step_avg:122.24ms +step:4425/20000 train_loss:6.0935 clean_loss:2.5385 diffusion_loss:5.8396 mask_ratio:0.2725 train_time:540807ms step_avg:122.22ms +step:4450/20000 train_loss:6.0185 clean_loss:2.3816 diffusion_loss:5.7804 mask_ratio:0.2725 train_time:543756ms step_avg:122.19ms +step:4475/20000 train_loss:6.0950 clean_loss:2.5135 diffusion_loss:5.8437 mask_ratio:0.2725 train_time:546701ms step_avg:122.17ms +step:4500/20000 train_loss:6.0993 clean_loss:2.5567 diffusion_loss:5.8436 mask_ratio:0.2725 train_time:549659ms step_avg:122.15ms +step:4525/20000 train_loss:6.1029 clean_loss:2.4503 diffusion_loss:5.8579 mask_ratio:0.2725 train_time:552604ms step_avg:122.12ms +step:4550/20000 train_loss:6.0130 clean_loss:2.3792 diffusion_loss:5.7751 mask_ratio:0.2725 train_time:555561ms step_avg:122.10ms +step:4575/20000 train_loss:6.0603 clean_loss:2.4567 diffusion_loss:5.8146 mask_ratio:0.2725 train_time:558504ms step_avg:122.08ms +step:4600/20000 train_loss:6.0480 clean_loss:2.3168 diffusion_loss:5.8163 mask_ratio:0.2734 train_time:561514ms step_avg:122.07ms +step:4625/20000 train_loss:6.0790 clean_loss:2.5533 diffusion_loss:5.8237 mask_ratio:0.2734 train_time:564463ms step_avg:122.05ms +step:4650/20000 train_loss:6.1029 clean_loss:2.3849 diffusion_loss:5.8644 mask_ratio:0.2734 train_time:567426ms step_avg:122.03ms +step:4675/20000 train_loss:6.0926 clean_loss:2.3924 diffusion_loss:5.8533 mask_ratio:0.2734 train_time:570376ms step_avg:122.01ms +step:4700/20000 train_loss:6.0964 clean_loss:2.5767 diffusion_loss:5.8387 mask_ratio:0.2734 train_time:573329ms step_avg:121.98ms +step:4725/20000 train_loss:6.0817 clean_loss:2.4927 diffusion_loss:5.8325 mask_ratio:0.2734 train_time:576273ms step_avg:121.96ms +step:4750/20000 train_loss:6.0438 clean_loss:2.2983 diffusion_loss:5.8140 mask_ratio:0.2734 train_time:579221ms step_avg:121.94ms +step:4775/20000 train_loss:6.0899 clean_loss:2.3859 diffusion_loss:5.8513 mask_ratio:0.2734 train_time:582222ms step_avg:121.93ms +step:4800/20000 train_loss:6.0906 clean_loss:2.5541 diffusion_loss:5.8352 mask_ratio:0.2744 train_time:585170ms step_avg:121.91ms +step:4825/20000 train_loss:6.0861 clean_loss:2.5435 diffusion_loss:5.8318 mask_ratio:0.2744 train_time:588114ms step_avg:121.89ms +step:4850/20000 train_loss:6.0720 clean_loss:2.4567 diffusion_loss:5.8263 mask_ratio:0.2744 train_time:591060ms step_avg:121.87ms +step:4875/20000 train_loss:6.0250 clean_loss:2.3449 diffusion_loss:5.7905 mask_ratio:0.2744 train_time:594007ms step_avg:121.85ms +step:4900/20000 train_loss:6.0922 clean_loss:2.4561 diffusion_loss:5.8466 mask_ratio:0.2744 train_time:596956ms step_avg:121.83ms +step:4925/20000 train_loss:6.1047 clean_loss:2.4257 diffusion_loss:5.8621 mask_ratio:0.2744 train_time:599903ms step_avg:121.81ms +step:4926/20000 val_loss:2.4479 val_bpb:1.4498 train_time:600064ms step_avg:121.82ms +stopping_early: wallclock_cap train_time:600064ms step:4926/20000 +peak memory allocated: 20212 MiB reserved: 20222 MiB +Serialized model: 67227352 bytes +Code size: 83831 bytes +Total submission size: 67311183 bytes +Serialized model int8+zlib: 15417107 bytes (payload:17179936 raw_torch:17225367 payload_ratio:3.91x) +Total submission size int8+zlib: 15500938 bytes +final_int8_zlib_roundtrip val_loss:2.4580 val_bpb:1.4558 eval_time:1398ms +final_int8_zlib_roundtrip_exact val_loss:2.45803749 val_bpb:1.45578825 +parallel_eval task:continuation steps:1 token_acc:0.0371 exact_seq:0.0000 tokens:1024 tok_per_sec:24882.73 +parallel_eval task:infill steps:1 token_acc:0.0371 exact_seq:0.0000 tokens:1024 tok_per_sec:107882.31 +parallel_eval task:continuation steps:2 token_acc:0.0371 exact_seq:0.0000 tokens:1024 tok_per_sec:39428.68 +parallel_eval task:infill steps:2 token_acc:0.0381 exact_seq:0.0000 tokens:1024 tok_per_sec:54150.42 +parallel_eval task:continuation steps:4 token_acc:0.0391 exact_seq:0.0000 tokens:1024 tok_per_sec:26892.63 +parallel_eval task:infill steps:4 token_acc:0.0439 exact_seq:0.0000 tokens:1024 tok_per_sec:27011.21 +parallel_eval task:continuation steps:8 token_acc:0.0381 exact_seq:0.0000 tokens:1024 tok_per_sec:13480.45 +parallel_eval task:infill steps:8 token_acc:0.0420 exact_seq:0.0000 tokens:1024 tok_per_sec:13433.94 diff --git a/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed42.log b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed42.log new file mode 100644 index 0000000000..b919a0cf6f --- /dev/null +++ b/records/track_non_record_16mb/2026-04-11_MercuryInRetrograde/train_seed42.log @@ -0,0 +1,267 @@ +running mercury_compact_mercury_hybrid35_mixsc_seed42_20260409_195929 with python3 + DATA_PATH=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024 + TOKENIZER_PATH=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model + SCREEN_SECONDS=600 + VARIANT=mercury_hybrid35_mixsc + +***************************************** +Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +***************************************** +logs/mercury_compact_mercury_hybrid35_mixsc_seed42_20260409_195929.txt +val_bpb:enabled tokenizer_kind=sentencepiece tokenizer_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.model tokenizer_source=sentencepiece-fallback tokenizer_meta_path=/tmp/parameter-golf-data/tokenizers/fineweb_1024_bpe.meta.npz +train_loader:dataset:fineweb10B_sp1024 train_shards:80 +val_loader:shards pattern=/tmp/parameter-golf-data/datasets/fineweb10B_sp1024/fineweb_val_*.bin tokens:62021632 +model_params:17060424 +world_size:8 grad_accum_steps:1 +sdp_backends:cudnn=False flash=True mem_efficient=False math=False +attention_mode:gqa num_heads:8 num_kv_heads:4 +tie_embeddings:True embed_lr:0.05 head_lr:0.0 matrix_lr:0.04 scalar_lr:0.04 +train_batch_tokens:524288 train_seq_len:1024 iterations:20000 warmup_steps:20 max_wallclock_seconds:600.000 +seed:42 +diffusion_enabled:True diffusion_train_mode:mercury corruption_mode:hybrid hybrid_mask_prob:0.500 diffusion_loss_weight:1.0 kd_temperature:1.0 span_len:64 spans_per_seq:8 mask_ratio:0.250->0.350 progressive:True warmstart_ce_only_steps:0 warmstart_ce_only_seconds:0.0 warmstart_ce_only_fraction:0.000 use_compile:True +mercury_task_mode:mixed mercury_continuation_prob:0.750 mercury_clean_loss_weight:0.100 mercury_self_condition_prob:1.000 mercury_self_condition_fraction:0.750 mercury_unroll_steps:1 mercury_first_pass_loss_weight:0.500 mercury_second_pass_loss_weight:1.000 mercury_commit_strategy:confidence +report_parallel_metrics:True parallel_eval_examples:16 parallel_eval_prefix_len:128 parallel_eval_suffix_len:64 parallel_eval_span_len:64 parallel_eval_steps:1,2,4,8 +warmup_step:1/20 +warmup_step:2/20 +warmup_step:3/20 +warmup_step:4/20 +warmup_step:5/20 +warmup_step:6/20 +warmup_step:7/20 +warmup_step:8/20 +warmup_step:9/20 +warmup_step:10/20 +warmup_step:11/20 +warmup_step:12/20 +warmup_step:13/20 +warmup_step:14/20 +warmup_step:15/20 +warmup_step:16/20 +warmup_step:17/20 +warmup_step:18/20 +warmup_step:19/20 +warmup_step:20/20 +step:1/20000 train_loss:7.6360 clean_loss:6.9365 diffusion_loss:6.9424 mask_ratio:0.2500 train_time:16419ms step_avg:16418.74ms +step:2/20000 train_loss:21.1372 clean_loss:16.1506 diffusion_loss:19.5221 mask_ratio:0.2500 train_time:16531ms step_avg:8265.27ms +step:3/20000 train_loss:20.5503 clean_loss:16.3411 diffusion_loss:18.9162 mask_ratio:0.2500 train_time:16661ms step_avg:5553.57ms +step:4/20000 train_loss:19.4333 clean_loss:15.8926 diffusion_loss:17.8440 mask_ratio:0.2500 train_time:16807ms step_avg:4201.82ms +step:5/20000 train_loss:17.8540 clean_loss:15.2130 diffusion_loss:16.3327 mask_ratio:0.2500 train_time:16941ms step_avg:3388.19ms +step:6/20000 train_loss:18.1954 clean_loss:14.4450 diffusion_loss:16.7509 mask_ratio:0.2500 train_time:17070ms step_avg:2844.92ms +step:7/20000 train_loss:16.1035 clean_loss:12.7005 diffusion_loss:14.8335 mask_ratio:0.2500 train_time:17204ms step_avg:2457.68ms +step:8/20000 train_loss:14.0051 clean_loss:11.1995 diffusion_loss:12.8852 mask_ratio:0.2500 train_time:17334ms step_avg:2166.74ms +step:9/20000 train_loss:11.2150 clean_loss:9.2424 diffusion_loss:10.2907 mask_ratio:0.2500 train_time:17463ms step_avg:1940.36ms +step:10/20000 train_loss:8.8937 clean_loss:7.6682 diffusion_loss:8.1268 mask_ratio:0.2500 train_time:17603ms step_avg:1760.34ms +step:25/20000 train_loss:6.8655 clean_loss:6.2150 diffusion_loss:6.2440 mask_ratio:0.2500 train_time:19389ms step_avg:775.58ms +step:50/20000 train_loss:6.6773 clean_loss:5.9963 diffusion_loss:6.0777 mask_ratio:0.2500 train_time:22358ms step_avg:447.16ms +step:75/20000 train_loss:6.5477 clean_loss:5.6595 diffusion_loss:5.9818 mask_ratio:0.2500 train_time:25339ms step_avg:337.85ms +step:100/20000 train_loss:6.5037 clean_loss:4.9691 diffusion_loss:6.0068 mask_ratio:0.2510 train_time:28290ms step_avg:282.90ms +step:125/20000 train_loss:6.4618 clean_loss:4.5044 diffusion_loss:6.0114 mask_ratio:0.2510 train_time:31244ms step_avg:249.95ms +step:150/20000 train_loss:6.3832 clean_loss:4.2769 diffusion_loss:5.9555 mask_ratio:0.2510 train_time:34197ms step_avg:227.98ms +step:175/20000 train_loss:6.3423 clean_loss:4.0563 diffusion_loss:5.9367 mask_ratio:0.2510 train_time:37148ms step_avg:212.27ms +step:200/20000 train_loss:6.3782 clean_loss:3.8093 diffusion_loss:5.9973 mask_ratio:0.2510 train_time:40166ms step_avg:200.83ms +step:225/20000 train_loss:6.3536 clean_loss:3.7844 diffusion_loss:5.9752 mask_ratio:0.2510 train_time:43127ms step_avg:191.67ms +step:250/20000 train_loss:6.3448 clean_loss:3.6428 diffusion_loss:5.9805 mask_ratio:0.2510 train_time:46077ms step_avg:184.31ms +step:275/20000 train_loss:6.3039 clean_loss:3.5929 diffusion_loss:5.9446 mask_ratio:0.2510 train_time:49027ms step_avg:178.28ms +step:300/20000 train_loss:6.1312 clean_loss:3.4136 diffusion_loss:5.7898 mask_ratio:0.2520 train_time:51976ms step_avg:173.25ms +step:325/20000 train_loss:6.2839 clean_loss:3.4640 diffusion_loss:5.9375 mask_ratio:0.2520 train_time:54930ms step_avg:169.01ms +step:350/20000 train_loss:6.2539 clean_loss:3.4984 diffusion_loss:5.9040 mask_ratio:0.2520 train_time:57885ms step_avg:165.39ms +step:375/20000 train_loss:6.2306 clean_loss:3.4217 diffusion_loss:5.8885 mask_ratio:0.2520 train_time:60845ms step_avg:162.25ms +step:400/20000 train_loss:6.2182 clean_loss:3.1216 diffusion_loss:5.9060 mask_ratio:0.2520 train_time:63871ms step_avg:159.68ms +step:425/20000 train_loss:6.2514 clean_loss:3.2808 diffusion_loss:5.9234 mask_ratio:0.2520 train_time:66836ms step_avg:157.26ms +step:450/20000 train_loss:6.1909 clean_loss:3.2279 diffusion_loss:5.8682 mask_ratio:0.2520 train_time:69787ms step_avg:155.08ms +step:475/20000 train_loss:6.2250 clean_loss:3.1274 diffusion_loss:5.9122 mask_ratio:0.2520 train_time:72737ms step_avg:153.13ms +step:500/20000 train_loss:6.2579 clean_loss:3.1370 diffusion_loss:5.9442 mask_ratio:0.2529 train_time:75686ms step_avg:151.37ms +step:525/20000 train_loss:6.2381 clean_loss:3.1418 diffusion_loss:5.9239 mask_ratio:0.2529 train_time:78635ms step_avg:149.78ms +step:550/20000 train_loss:6.1560 clean_loss:3.0258 diffusion_loss:5.8534 mask_ratio:0.2529 train_time:81584ms step_avg:148.34ms +step:575/20000 train_loss:6.1588 clean_loss:3.1148 diffusion_loss:5.8474 mask_ratio:0.2529 train_time:84607ms step_avg:147.14ms +step:600/20000 train_loss:6.2217 clean_loss:3.1268 diffusion_loss:5.9091 mask_ratio:0.2529 train_time:87557ms step_avg:145.93ms +step:625/20000 train_loss:6.2381 clean_loss:3.0647 diffusion_loss:5.9316 mask_ratio:0.2529 train_time:90531ms step_avg:144.85ms +step:650/20000 train_loss:6.1864 clean_loss:2.9306 diffusion_loss:5.8933 mask_ratio:0.2529 train_time:93588ms step_avg:143.98ms +step:675/20000 train_loss:6.1959 clean_loss:2.8951 diffusion_loss:5.9064 mask_ratio:0.2529 train_time:96540ms step_avg:143.02ms +step:700/20000 train_loss:6.1823 clean_loss:2.9914 diffusion_loss:5.8832 mask_ratio:0.2539 train_time:99489ms step_avg:142.13ms +step:725/20000 train_loss:6.1753 clean_loss:2.9787 diffusion_loss:5.8775 mask_ratio:0.2539 train_time:102440ms step_avg:141.30ms +step:750/20000 train_loss:6.1817 clean_loss:2.8233 diffusion_loss:5.8994 mask_ratio:0.2539 train_time:105405ms step_avg:140.54ms +step:775/20000 train_loss:6.1350 clean_loss:2.8793 diffusion_loss:5.8470 mask_ratio:0.2539 train_time:108438ms step_avg:139.92ms +step:800/20000 train_loss:6.1557 clean_loss:2.8258 diffusion_loss:5.8731 mask_ratio:0.2539 train_time:111405ms step_avg:139.26ms +step:825/20000 train_loss:6.1995 clean_loss:2.9464 diffusion_loss:5.9049 mask_ratio:0.2539 train_time:114373ms step_avg:138.63ms +step:850/20000 train_loss:6.0907 clean_loss:3.2226 diffusion_loss:5.7684 mask_ratio:0.2539 train_time:117334ms step_avg:138.04ms +step:875/20000 train_loss:6.1827 clean_loss:2.8566 diffusion_loss:5.8971 mask_ratio:0.2539 train_time:120287ms step_avg:137.47ms +step:900/20000 train_loss:6.1049 clean_loss:2.8871 diffusion_loss:5.8162 mask_ratio:0.2549 train_time:123236ms step_avg:136.93ms +step:925/20000 train_loss:5.3808 clean_loss:3.0286 diffusion_loss:5.0779 mask_ratio:0.2549 train_time:126184ms step_avg:136.42ms +step:950/20000 train_loss:6.1736 clean_loss:2.9234 diffusion_loss:5.8813 mask_ratio:0.2549 train_time:129133ms step_avg:135.93ms +step:975/20000 train_loss:5.9759 clean_loss:2.7420 diffusion_loss:5.7017 mask_ratio:0.2549 train_time:132145ms step_avg:135.53ms +step:1000/20000 train_loss:6.1732 clean_loss:2.8374 diffusion_loss:5.8894 mask_ratio:0.2549 train_time:135094ms step_avg:135.09ms +step:1025/20000 train_loss:6.1626 clean_loss:2.8892 diffusion_loss:5.8737 mask_ratio:0.2549 train_time:138056ms step_avg:134.69ms +step:1050/20000 train_loss:6.1812 clean_loss:2.9663 diffusion_loss:5.8845 mask_ratio:0.2549 train_time:141283ms step_avg:134.55ms +step:1075/20000 train_loss:6.1276 clean_loss:2.7313 diffusion_loss:5.8545 mask_ratio:0.2549 train_time:144831ms step_avg:134.73ms +step:1100/20000 train_loss:6.1447 clean_loss:2.7752 diffusion_loss:5.8672 mask_ratio:0.2559 train_time:147781ms step_avg:134.35ms +step:1125/20000 train_loss:6.1923 clean_loss:2.8176 diffusion_loss:5.9106 mask_ratio:0.2559 train_time:150739ms step_avg:133.99ms +step:1150/20000 train_loss:6.1553 clean_loss:2.7510 diffusion_loss:5.8802 mask_ratio:0.2559 train_time:153751ms step_avg:133.70ms +step:1175/20000 train_loss:6.1259 clean_loss:2.6999 diffusion_loss:5.8559 mask_ratio:0.2559 train_time:156714ms step_avg:133.37ms +step:1200/20000 train_loss:6.1504 clean_loss:2.8606 diffusion_loss:5.8644 mask_ratio:0.2559 train_time:159663ms step_avg:133.05ms +step:1225/20000 train_loss:6.1445 clean_loss:2.8826 diffusion_loss:5.8563 mask_ratio:0.2559 train_time:162622ms step_avg:132.75ms +step:1250/20000 train_loss:6.1257 clean_loss:2.6713 diffusion_loss:5.8585 mask_ratio:0.2559 train_time:165570ms step_avg:132.46ms +step:1275/20000 train_loss:6.1168 clean_loss:2.8071 diffusion_loss:5.8361 mask_ratio:0.2568 train_time:168525ms step_avg:132.18ms +step:1300/20000 train_loss:6.2516 clean_loss:2.8493 diffusion_loss:5.9667 mask_ratio:0.2568 train_time:171474ms step_avg:131.90ms +step:1325/20000 train_loss:6.1198 clean_loss:2.7912 diffusion_loss:5.8407 mask_ratio:0.2568 train_time:174427ms step_avg:131.64ms +step:1350/20000 train_loss:6.1663 clean_loss:2.7370 diffusion_loss:5.8926 mask_ratio:0.2568 train_time:177437ms step_avg:131.43ms +step:1375/20000 train_loss:6.1428 clean_loss:2.8304 diffusion_loss:5.8598 mask_ratio:0.2568 train_time:180407ms step_avg:131.20ms +step:1400/20000 train_loss:6.1347 clean_loss:2.8769 diffusion_loss:5.8470 mask_ratio:0.2568 train_time:183376ms step_avg:130.98ms +step:1425/20000 train_loss:6.1829 clean_loss:2.7791 diffusion_loss:5.9050 mask_ratio:0.2568 train_time:186329ms step_avg:130.76ms +step:1450/20000 train_loss:6.0290 clean_loss:2.7294 diffusion_loss:5.7560 mask_ratio:0.2568 train_time:189279ms step_avg:130.54ms +step:1475/20000 train_loss:6.1314 clean_loss:2.8039 diffusion_loss:5.8510 mask_ratio:0.2578 train_time:192230ms step_avg:130.33ms +step:1500/20000 train_loss:6.1077 clean_loss:2.7218 diffusion_loss:5.8355 mask_ratio:0.2578 train_time:195179ms step_avg:130.12ms +step:1525/20000 train_loss:6.2069 clean_loss:2.8209 diffusion_loss:5.9248 mask_ratio:0.2578 train_time:198129ms step_avg:129.92ms +step:1550/20000 train_loss:6.0860 clean_loss:2.6288 diffusion_loss:5.8231 mask_ratio:0.2578 train_time:201141ms step_avg:129.77ms +step:1575/20000 train_loss:6.1546 clean_loss:2.7207 diffusion_loss:5.8825 mask_ratio:0.2578 train_time:204093ms step_avg:129.58ms +step:1600/20000 train_loss:6.0344 clean_loss:2.5657 diffusion_loss:5.7779 mask_ratio:0.2578 train_time:207042ms step_avg:129.40ms +step:1625/20000 train_loss:6.1058 clean_loss:2.6606 diffusion_loss:5.8397 mask_ratio:0.2578 train_time:209992ms step_avg:129.23ms +step:1650/20000 train_loss:6.1172 clean_loss:2.6790 diffusion_loss:5.8493 mask_ratio:0.2578 train_time:212941ms step_avg:129.05ms +step:1675/20000 train_loss:6.1636 clean_loss:2.6999 diffusion_loss:5.8936 mask_ratio:0.2588 train_time:215890ms step_avg:128.89ms +step:1700/20000 train_loss:6.0634 clean_loss:2.6137 diffusion_loss:5.8020 mask_ratio:0.2588 train_time:218838ms step_avg:128.73ms +step:1725/20000 train_loss:6.1647 clean_loss:2.7109 diffusion_loss:5.8936 mask_ratio:0.2588 train_time:221852ms step_avg:128.61ms +step:1750/20000 train_loss:6.1473 clean_loss:2.6850 diffusion_loss:5.8788 mask_ratio:0.2588 train_time:224818ms step_avg:128.47ms +step:1775/20000 train_loss:6.0005 clean_loss:2.6964 diffusion_loss:5.7309 mask_ratio:0.2588 train_time:227768ms step_avg:128.32ms +step:1800/20000 train_loss:6.1200 clean_loss:2.6492 diffusion_loss:5.8551 mask_ratio:0.2588 train_time:230730ms step_avg:128.18ms +step:1825/20000 train_loss:6.1444 clean_loss:2.7168 diffusion_loss:5.8727 mask_ratio:0.2588 train_time:233680ms step_avg:128.04ms +step:1850/20000 train_loss:6.1103 clean_loss:2.7293 diffusion_loss:5.8374 mask_ratio:0.2588 train_time:236628ms step_avg:127.91ms +step:1875/20000 train_loss:6.1281 clean_loss:2.6506 diffusion_loss:5.8630 mask_ratio:0.2598 train_time:239578ms step_avg:127.77ms +step:1900/20000 train_loss:6.1023 clean_loss:2.6541 diffusion_loss:5.8369 mask_ratio:0.2598 train_time:242543ms step_avg:127.65ms +step:1925/20000 train_loss:6.1325 clean_loss:2.6994 diffusion_loss:5.8625 mask_ratio:0.2598 train_time:245566ms step_avg:127.57ms +step:1950/20000 train_loss:6.1137 clean_loss:2.6504 diffusion_loss:5.8487 mask_ratio:0.2598 train_time:248527ms step_avg:127.45ms +step:1975/20000 train_loss:6.1591 clean_loss:2.7350 diffusion_loss:5.8856 mask_ratio:0.2598 train_time:251476ms step_avg:127.33ms +step:2000/20000 train_loss:6.1141 clean_loss:2.6648 diffusion_loss:5.8476 mask_ratio:0.2598 train_time:254432ms step_avg:127.22ms +step:2025/20000 train_loss:6.1576 clean_loss:2.7714 diffusion_loss:5.8805 mask_ratio:0.2598 train_time:257380ms step_avg:127.10ms +step:2050/20000 train_loss:6.1576 clean_loss:2.6861 diffusion_loss:5.8890 mask_ratio:0.2598 train_time:260335ms step_avg:126.99ms +step:2075/20000 train_loss:6.0983 clean_loss:2.6159 diffusion_loss:5.8368 mask_ratio:0.2607 train_time:263285ms step_avg:126.88ms +step:2100/20000 train_loss:6.1355 clean_loss:2.6989 diffusion_loss:5.8656 mask_ratio:0.2607 train_time:266295ms step_avg:126.81ms +step:2125/20000 train_loss:6.1228 clean_loss:2.7085 diffusion_loss:5.8519 mask_ratio:0.2607 train_time:269244ms step_avg:126.70ms +step:2150/20000 train_loss:6.0887 clean_loss:2.6175 diffusion_loss:5.8269 mask_ratio:0.2607 train_time:272192ms step_avg:126.60ms +step:2175/20000 train_loss:6.1507 clean_loss:2.5926 diffusion_loss:5.8914 mask_ratio:0.2607 train_time:275141ms step_avg:126.50ms +step:2200/20000 train_loss:5.9702 clean_loss:2.5478 diffusion_loss:5.7154 mask_ratio:0.2607 train_time:278090ms step_avg:126.40ms +step:2225/20000 train_loss:6.1185 clean_loss:2.6246 diffusion_loss:5.8561 mask_ratio:0.2607 train_time:281042ms step_avg:126.31ms +step:2250/20000 train_loss:6.1334 clean_loss:2.6067 diffusion_loss:5.8728 mask_ratio:0.2617 train_time:283989ms step_avg:126.22ms +step:2275/20000 train_loss:6.0993 clean_loss:2.7015 diffusion_loss:5.8292 mask_ratio:0.2617 train_time:286937ms step_avg:126.13ms +step:2300/20000 train_loss:6.0425 clean_loss:2.7827 diffusion_loss:5.7642 mask_ratio:0.2617 train_time:289949ms step_avg:126.06ms +step:2325/20000 train_loss:6.1131 clean_loss:2.6229 diffusion_loss:5.8508 mask_ratio:0.2617 train_time:292920ms step_avg:125.99ms +step:2350/20000 train_loss:6.0903 clean_loss:2.6088 diffusion_loss:5.8294 mask_ratio:0.2617 train_time:295872ms step_avg:125.90ms +step:2375/20000 train_loss:6.1065 clean_loss:2.5428 diffusion_loss:5.8522 mask_ratio:0.2617 train_time:298823ms step_avg:125.82ms +step:2400/20000 train_loss:6.1126 clean_loss:2.6247 diffusion_loss:5.8502 mask_ratio:0.2617 train_time:301769ms step_avg:125.74ms +step:2425/20000 train_loss:6.1441 clean_loss:2.6618 diffusion_loss:5.8779 mask_ratio:0.2617 train_time:304723ms step_avg:125.66ms +step:2450/20000 train_loss:6.0633 clean_loss:2.6136 diffusion_loss:5.8020 mask_ratio:0.2627 train_time:307670ms step_avg:125.58ms +step:2475/20000 train_loss:6.1044 clean_loss:2.6233 diffusion_loss:5.8421 mask_ratio:0.2627 train_time:310627ms step_avg:125.51ms +step:2500/20000 train_loss:6.0736 clean_loss:2.5467 diffusion_loss:5.8189 mask_ratio:0.2627 train_time:313639ms step_avg:125.46ms +step:2525/20000 train_loss:6.0667 clean_loss:2.5574 diffusion_loss:5.8109 mask_ratio:0.2627 train_time:316587ms step_avg:125.38ms +step:2550/20000 train_loss:6.0852 clean_loss:2.5473 diffusion_loss:5.8305 mask_ratio:0.2627 train_time:319534ms step_avg:125.31ms +step:2575/20000 train_loss:6.0838 clean_loss:2.6272 diffusion_loss:5.8210 mask_ratio:0.2627 train_time:322482ms step_avg:125.24ms +step:2600/20000 train_loss:6.0855 clean_loss:2.7898 diffusion_loss:5.8066 mask_ratio:0.2627 train_time:325433ms step_avg:125.17ms +step:2625/20000 train_loss:6.0094 clean_loss:2.5038 diffusion_loss:5.7590 mask_ratio:0.2627 train_time:328380ms step_avg:125.10ms +step:2650/20000 train_loss:6.0968 clean_loss:2.6690 diffusion_loss:5.8299 mask_ratio:0.2637 train_time:331330ms step_avg:125.03ms +step:2675/20000 train_loss:6.1162 clean_loss:2.7113 diffusion_loss:5.8451 mask_ratio:0.2637 train_time:334344ms step_avg:124.99ms +step:2700/20000 train_loss:6.0825 clean_loss:2.5640 diffusion_loss:5.8261 mask_ratio:0.2637 train_time:337293ms step_avg:124.92ms +step:2725/20000 train_loss:5.9933 clean_loss:2.5256 diffusion_loss:5.7407 mask_ratio:0.2637 train_time:340240ms step_avg:124.86ms +step:2750/20000 train_loss:6.1239 clean_loss:2.7595 diffusion_loss:5.8480 mask_ratio:0.2637 train_time:343187ms step_avg:124.80ms +step:2775/20000 train_loss:6.0827 clean_loss:2.5697 diffusion_loss:5.8257 mask_ratio:0.2637 train_time:346136ms step_avg:124.73ms +step:2800/20000 train_loss:6.1609 clean_loss:2.6631 diffusion_loss:5.8946 mask_ratio:0.2637 train_time:349084ms step_avg:124.67ms +step:2825/20000 train_loss:6.1228 clean_loss:2.7547 diffusion_loss:5.8473 mask_ratio:0.2637 train_time:352032ms step_avg:124.61ms +step:2850/20000 train_loss:6.0845 clean_loss:2.6184 diffusion_loss:5.8227 mask_ratio:0.2646 train_time:354982ms step_avg:124.56ms +step:2875/20000 train_loss:6.0843 clean_loss:2.7156 diffusion_loss:5.8127 mask_ratio:0.2646 train_time:358003ms step_avg:124.52ms +step:2900/20000 train_loss:6.1313 clean_loss:2.5971 diffusion_loss:5.8716 mask_ratio:0.2646 train_time:360950ms step_avg:124.47ms +step:2925/20000 train_loss:6.0765 clean_loss:2.5334 diffusion_loss:5.8231 mask_ratio:0.2646 train_time:363913ms step_avg:124.41ms +step:2950/20000 train_loss:6.0855 clean_loss:2.6156 diffusion_loss:5.8239 mask_ratio:0.2646 train_time:366860ms step_avg:124.36ms +step:2975/20000 train_loss:6.0957 clean_loss:2.5601 diffusion_loss:5.8397 mask_ratio:0.2646 train_time:369825ms step_avg:124.31ms +step:3000/20000 train_loss:6.1180 clean_loss:2.6183 diffusion_loss:5.8562 mask_ratio:0.2646 train_time:372774ms step_avg:124.26ms +step:3025/20000 train_loss:6.0848 clean_loss:2.5864 diffusion_loss:5.8262 mask_ratio:0.2646 train_time:375722ms step_avg:124.21ms +step:3050/20000 train_loss:6.0977 clean_loss:2.5667 diffusion_loss:5.8410 mask_ratio:0.2656 train_time:378670ms step_avg:124.15ms +step:3075/20000 train_loss:6.0947 clean_loss:2.6729 diffusion_loss:5.8274 mask_ratio:0.2656 train_time:381685ms step_avg:124.13ms +step:3100/20000 train_loss:6.0746 clean_loss:2.5781 diffusion_loss:5.8168 mask_ratio:0.2656 train_time:384632ms step_avg:124.07ms +step:3125/20000 train_loss:6.0454 clean_loss:2.5067 diffusion_loss:5.7948 mask_ratio:0.2656 train_time:387583ms step_avg:124.03ms +step:3150/20000 train_loss:6.0940 clean_loss:2.5542 diffusion_loss:5.8386 mask_ratio:0.2656 train_time:390530ms step_avg:123.98ms +step:3175/20000 train_loss:6.1054 clean_loss:2.6797 diffusion_loss:5.8374 mask_ratio:0.2656 train_time:393477ms step_avg:123.93ms +step:3200/20000 train_loss:6.0793 clean_loss:2.5855 diffusion_loss:5.8207 mask_ratio:0.2656 train_time:396426ms step_avg:123.88ms +step:3225/20000 train_loss:6.1102 clean_loss:2.6539 diffusion_loss:5.8448 mask_ratio:0.2666 train_time:399374ms step_avg:123.84ms +step:3250/20000 train_loss:6.1269 clean_loss:2.4973 diffusion_loss:5.8771 mask_ratio:0.2666 train_time:402386ms step_avg:123.81ms +step:3275/20000 train_loss:6.0566 clean_loss:2.6422 diffusion_loss:5.7924 mask_ratio:0.2666 train_time:405335ms step_avg:123.77ms +step:3300/20000 train_loss:6.1165 clean_loss:2.6325 diffusion_loss:5.8532 mask_ratio:0.2666 train_time:408284ms step_avg:123.72ms +step:3325/20000 train_loss:6.1310 clean_loss:2.6532 diffusion_loss:5.8657 mask_ratio:0.2666 train_time:411233ms step_avg:123.68ms +step:3350/20000 train_loss:6.0390 clean_loss:2.4819 diffusion_loss:5.7908 mask_ratio:0.2666 train_time:414180ms step_avg:123.64ms +step:3375/20000 train_loss:6.0601 clean_loss:2.4901 diffusion_loss:5.8111 mask_ratio:0.2666 train_time:417128ms step_avg:123.59ms +step:3400/20000 train_loss:6.1045 clean_loss:2.5612 diffusion_loss:5.8484 mask_ratio:0.2666 train_time:420076ms step_avg:123.55ms +step:3425/20000 train_loss:6.0178 clean_loss:2.4804 diffusion_loss:5.7697 mask_ratio:0.2676 train_time:423026ms step_avg:123.51ms +step:3450/20000 train_loss:6.0641 clean_loss:2.4956 diffusion_loss:5.8145 mask_ratio:0.2676 train_time:426035ms step_avg:123.49ms +step:3475/20000 train_loss:6.1330 clean_loss:2.5131 diffusion_loss:5.8817 mask_ratio:0.2676 train_time:428983ms step_avg:123.45ms +step:3500/20000 train_loss:6.1372 clean_loss:2.6474 diffusion_loss:5.8725 mask_ratio:0.2676 train_time:431931ms step_avg:123.41ms +step:3525/20000 train_loss:6.1112 clean_loss:2.6154 diffusion_loss:5.8497 mask_ratio:0.2676 train_time:434880ms step_avg:123.37ms +step:3550/20000 train_loss:6.1057 clean_loss:2.7621 diffusion_loss:5.8295 mask_ratio:0.2676 train_time:437828ms step_avg:123.33ms +step:3575/20000 train_loss:6.1458 clean_loss:2.5971 diffusion_loss:5.8860 mask_ratio:0.2676 train_time:440775ms step_avg:123.29ms +step:3600/20000 train_loss:6.1020 clean_loss:2.5129 diffusion_loss:5.8508 mask_ratio:0.2676 train_time:443724ms step_avg:123.26ms +step:3625/20000 train_loss:6.1295 clean_loss:2.5431 diffusion_loss:5.8752 mask_ratio:0.2686 train_time:446733ms step_avg:123.24ms +step:3650/20000 train_loss:6.1329 clean_loss:2.5881 diffusion_loss:5.8741 mask_ratio:0.2686 train_time:449680ms step_avg:123.20ms +step:3675/20000 train_loss:6.1002 clean_loss:2.7184 diffusion_loss:5.8284 mask_ratio:0.2686 train_time:452632ms step_avg:123.17ms +step:3700/20000 train_loss:6.0542 clean_loss:2.5395 diffusion_loss:5.8003 mask_ratio:0.2686 train_time:455580ms step_avg:123.13ms +step:3725/20000 train_loss:6.1148 clean_loss:2.5858 diffusion_loss:5.8563 mask_ratio:0.2686 train_time:458532ms step_avg:123.10ms +step:3750/20000 train_loss:6.0657 clean_loss:2.5192 diffusion_loss:5.8137 mask_ratio:0.2686 train_time:461480ms step_avg:123.06ms +step:3775/20000 train_loss:6.0866 clean_loss:2.4668 diffusion_loss:5.8399 mask_ratio:0.2686 train_time:464428ms step_avg:123.03ms +step:3800/20000 train_loss:6.1060 clean_loss:2.5875 diffusion_loss:5.8473 mask_ratio:0.2686 train_time:467386ms step_avg:123.00ms +step:3825/20000 train_loss:6.1247 clean_loss:2.6066 diffusion_loss:5.8640 mask_ratio:0.2695 train_time:470402ms step_avg:122.98ms +step:3850/20000 train_loss:6.0922 clean_loss:2.5589 diffusion_loss:5.8363 mask_ratio:0.2695 train_time:473355ms step_avg:122.95ms +step:3875/20000 train_loss:6.0563 clean_loss:2.5167 diffusion_loss:5.8046 mask_ratio:0.2695 train_time:476413ms step_avg:122.95ms +step:3900/20000 train_loss:5.8877 clean_loss:2.3924 diffusion_loss:5.6485 mask_ratio:0.2695 train_time:479376ms step_avg:122.92ms +step:3925/20000 train_loss:6.1317 clean_loss:2.5296 diffusion_loss:5.8788 mask_ratio:0.2695 train_time:482346ms step_avg:122.89ms +step:3950/20000 train_loss:6.0802 clean_loss:2.4973 diffusion_loss:5.8304 mask_ratio:0.2695 train_time:485306ms step_avg:122.86ms +step:3975/20000 train_loss:6.0638 clean_loss:2.5208 diffusion_loss:5.8117 mask_ratio:0.2695 train_time:488259ms step_avg:122.83ms +step:4000/20000 train_loss:6.1084 clean_loss:2.5104 diffusion_loss:5.8574 mask_ratio:0.2695 train_time:491224ms step_avg:122.81ms +step:4025/20000 train_loss:6.0825 clean_loss:2.6162 diffusion_loss:5.8209 mask_ratio:0.2705 train_time:494567ms step_avg:122.87ms +step:4050/20000 train_loss:6.0874 clean_loss:2.4729 diffusion_loss:5.8402 mask_ratio:0.2705 train_time:497912ms step_avg:122.94ms +step:4075/20000 train_loss:6.0659 clean_loss:2.4449 diffusion_loss:5.8214 mask_ratio:0.2705 train_time:500859ms step_avg:122.91ms +step:4100/20000 train_loss:6.1087 clean_loss:2.5357 diffusion_loss:5.8551 mask_ratio:0.2705 train_time:503825ms step_avg:122.88ms +step:4125/20000 train_loss:6.1041 clean_loss:2.5103 diffusion_loss:5.8531 mask_ratio:0.2705 train_time:506770ms step_avg:122.85ms +step:4150/20000 train_loss:6.1337 clean_loss:2.6587 diffusion_loss:5.8678 mask_ratio:0.2705 train_time:509724ms step_avg:122.83ms +step:4175/20000 train_loss:6.1208 clean_loss:2.4882 diffusion_loss:5.8720 mask_ratio:0.2705 train_time:512672ms step_avg:122.80ms +step:4200/20000 train_loss:6.0862 clean_loss:2.5070 diffusion_loss:5.8355 mask_ratio:0.2705 train_time:515685ms step_avg:122.78ms +step:4225/20000 train_loss:6.1024 clean_loss:2.5993 diffusion_loss:5.8424 mask_ratio:0.2715 train_time:518634ms step_avg:122.75ms +step:4250/20000 train_loss:6.0638 clean_loss:2.4741 diffusion_loss:5.8164 mask_ratio:0.2715 train_time:521582ms step_avg:122.73ms +step:4275/20000 train_loss:6.0533 clean_loss:2.4163 diffusion_loss:5.8117 mask_ratio:0.2715 train_time:524530ms step_avg:122.70ms +step:4300/20000 train_loss:6.0132 clean_loss:2.3355 diffusion_loss:5.7796 mask_ratio:0.2715 train_time:527478ms step_avg:122.67ms +step:4325/20000 train_loss:6.0760 clean_loss:2.4380 diffusion_loss:5.8322 mask_ratio:0.2715 train_time:530425ms step_avg:122.64ms +step:4350/20000 train_loss:6.1173 clean_loss:2.5496 diffusion_loss:5.8623 mask_ratio:0.2715 train_time:533373ms step_avg:122.61ms +step:4375/20000 train_loss:6.0621 clean_loss:2.4706 diffusion_loss:5.8150 mask_ratio:0.2715 train_time:536319ms step_avg:122.59ms +step:4400/20000 train_loss:6.0780 clean_loss:2.4567 diffusion_loss:5.8324 mask_ratio:0.2725 train_time:539337ms step_avg:122.58ms +step:4425/20000 train_loss:6.0858 clean_loss:2.5442 diffusion_loss:5.8313 mask_ratio:0.2725 train_time:542283ms step_avg:122.55ms +step:4450/20000 train_loss:6.0016 clean_loss:2.3874 diffusion_loss:5.7628 mask_ratio:0.2725 train_time:545230ms step_avg:122.52ms +step:4475/20000 train_loss:6.1007 clean_loss:2.5182 diffusion_loss:5.8488 mask_ratio:0.2725 train_time:548177ms step_avg:122.50ms +step:4500/20000 train_loss:6.1022 clean_loss:2.5519 diffusion_loss:5.8470 mask_ratio:0.2725 train_time:551129ms step_avg:122.47ms +step:4525/20000 train_loss:6.1456 clean_loss:2.4534 diffusion_loss:5.9003 mask_ratio:0.2725 train_time:554078ms step_avg:122.45ms +step:4550/20000 train_loss:5.9952 clean_loss:2.3848 diffusion_loss:5.7567 mask_ratio:0.2725 train_time:557025ms step_avg:122.42ms +step:4575/20000 train_loss:6.0612 clean_loss:2.4650 diffusion_loss:5.8147 mask_ratio:0.2725 train_time:559972ms step_avg:122.40ms +step:4600/20000 train_loss:6.0595 clean_loss:2.3231 diffusion_loss:5.8272 mask_ratio:0.2734 train_time:562985ms step_avg:122.39ms +step:4625/20000 train_loss:6.0706 clean_loss:2.5576 diffusion_loss:5.8148 mask_ratio:0.2734 train_time:565936ms step_avg:122.36ms +step:4650/20000 train_loss:6.0524 clean_loss:2.3959 diffusion_loss:5.8128 mask_ratio:0.2734 train_time:568886ms step_avg:122.34ms +step:4675/20000 train_loss:6.0996 clean_loss:2.3921 diffusion_loss:5.8604 mask_ratio:0.2734 train_time:571839ms step_avg:122.32ms +step:4700/20000 train_loss:6.1061 clean_loss:2.5844 diffusion_loss:5.8476 mask_ratio:0.2734 train_time:574787ms step_avg:122.30ms +step:4725/20000 train_loss:6.0819 clean_loss:2.4972 diffusion_loss:5.8321 mask_ratio:0.2734 train_time:577735ms step_avg:122.27ms +step:4750/20000 train_loss:6.0045 clean_loss:2.3114 diffusion_loss:5.7733 mask_ratio:0.2734 train_time:580682ms step_avg:122.25ms +step:4775/20000 train_loss:6.0900 clean_loss:2.3871 diffusion_loss:5.8513 mask_ratio:0.2734 train_time:583703ms step_avg:122.24ms +step:4800/20000 train_loss:6.1068 clean_loss:2.5575 diffusion_loss:5.8511 mask_ratio:0.2744 train_time:586656ms step_avg:122.22ms +step:4825/20000 train_loss:6.0754 clean_loss:2.5540 diffusion_loss:5.8200 mask_ratio:0.2744 train_time:589623ms step_avg:122.20ms +step:4850/20000 train_loss:6.0817 clean_loss:2.4619 diffusion_loss:5.8355 mask_ratio:0.2744 train_time:592573ms step_avg:122.18ms +step:4875/20000 train_loss:6.0341 clean_loss:2.3449 diffusion_loss:5.7996 mask_ratio:0.2744 train_time:595525ms step_avg:122.16ms +step:4900/20000 train_loss:6.1053 clean_loss:2.4608 diffusion_loss:5.8593 mask_ratio:0.2744 train_time:598516ms step_avg:122.15ms +step:4912/20000 val_loss:2.4536 val_bpb:1.4531 train_time:599960ms step_avg:122.14ms +stopping_early: wallclock_cap train_time:599960ms step:4912/20000 +peak memory allocated: 20212 MiB reserved: 20222 MiB +Serialized model: 67227352 bytes +Code size: 83831 bytes +Total submission size: 67311183 bytes +Serialized model int8+zlib: 15447352 bytes (payload:17179936 raw_torch:17225367 payload_ratio:3.91x) +Total submission size int8+zlib: 15531183 bytes +final_int8_zlib_roundtrip val_loss:2.4658 val_bpb:1.4604 eval_time:1397ms +final_int8_zlib_roundtrip_exact val_loss:2.46577237 val_bpb:1.46036929 +parallel_eval task:continuation steps:1 token_acc:0.0312 exact_seq:0.0000 tokens:1024 tok_per_sec:28785.43 +parallel_eval task:infill steps:1 token_acc:0.0400 exact_seq:0.0000 tokens:1024 tok_per_sec:107351.05 +parallel_eval task:continuation steps:2 token_acc:0.0303 exact_seq:0.0000 tokens:1024 tok_per_sec:39006.19 +parallel_eval task:infill steps:2 token_acc:0.0420 exact_seq:0.0000 tokens:1024 tok_per_sec:53602.01 +parallel_eval task:continuation steps:4 token_acc:0.0312 exact_seq:0.0000 tokens:1024 tok_per_sec:26696.55 +parallel_eval task:infill steps:4 token_acc:0.0430 exact_seq:0.0000 tokens:1024 tok_per_sec:26763.10 +parallel_eval task:continuation steps:8 token_acc:0.0312 exact_seq:0.0000 tokens:1024 tok_per_sec:13381.45 +parallel_eval task:infill steps:8 token_acc:0.0410 exact_seq:0.0000 tokens:1024 tok_per_sec:13303.03