Diffusers-reference t2v/t2i output diverges progressively (oversaturated, speckled) — reproduced on CPU/MPS, torch 2.11/2.12/2.13, dense & MoE

[naturalspace]

Summary

Using the public diffusers-reference path (--backend diffusers), both robbyant/lingbot-video-dense-1.3b and robbyant/lingbot-video-moe-30b-a3b produce heavily corrupted output: scene structure and motion follow the prompt, but colors oversaturate into psychedelic tones with block/speckle artifacts. Crucially, corruption grows with step count (4 steps ≈ soft but sane; 8 steps worse; 40 steps fully degraded), which looks like a small per-step mismatch compounded by the UniPC solver.

Repro

python scripts/inference.py --backend diffusers --model_dir <dense-1.3b> --mode t2v \
  --prompt_json assets-style structured caption (also plain --prompt) \
  --output out.mp4 --height 480 --width 832 --num_frames 121 --steps 40 \
  --guidance_scale 3 --shift 3 --seed 20260714 --fps 24 \
  --transformer_dtype bf16 --text_encoder_dtype bf16 --vae_dtype fp32

(no --batch_cfg: FA3 unavailable; sequential CFG)

Environment matrix — all reproduce identically (deterministic per seed)

Device	dtype	torch	result
Apple M4 Max (MPS)	bf16	2.13.0	corrupted
Apple M4 Max (MPS)	fp32	2.13.0	corrupted
CPU	fp32	2.13.0	corrupted
Apple M4 Max (MPS)	bf16	2.12.1	corrupted
MoE 30B-A3B (MPS, expert loop)	bf16	2.13.0	corrupted

diffusers==0.39.0, transformers==5.8.1 (as pinned), text encoder attn_implementation=sdpa (LINGBOT_QWEN_ATTN_IMPLEMENTATION=sdpa; FA3 not installed).

What we ruled out

• VAE: encode→decode roundtrip of a synthetic video is pixel-perfect (mean err 0.007)
• Weights: output_loading_info → 0 missing / 0 unexpected / 0 mismatched keys
• Scheduler: FlowUniPCMultistepScheduler loads from model repo; set_timesteps(8, shift=3) → timesteps [999, 954, ... 299], sigma*1000 convention matches the transformer docstring
• RoPE on MPS: apply_rotary_emb MPS vs CPU max diff 0.0
• patchify/unpatchify: input reshape/permute and output permute are mutually consistent
• t2i is affected too at 40 steps (480p), so it is not temporal-only

Question

Does the public diffusers-reference path require the pinned torch nightly (2.12.0.dev20260220+cu130) / FA3 / --batch_cfg for correctness (not just speed)? Or could the released HF weights differ from the ones validated with this code? Sample frames available on request.

Great project — happy to test fixes.

[naturalspace]

Update — scoped to macOS/arm64. We re-ran the identical config (same structured caption, seed, 480p×121f, 40 steps, sequential CFG, sdpa text encoder, no FA3) on RunPod H100s (torch 2.11.0+cu128, diffusers 0.39.0, transformers 5.8.1):

• 1×H100, dense-1.3b: beautiful, fully coherent output (117s denoise)
• 2×H100 FSDP, moe-30b-a3b: excellent output (334s denoise)

So the released code + weights are correct on Linux/CUDA. The progressive divergence we reported reproduces only on macOS arm64 — on BOTH the MPS backend and plain CPU (fp32), across torch 2.12.1/2.13.0. Since CPU fp32 is affected too, this looks like a numerical bug in some op's arm64/macOS kernel used by this model rather than an MPS-specific issue. Feel free to retitle/close as out-of-scope for the repo — leaving the matrix here for anyone trying Apple Silicon.

[naturalspace]

Correction / root cause found. My earlier 'macOS-only' framing was incomplete. Controlled test on 1×H100 (torch 2.11+cu128, diffusers 0.39.0):

• plain --prompt string → garbage (oversaturated/speckled), even on H100
• structured --prompt_json caption (rewriter output) → beautiful, same seed/config

So the model requires the two-stage rewriter's structured caption; feeding a bare prompt string through the DiT produces the divergence I reported. This lines up with #10 ("is the ReWriter important") — for this reference path it's effectively mandatory, not optional. Suggest the README call this out prominently (the --prompt flag is a footgun that looks like it should work). Thanks — closing my end as user error rather than a code bug.