fix(memory): refuse oversized prefill chunks before the forward (anti-panic)

[panwudi]

What this is now (replaces the rejected load-admission headroom)

The PR originally rejected oversized models at load. Per user feedback that's the wrong shape -- a model must be loadable; refuse the request, not the model. That approach is reverted (engine_pool/settings back to main). This now stops the crash at the prefill level.

Why

glm4.5-air-106b (85GB) on a 128GB box (Metal cap 107.5GB) leaves ~22GB for KV+prefill. A per-chunk MoE-dequant transient (measured up to 7.4GB on m5max, peaks hit 110GB) crosses the cap. oMLX checks memory AFTER the forward, but on Apple UMA an over-cap allocation kernel-panics the whole machine before any Python check runs -- rebooted the box twice (2026-06-01, -06-06).

What

• Scheduler._prefill_forward_gate: before each prefill forward (external + chunked-step loops), predict current(high-water: max active/phys/recent_peak) + estimate_prefill_peak_bytes(KV+SDPA) + margin; raise BEFORE the forward if it would breach the hard cap. The existing [Bug] Prefill requests stuck after force-stopped due to memory limit exceeded - never released jundot/omlx#1405 cleanup turns that into a finish_reason=error (503-class) -- request refused cleanly, machine not crashed. Legacy post-forward check kept as backstop.
• MemorySettings.prefill_transient_margin_gb (default 10GB), propagated settings -> enforcer -> scheduler. estimate_prefill_peak_bytes does not model the MoE expert-dequant spike, so this margin carries the guarantee: margin(10) > worst observed single-step jump(7.4GB), measured from m5max crash logs.
• Preflight also maxes against the enforcer recent high-water mark.

Honest residual (read before merge)

The gate reads current right after the prior chunk's cache clear (active-memory trough), leaning on phys_footprint stickiness + recent_peak to avoid a trough misread. A trough misread could still admit a crashing chunk -- this cannot be mock-tested; it needs on-hardware [memgate] log validation. This is "crash -> 503 for the common cases", NOT a literal never-panic guarantee. The real fix is preemptive KV offload (mlx-lm's extract_cache/remove/insert primitives support it -- separate, larger work).

Also: with a 10GB margin, glm4.5 on this box refuses requests aggressively (~97GB trip point). It stops the crash; it does NOT make glm4.5 comfortably usable for long-context agentic work on a 128GB box -- that needs a smaller quant or the preemption work.

Test plan

• Targeted (self-run, m2max): test_scheduler_prefill_forward_gate.py + admission + enforcer + engine_pool -> 157 passed.
• 12 new gate tests; verified to FAIL with the gate neutered (pins the fix).
• Full suite (m2max): 4542 pass / 3 fail / 19 skip. The 3 fails are pre-existing test_settings.py api_key env-bleed (confirmed identical on clean main). Zero regression.

Deploy validation (after merge, on m5max)

1. Deploy + restart serve.
2. Load glm4.5-air-106b + send a long prompt -> expect a clean 503-class refusal (NOT a crash). Watch [memgate] logs: confirm current reflects high-water (not a sub-97GB trough) before a would-overshoot chunk.
3. Load a normal model (e.g. 27b) -> confirm normal requests still run.

[panwudi]

Superseded by #51. The forward gate read the mid-prefill trough with margin 10 and under-reported (10 < the observed ~10.6GB transient). #51 makes entry-point budget admission the primary fix (stable read, queue-not-stack), bumps the margin to 12, and keeps this forward gate un-retired as the per-chunk concurrent-drift backstop.

---

被 #51 取代. forward gate 读 prefill 中途谷值 + margin 10 漏报 (10 < 实测 ~10.6GB 瞬时). #51 以入口预算准入为主 (稳定读数, 排队不叠加), margin 升 12, 并保留此 forward gate 作 per-chunk 并发漂移兜底.