fix(engine_pool): skip the settle wait when other engines are serving

[JimStenstrom]

Addresses the freed=-51GB / eviction-stall portion of #1774 (it does not close the issue — it removes the eviction-time amplification under concurrent load). Complements the fixes already in 0.4.4.dev1, none of which touch the settle barrier itself.

Summary

• _unload_engine()'s settle barrier verifies an unload by polling mx.get_active_memory() and comparing the delta against the model's estimated size. That gauge is process-global: while another engine allocates (prefill/KV growth), the delta no longer measures this unload — under concurrent load it reads negative, which is exactly the freed=-51GB in Memory enforcer fails under concurrent load in 0.4.3 — process grows to 116 GB on a ~27 GB model pair, eviction reports freed=-51GB and breaches the configured ceiling → host OOM #1774's report.
• The barrier then burns all 10 settle rounds plus 3 emergency-reclaim rounds — ~8s of sleeps and 13 gc.collect() + mx.synchronize() + mx.clear_cache() cycles (beyond the initial release pass) serialized onto the MLX executor against live decode — while the memory enforcer holds the pool lock across the eviction, so admission is blocked too. Under pressure the eviction itself becomes a stall-and-spike, consistent with the reporter's "routine soft-pressure eviction … spiked to 115.9 GB in ~17 s".
• Fix: bail out of the settle wait when any other loaded entry is serving (active requests, or a held in-use lease from Engine pool evicts an acquired-but-not-yet-active engine → RuntimeError: Engine not started (acquire→use race; sibling of #1595) #1667). One INFO line, no retry rounds, no emergency reclaim. With the pool otherwise idle the barrier is byte-for-byte unchanged, so the original stuck-unload protection (0.3.5-rc1: engine_pool settle_tolerance fixed at 2 GB is too tight for large (>40 GB) models #768) is fully preserved.

Why this is safe

• Accounting is unchanged: the bail-out path accounts the unload exactly as the existing timed-out path already does.
• Unreleased memory stays visible: pre-load admission reads max(mx.get_active_memory(), get_phys_footprint(), _current_model_memory) (the v0.4.0 regression: Pre-load eviction fails when loading a second large model #1623 guard), so the live gauge dominates while anything is still resident — and _unload_engine ends with _wake_process_memory_enforcer(), so the enforcer re-polls immediately rather than waiting for the next tick. Skipping the emergency reclaim defers recovery to those two paths instead of losing it.
• Conservative failure direction: if an engine doesn't expose has_active_requests() (AttributeError, matching _find_lru_victim's idiom), the entry simply doesn't count as serving and the barrier falls back to today's full wait.
• Policy: this continues the direction of the 0.4.3 enforcer change and the fix(dflash): enforce prefill memory guard on DFlash primary path #1770 follow-up — control paths shouldn't fight the process-global MLX gauge while other work is in flight.

Trade-off (stated)

Under sustained concurrent pressure the enforcer's eviction loop now paces faster (no ~8s barrier stall between victims), so it can evict an additional idle victim where the old code would have stalled first. The old path reached the same evictions after timing out — just slower, with admission blocked and the executor saturated by sync/clear rounds in the meantime. If a pacing pause is still wanted there, a small enforcer-side delay would be the right knob; happy to follow up. Same offer for a settle_indeterminate counter in the pool stats (the #1406 pattern) if INFO-level logging feels too quiet for this condition.

Changes

omlx/engine_pool.py only:

• New _other_entries_serving(model_id) — true when any other loaded entry has in_use > 0 or has_active_requests(). Iterates a snapshot of _entries since the admin unload routes call _unload_engine without the pool lock.
• Settle loop: when a round fails the freed check and another entry is serving, log the indeterminate sample at INFO and break; the emergency-reclaim rounds are skipped on that path (the in-code comment documents the recovery story above).
• Every eviction flavor funnels through _unload_engine (enforcer soft/hard pressure, pre-load admission eviction, prefill-headroom eviction, TTL expiry, admin unload), so this single change covers all callers.

Tests

tests/test_engine_pool.py::TestMemorySettleBarrier:

• test_settle_bails_out_under_concurrent_activity — second entry serving + rising global gauge: the barrier exits after one sample (no 0.5s retries, no timeout warning, no emergency reclaim, exactly the one initial executor sync/clear cycle), and the unload still completes and is accounted.
• test_settle_still_waits_when_pool_otherwise_idle — same rising gauge with no other entry serving: full barrier behavior preserved (10 retries + emergency reclaim, 14 executor sync/clear cycles).
• test_other_entries_serving_in_use_lease_counts — the Engine pool evicts an acquired-but-not-yet-active engine → RuntimeError: Engine not started (acquire→use race; sibling of #1595) #1667 acquire-vs-use lease counts as serving.

.venv/bin/python -m pytest tests/test_engine_pool.py tests/test_process_memory_enforcer.py -q

201 passed

Also verified the new tests fail when the fix is reverted: restoring main's engine_pool.py fails test_settle_bails_out_under_concurrent_activity (the barrier burns every round) and test_other_entries_serving_in_use_lease_counts.

[jundot]

Thanks for the follow-up. I checked that this only shortcuts the settle wait when another engine is actually serving; idle loaded engines still keep the existing full settle path. The local test run passed with 201 tests, so this looks good to me and I'm going to merge it.