feat(cache): add per-model cache hit-rate observability

[ivaniguarans]

Summary

The dashboard shows Cache Efficiency as a single cumulative percentage — useful at a glance, but it doesn't answer the questions operators actually ask: Is the prefix cache hitting? Are hits served from memory or disk? How full is each cache tier? Are evictions building up? Is it safe to flush the hot tier without restarting?

This turns the Runtime Cache Observability card into a complete operational surface for both cache tiers — memory (hot cache) and SSD (disk) — with per-model breakdowns, capacity gauges, rate metrics, and safe clear controls.

What changed

Capacity gauges — the card header now shows memory and SSD usage bars with live fill percentages. Memory gauge appears only when hot cache is enabled and models are loaded. Both tiers have clear buttons: the existing SSD clear, and a new memory-only clear that flushes the in-memory hot tier without touching persistent SSD blocks.

Cache rate metrics — a 4-metric strip inside the card shows session-scoped rates: Prefix Hit Rate, Memory Hit Rate, Prefix Evictions, and Memory Evictions. Hidden in All-Time view since the counters are session-scoped. When filtering by model, the strip shows that model's rates; in All Models view, each rate is computed from its own aggregate numerator/denominator (prefix hit rate from total prefix hits/misses, memory hit rate from total hot hits/disk loads — no cross-weighting).

Per-model table columns — the existing SSD Files/Size columns are joined by Memory Entries/Size columns (gated on hot cache being enabled), so operators can see per-model cache footprint at a glance.

Hot cache disabled — when --hot-cache-max-size is 0 (the default), the Memory gauge, Memory Entries/Size columns, and Memory rate metrics hide automatically. Only prefix and SSD metrics are shown.

Hot cache clear endpoint — POST /api/hot-cache/clear mirrors the existing SSD clear route. Iterates loaded model schedulers via a new _iter_loaded_schedulers() helper (which also de-duplicates the getattr chain from the SSD clear route), calls clear_hot_cache() on each manager, and resets the rate tracker so window rates don't carry stale pre-clear history.

Token-level match efficiency — prefix cache accounting now tracks tokens_matched_total and tokens_requested_total alongside the existing hit/miss counts. A 2-token match on a 2048-token request is very different from a 2000-token match; prefix_match_efficiency captures that distinction. Exposed in the API response for scripting consumers.

How it works

CacheRateTracker (149 lines, omlx/cache/observability.py) stores up to 90 snapshots of cumulative cache counters at ~10-second intervals. Snapshots are lazy — taken only when the dashboard polls /api/stats — so there are zero hot-path changes beyond two counter increments in fetch_cache(). The scheduler's _collect_cache_counters reads from the public get_stats() API on PagedSSDCacheManager, avoiding private attribute access.

The tracker computes:

• Windowed rates (1m/5m/15m) via start/end snapshot differencing — available in the API response for scripting consumers
• Cumulative session rates — what the dashboard displays, since windowed rates go stale too fast on a single-user local server

The SSD tier avoids double-counting by computing ssd_disk_loads = total_loads - hot_cache_hits, since loads includes both hot and disk reads. On scheduler reset, the tracker is cleared alongside prefix cache stats.

Known behavior surfaced by this PR

While validating the gauges against real multi-model workloads, we observed two pre-existing cache behaviors worth tracking separately:

Hot cache capacity is per loaded model, not process-global. Each scheduler creates its own PagedSSDCacheManager with an independent hot_cache_max_bytes budget. With 3 models loaded and --hot-cache-max-size 4GB, the dashboard correctly reports 12 GB total hot-cache capacity because the gauge sums per-model pools. The CLI help text says "Maximum in-memory hot cache size" without clarifying it applies per model — users setting 4 GB expect a 4 GB process budget, not 4 GB × N.

SSD usage can exceed the configured limit with multiple models. All model schedulers write to the same cache directory, but each manager enforces max_size independently. Under eviction pressure, the background writer queue can fill up, causing eviction deletes to be dropped ("SSD write queue full, dropping evicted block") while writes continue. We observed 68.8 GB on a 50 GB configured limit with all models idle.

This PR reports those values truthfully but does not change cache ownership or enforcement semantics — that belongs in separate issues.

Changes

omlx/cache/observability.py          | 149 new    — CacheRateTracker module
tests/test_cache_observability.py    | 202 new    — 15 tests (snapshots, rates, threads, clear)
omlx/cache/prefix_cache.py          |  13 +      — token-level match counters in fetch_cache()
omlx/cache/stats.py                 |   4 +      — counter fields in PrefixCacheStats
omlx/cache/paged_ssd_cache.py       |  14 +      — clear_hot_cache() method
omlx/scheduler.py                   |  42 +/-    — counter collection via public API + tracker integration
omlx/admin/routes.py                | 180 +/-    — _iter_loaded_schedulers(), hot-cache clear endpoint, cache aggregation
omlx/admin/static/js/dashboard.js   |  64 +      — cacheObsCumulative aggregation, gauge getters, clear handler
omlx/admin/templates/_status.html   |  77 +      — gauges, rate strip, per-model memory columns
omlx/admin/static/css/dashboard.css |   4 +      — gauge track dark mode

Test plan

• uv run pytest tests/test_cache_observability.py tests/test_hot_cache.py -v — 37 tests covering snapshot mechanics, rate computation (steady-state, zero-activity, eviction rates, SSD hot ratio, token match efficiency), thread safety under concurrent access, clear/reset behavior, hot cache lifecycle, LRU eviction, byte accounting, and write-back to SSD
• Live-tested on three models simultaneously (Qwen3.6-27B-oQ6-mtp, Qwen3.6-27B-oQ8-mtp, Qwen3.6-35B-A3B-oQ4-mtp) — verified per-model filtering, aggregate rate computation, gauge accuracy, and both clear endpoints
• Verified Session/All-Time toggle hides the rate strip correctly while keeping gauges visible
• Verified memory gauge and columns hide when hot cache is disabled
• Verified hot cache clear flushes entries and resets gauges without affecting SSD
• 36 pre-existing test failures in unrelated files (server endpoints, grammar, VLM, MTP, oQ, admin update check) — none in our changed files

Follow-ups

Two extensions that build on this without expanding scope:

1. Average prefix match depth — tokens_matched / hits to distinguish shallow matches from deep ones. The counters are already tracked server-side; this would be a dashboard-only addition.
2. Windowed rate selector — the backend already computes 1m/5m/15m windowed rates. A dropdown in the dashboard would let operators see recent trends instead of only cumulative session values.

Related: #905 (paged KV occupancy — complementary surface), #1171 (hot cache byte accounting — no conflict)

[blightbow]

Heads-up: three TestRuntimeCacheObservability tests pass on main but fail on this PR's HEAD. This surfaced when attempting to rebase my own PR #1149 onto 6c91508.

$ git checkout 172be39  # current upstream/main
$ uv run pytest tests/test_admin_api_key.py::TestRuntimeCacheObservability -q
3 passed

$ git checkout 6c91508  # this PR's HEAD
$ uv run pytest tests/test_admin_api_key.py::TestRuntimeCacheObservability -q
3 failed

All three failures share the same signature:

 omlx/admin/routes.py:3624: TypeError: '>' not supported between instances of 'int' and 'MagicMock'
    disk_max = max(disk_max, m.get("max_size_bytes", 0))

Root cause is the new aggregation block at routes.py:3617-3625 (Aggregate hot-cache and disk-max across models). It initializes disk_max = payload["disk_max_bytes"], whose value flows from global_settings.cache.get_ssd_cache_max_size_bytes(...). The test mocks global_settings with MagicMock() and only configures a couple of explicit return values; get_ssd_cache_max_size_bytes is autospec'd and returns a MagicMock. The per-model m.get("max_size_bytes", 0) is correctly an int (the dict construction at routes.py:3591 already does the int(... or 0) coercion), but max(MagicMock, int) raises.

One related observation: even after the TypeError is fixed, the test's assert payload["models"] == [...] expected-dict at tests/test_admin_api_key.py:739-768 doesn't include the new max_size_bytes key that routes.py:3591 writes, so the assertion will still fail on key-set mismatch. The hot_cache_max_bytes / hot_cache_size_bytes / hot_cache_entries keys are present in the expected dict; max_size_bytes is the lone omission. Worth one pass to make sure the expected payload tracks what the implementation produces.

[ivaniguarans]

Thanks for the detailed report — spot on. The new disk_max aggregation block reads get_ssd_cache_max_size_bytes which those tests left unconfigured, so MagicMock leaked into max(). Fixed that and added the missing max_size_bytes key to the expected payloads. Also rebased onto current main. Should be green now — you should be unblocked for #1149.

[jundot]

nice, the per-tier gauges + rate breakdown beat the single cumulative %, and CacheRateTracker stays off the hot path (snapshot-driven). Tests pass. Merging.