setup: start genie-ai-runtime before memory-heavy services - #76
Conversation
ai-hpc
left a comment
There was a problem hiding this comment.
Right shape for the right problem. Issue #75's evidence is the smoking gun: same -c 4096 request fits only ~1.7k ctx after the full stack is resident, but fits 4k / 6k / 8k cleanly when genie-ai-runtime loads first. The runtime's -c is clamped by available memory at load time, so startup order materially changes the realized KV-cache capacity.
Three coordinated changes:
-
Before=genie-whisper.service genie-whisper-warmup.service homeassistant.service genie-core.serviceon the unit. systemd ordering directive that makes the LLM unit's load happen before the other memory-heavy services claim DRAM. Defense in depth with PR #72'sAfter=genie-ai-runtime.service genie-llm.serviceongenie-core— both sides now agree, and the LLM-vs-whisper / LLM-vs-HA dependencies are also covered.Before=against a unit that isn't installed is a no-op, so this is safe on installs that don't ship homeassistant or whisper. -
GENIEPOD_AI_RUNTIME_CONTEXT=2048 → 8192. Matches the largest context the issue verified loads cleanly on Orin Nano 8 GB. The env knob stays — operators on smaller Jetsons (Nano 4 GB) can override via systemd drop-in without touching the unit file. -
start_all.shreorders so the configured LLM unit + warmup run before homeassistant + whisper in the manual lifecycle path too, mirroring the systemdBefore=for operators who use the script directly.
Tests added to tool_dispatch_test.rs lock both invariants:
start_all_uses_configured_llm_backendextension uses substring-position comparison on theUNITS=(...)array to assert$configured_llm_unitappears beforehomeassistant.serviceandgenie-whisper.service. Substring-position is fragile (a future rename of the$configured_llm_unitsymbol breaks the test), but the assertion message is precise enough that the failure mode points at the right line.genie_ai_runtime_service_preserves_model_page_cacheextension assertsGENIEPOD_AI_RUNTIME_CONTEXT=8192and that the newBefore=clause is present verbatim.
Worth flagging for a follow-up, not blocking:
- PR #74's
GENIE_RUNTIME_MAX_BODY_BYTES = 4 * 1024body-compaction threshold is now mismatched with the new context. Compaction triggers at 4 KB of body bytes regardless of the runtime's actual capacity, so with 8192-token runtime context (which can comfortably hold ~16-24 KB of typical English chat text), the client will still aggressively compact prompts the runtime could now handle. Same observation I left on #74. Right path is to make the threshold a function ofGENIEPOD_AI_RUNTIME_CONTEXTor to probe runtime capacity at connection time. The static 4 KB ceiling is now leaving more performance on the table than it was at 2048-ctx. - Compatibility with PR #70 (warm page cache) is preserved —
Before=only affects boot-time ordering, notsystemctl restartof the runtime alone, so the page-cache warmth across restart-of-runtime-only is intact. - Smaller-Jetson safety: an Orin Nano 4 GB or earlier Jetson without
--int8-kvsupport could fail to allocate 8192 ctx at boot. The unit'sRestart=on-failurewould loop in that case. Operators on those hosts will need a drop-in:Environment=GENIEPOD_AI_RUNTIME_CONTEXT=2048(or smaller). Worth a sentence in the next ROADMAP / GETTING_STARTED update if alpha.10 is targeting wider hardware.
All 7 CI checks green on c1cae29 (fmt, clippy, test, aarch64 cross-compile, shellcheck, ruff, --no-default-features). End-user verified on Jetson per your test. Going in.
|
Merged at |
…uring chat (#87) Closes #85 with the structurally-right fix to the PR #74 compaction, plus a useful bonus that keeps `/api/health`, `/api/services`, dashboard polls, and memory endpoints responsive while a chat turn is in flight. ### Runtime prompt compaction (commit `a26a25a`) Replaces PR #74's "compact to a single user turn with a hardcoded system blurb" with the design that PR's description actually claimed: - **All system messages** (tool manifest, household-preference injection, memory context) are preserved as-is. The reason #85 fired in the first place was that the system prompt got compacted away, leaving the LLM with no way to know `memory_recall` existed. - The **latest user turn** is preserved. - **Older non-system history** is walked backwards from the latest turn and kept as long as the running byte estimate fits in budget. - **`GENIE_RUNTIME_MAX_BODY_BYTES` raised `4 KB → 24 KB`** to match the 8192-token INT8-KV runtime context from PR #76. New `GENIE_RUNTIME_BODY_OVERHEAD_BYTES = 768` reserves space for the JSON envelope so the budget math doesn't quietly overshoot. - **Fallback path** preserves system messages alone if there's no user turn at all. Per the author's Jetson acceptance in the PR thread: `/api/chat` now returns `Your name is Jared` with `tool=memory_recall` — the exact reproduction from #85 that previously hallucinated `"Your name is GeniePod"`. ### Concurrent request handling (commit `a77d67b`) Changes `ChatServer::serve` from sequential `accept; handle; accept; handle; ...` to `tokio::task::LocalSet::spawn_local` so multiple HTTP requests can be in-flight on the single OS thread. A new `chat_turn_lock: tokio::sync::Mutex<()>` explicitly serializes only the chat-turn endpoints (`POST /api/chat`, `POST /api/chat/stream`, `POST /v1/chat/completions`). Everything else — `/api/health`, `/api/services`, `/api/memories`, runtime contract, history list — now runs concurrently with an in-flight chat turn. Net effect: the `:3080` dashboard's 5s polling stays responsive even when an LLM call is taking 4-8 seconds. Previously the whole HTTP server queue was serialized behind the LLM call (the old "LLM calls are seconds, HTTP is microseconds" comment was true but irrelevant when the queue was strict-sequential). ### Refactor that came along - `(method, path)` tuple matching at every route → `classify_route()` returning a named `RequestRoute<'a>` enum. - 4× repeated `if matches!(request_origin, Unknown) { Api } else { request_origin }` → `normalized_origin()` helper. - `with_chat_turn_lock(lock, fut)` encapsulates the "lock then await" pattern. ### Breaking change (internal) `ChatServer::serve` signature flipped from `&self` to `self` (because the body wraps in `Rc` for `spawn_local`). Only call sites are in `crates/genie-core/src/main.rs` — 4 mutually-exclusive branches that each move `chat_server` once. Verified via the `cargo clippy + test (--no-default-features)` and `cargo test` jobs both being green. ### Test updates - `genie_runtime_profile_compacts_large_core_prompt` expects 2 messages (system + user) instead of 1, asserts "memory_recall" and "household context" survive into the wire payload. - New `genie_runtime_profile_keeps_runtime_prompt_under_expanded_budget` covers "real-sized 10 KB prompt should NOT compact at all under the new 24 KB budget" — catches future regressions where someone shrinks the threshold back. - `genie_runtime_compaction_falls_back_to_latest_non_system_message` updated for the new 2-message output shape. All 6 CI checks green on `a77d67b` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, PR body checklist).
…tighter compaction + identity-recall fast-path (#106) Closes #107. The `GENIEPOD_AI_RUNTIME_CONTEXT=8192` default that landed in PR #76 was measured cold-boot-only and destabilized the box under steady-state load (chat UI lag, dashboard stutter, swap-file engagement). This PR implements option A from #107 — drop the context to `4096` with `--int8-kv` — plus three structural improvements that make the whole stack more robust at the new tighter context. Four changes: **1. Runtime context `8192 → 4096`** in `deploy/systemd/genie-ai-runtime.service`. The unit comment is updated honestly to explain why: "Keep the default at 4k because 8k can still fail to start on memory-fragmented full-stack restarts." README's "Why minimal-first" narrative also adjusted to match. Test in `tool_dispatch_test.rs` now pins `GENIEPOD_AI_RUNTIME_CONTEXT=4096` so a future PR can't silently raise it back without updating both the test and the rationale comment. **2. `start_all.sh` waits for the LLM `/health` endpoint** before starting memory-heavy services. New `wait_for_http_health` helper polls `curl -fsS --max-time 2 "$url"` up to 180 times. The existing `Before=` systemd ordering from PR #76 only waits for the LLM unit to reach `active`, which fires the moment `ExecStart` returns — long before the model is actually loaded. The health-gate closes that gap so whisper / genie-core / Home Assistant don't race the LLM's model load. `[services.llm].url` is read from config with a fallback to `http://127.0.0.1:8080/health`. **3. Compaction tuned for the 4k runtime** in `crates/genie-core/src/llm/openai_compat.rs`: - `GENIE_RUNTIME_MAX_BODY_BYTES` lowered `24 KB → 4 KB`. The 24 KB threshold was sized for the 8192-token runtime; at 4096 tokens it would overrun. - `compact_genie_runtime_system` rebuilds the system prompt structurally instead of passing it through verbatim. Generated sections: a richer prefix that explicitly tells the LLM the tool-call JSON contract (`{"tool":..,"arguments":..}`, no markdown), a filtered tool-list (`compact_genie_runtime_tool_lines` — only includes tools actually referenced in the source prompt, so chat-only deploys don't carry irrelevant `home_control` text), an adaptive rules block (`compact_genie_runtime_rules` — varies by whether HA is connected, web_search is enabled, etc.), and a household-context tail capped at 900 bytes via UTF-8-safe `truncate_utf8`. - The hardcoded `GENIE_RUNTIME_COMPACT_SYSTEM` blurb is replaced by the richer prefix. Eliminates a class of "model emits prose mentioning a tool name instead of a structured call" failures. **4. Identity-recall fast-path** in `crates/genie-core/src/tools/quick.rs`. New `memory_recall_query` recognizes `"what is my name"`, `"do you remember my name"`, `"who am i"`, `"what do you remember about X"`, `"search memory for X"`, and similar, then dispatches directly to `memory_recall` without going through the LLM at all. Per the PR body, returns `"Your name is Jared"` in 21 ms on Jetson — bypasses the entire LLM round-trip for the most common identity questions. Two new unit tests pin both the name-form and the search-form. The old `does_not_route_memory_search_to_web` test is correctly *deleted* — the new behavior actively does route memory searches (just to `memory_recall`, not `web_search`). Knock-on: this fix at the routing layer also robustly addresses issue #85 ("LLM hallucinated `Your name is GeniePod`"). Even on a runtime that doesn't see the tool manifest correctly, the identity question never reaches the LLM in the first place. Tests: - `genie_runtime_profile_compacts_runtime_prompt_under_4k_budget` (renamed from `_under_expanded_budget`) flips the asserted behavior to compaction-required at the new threshold, pins that the new structured system prompt survives and noisy text is dropped. - `start_all_uses_configured_llm_backend` extended with three assertions covering `read_llm_url`, `wait_for_http_health`, and the new `Configured LLM health` echo line. - `genie_ai_runtime_service_preserves_model_page_cache` extended to assert `GENIEPOD_AI_RUNTIME_CONTEXT=4096`. End-to-end Jetson verification in PR body: runtime started with `Context: 4096 tokens`, `KV cache: 288 MB`, `FREE: 1104 MB`. Chat at port 3000 works with streaming. Memory recall fires `tool: memory_recall` in 21 ms. All 8 CI checks green on `18bad34` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, shellcheck, ruff, PR body checklist).
Summary
GENIEPOD_AI_RUNTIME_CONTEXTto the Jetson-tested8192with--int8-kvgenie-ai-runtime.servicebefore Whisper, Home Assistant, andgenie-coreso runtime claims KV cache before memory-heavy services at bootstart_all.shso it starts the configured LLM before Home Assistant and Whisper without stopping those services manuallyFixes #75.
Verification
cargo fmt --checkcargo test -p genie-core --test tool_dispatch_test start_all_uses_configured_llm_backendcargo test -p genie-core --test tool_dispatch_test genie_ai_runtime_service_preserves_model_page_cachebash -n deploy/scripts/start_all.shgit diff --checkJetson context
Issue #75 captured Jetson testing where
8192loaded with INT8 KV whengenie-ai-runtimestarted before memory-heavy services. This PR preserves that condition through systemd ordering and the lifecycle script, instead of requiring the operator to manually stop the stack first.