Skip to content

setup: start genie-ai-runtime before memory-heavy services - #76

Merged
ai-hpc merged 1 commit into
GeniePod:mainfrom
carlos4s:fix/issue-75-runtime-context-order
May 18, 2026
Merged

setup: start genie-ai-runtime before memory-heavy services#76
ai-hpc merged 1 commit into
GeniePod:mainfrom
carlos4s:fix/issue-75-runtime-context-order

Conversation

@carlos4s

Copy link
Copy Markdown
Contributor

Summary

  • raise the default GENIEPOD_AI_RUNTIME_CONTEXT to the Jetson-tested 8192 with --int8-kv
  • order genie-ai-runtime.service before Whisper, Home Assistant, and genie-core so runtime claims KV cache before memory-heavy services at boot
  • reorder start_all.sh so it starts the configured LLM before Home Assistant and Whisper without stopping those services manually

Fixes #75.

Verification

  • cargo fmt --check
  • cargo test -p genie-core --test tool_dispatch_test start_all_uses_configured_llm_backend
  • cargo test -p genie-core --test tool_dispatch_test genie_ai_runtime_service_preserves_model_page_cache
  • bash -n deploy/scripts/start_all.sh
  • git diff --check

Jetson context

Issue #75 captured Jetson testing where 8192 loaded with INT8 KV when genie-ai-runtime started before memory-heavy services. This PR preserves that condition through systemd ordering and the lifecycle script, instead of requiring the operator to manually stop the stack first.

@ai-hpc ai-hpc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right shape for the right problem. Issue #75's evidence is the smoking gun: same -c 4096 request fits only ~1.7k ctx after the full stack is resident, but fits 4k / 6k / 8k cleanly when genie-ai-runtime loads first. The runtime's -c is clamped by available memory at load time, so startup order materially changes the realized KV-cache capacity.

Three coordinated changes:

  1. Before=genie-whisper.service genie-whisper-warmup.service homeassistant.service genie-core.service on the unit. systemd ordering directive that makes the LLM unit's load happen before the other memory-heavy services claim DRAM. Defense in depth with PR #72's After=genie-ai-runtime.service genie-llm.service on genie-core — both sides now agree, and the LLM-vs-whisper / LLM-vs-HA dependencies are also covered. Before= against a unit that isn't installed is a no-op, so this is safe on installs that don't ship homeassistant or whisper.

  2. GENIEPOD_AI_RUNTIME_CONTEXT=2048 → 8192. Matches the largest context the issue verified loads cleanly on Orin Nano 8 GB. The env knob stays — operators on smaller Jetsons (Nano 4 GB) can override via systemd drop-in without touching the unit file.

  3. start_all.sh reorders so the configured LLM unit + warmup run before homeassistant + whisper in the manual lifecycle path too, mirroring the systemd Before= for operators who use the script directly.

Tests added to tool_dispatch_test.rs lock both invariants:

  • start_all_uses_configured_llm_backend extension uses substring-position comparison on the UNITS=(...) array to assert $configured_llm_unit appears before homeassistant.service and genie-whisper.service. Substring-position is fragile (a future rename of the $configured_llm_unit symbol breaks the test), but the assertion message is precise enough that the failure mode points at the right line.
  • genie_ai_runtime_service_preserves_model_page_cache extension asserts GENIEPOD_AI_RUNTIME_CONTEXT=8192 and that the new Before= clause is present verbatim.

Worth flagging for a follow-up, not blocking:

  • PR #74's GENIE_RUNTIME_MAX_BODY_BYTES = 4 * 1024 body-compaction threshold is now mismatched with the new context. Compaction triggers at 4 KB of body bytes regardless of the runtime's actual capacity, so with 8192-token runtime context (which can comfortably hold ~16-24 KB of typical English chat text), the client will still aggressively compact prompts the runtime could now handle. Same observation I left on #74. Right path is to make the threshold a function of GENIEPOD_AI_RUNTIME_CONTEXT or to probe runtime capacity at connection time. The static 4 KB ceiling is now leaving more performance on the table than it was at 2048-ctx.
  • Compatibility with PR #70 (warm page cache) is preserved — Before= only affects boot-time ordering, not systemctl restart of the runtime alone, so the page-cache warmth across restart-of-runtime-only is intact.
  • Smaller-Jetson safety: an Orin Nano 4 GB or earlier Jetson without --int8-kv support could fail to allocate 8192 ctx at boot. The unit's Restart=on-failure would loop in that case. Operators on those hosts will need a drop-in: Environment=GENIEPOD_AI_RUNTIME_CONTEXT=2048 (or smaller). Worth a sentence in the next ROADMAP / GETTING_STARTED update if alpha.10 is targeting wider hardware.

All 7 CI checks green on c1cae29 (fmt, clippy, test, aarch64 cross-compile, shellcheck, ruff, --no-default-features). End-user verified on Jetson per your test. Going in.

@ai-hpc
ai-hpc merged commit 9ab2974 into GeniePod:main May 18, 2026
7 checks passed
@ai-hpc

ai-hpc commented May 18, 2026

Copy link
Copy Markdown
Contributor

Merged at 9ab2974e6b0acd3b422576fb57715e71a2086d5b.
Thanks to @carlos4s!

ai-hpc pushed a commit that referenced this pull request May 18, 2026
…uring chat (#87)

Closes #85 with the structurally-right fix to the PR #74 compaction, plus a useful bonus that keeps `/api/health`, `/api/services`, dashboard polls, and memory endpoints responsive while a chat turn is in flight.

### Runtime prompt compaction (commit `a26a25a`)

Replaces PR #74's "compact to a single user turn with a hardcoded system blurb" with the design that PR's description actually claimed:

- **All system messages** (tool manifest, household-preference injection, memory context) are preserved as-is. The reason #85 fired in the first place was that the system prompt got compacted away, leaving the LLM with no way to know `memory_recall` existed.
- The **latest user turn** is preserved.
- **Older non-system history** is walked backwards from the latest turn and kept as long as the running byte estimate fits in budget.
- **`GENIE_RUNTIME_MAX_BODY_BYTES` raised `4 KB → 24 KB`** to match the 8192-token INT8-KV runtime context from PR #76. New `GENIE_RUNTIME_BODY_OVERHEAD_BYTES = 768` reserves space for the JSON envelope so the budget math doesn't quietly overshoot.
- **Fallback path** preserves system messages alone if there's no user turn at all.

Per the author's Jetson acceptance in the PR thread: `/api/chat` now returns `Your name is Jared` with `tool=memory_recall` — the exact reproduction from #85 that previously hallucinated `"Your name is GeniePod"`.

### Concurrent request handling (commit `a77d67b`)

Changes `ChatServer::serve` from sequential `accept; handle; accept; handle; ...` to `tokio::task::LocalSet::spawn_local` so multiple HTTP requests can be in-flight on the single OS thread. A new `chat_turn_lock: tokio::sync::Mutex<()>` explicitly serializes only the chat-turn endpoints (`POST /api/chat`, `POST /api/chat/stream`, `POST /v1/chat/completions`). Everything else — `/api/health`, `/api/services`, `/api/memories`, runtime contract, history list — now runs concurrently with an in-flight chat turn.

Net effect: the `:3080` dashboard's 5s polling stays responsive even when an LLM call is taking 4-8 seconds. Previously the whole HTTP server queue was serialized behind the LLM call (the old "LLM calls are seconds, HTTP is microseconds" comment was true but irrelevant when the queue was strict-sequential).

### Refactor that came along

- `(method, path)` tuple matching at every route → `classify_route()` returning a named `RequestRoute<'a>` enum.
- 4× repeated `if matches!(request_origin, Unknown) { Api } else { request_origin }` → `normalized_origin()` helper.
- `with_chat_turn_lock(lock, fut)` encapsulates the "lock then await" pattern.

### Breaking change (internal)

`ChatServer::serve` signature flipped from `&self` to `self` (because the body wraps in `Rc` for `spawn_local`). Only call sites are in `crates/genie-core/src/main.rs` — 4 mutually-exclusive branches that each move `chat_server` once. Verified via the `cargo clippy + test (--no-default-features)` and `cargo test` jobs both being green.

### Test updates

- `genie_runtime_profile_compacts_large_core_prompt` expects 2 messages (system + user) instead of 1, asserts "memory_recall" and "household context" survive into the wire payload.
- New `genie_runtime_profile_keeps_runtime_prompt_under_expanded_budget` covers "real-sized 10 KB prompt should NOT compact at all under the new 24 KB budget" — catches future regressions where someone shrinks the threshold back.
- `genie_runtime_compaction_falls_back_to_latest_non_system_message` updated for the new 2-message output shape.

All 6 CI checks green on `a77d67b` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, PR body checklist).
ai-hpc pushed a commit that referenced this pull request May 18, 2026
…tighter compaction + identity-recall fast-path (#106)

Closes #107. The `GENIEPOD_AI_RUNTIME_CONTEXT=8192` default that landed in PR #76 was measured cold-boot-only and destabilized the box under steady-state load (chat UI lag, dashboard stutter, swap-file engagement). This PR implements option A from #107 — drop the context to `4096` with `--int8-kv` — plus three structural improvements that make the whole stack more robust at the new tighter context.

Four changes:

**1. Runtime context `8192 → 4096`** in `deploy/systemd/genie-ai-runtime.service`. The unit comment is updated honestly to explain why: "Keep the default at 4k because 8k can still fail to start on memory-fragmented full-stack restarts." README's "Why minimal-first" narrative also adjusted to match. Test in `tool_dispatch_test.rs` now pins `GENIEPOD_AI_RUNTIME_CONTEXT=4096` so a future PR can't silently raise it back without updating both the test and the rationale comment.

**2. `start_all.sh` waits for the LLM `/health` endpoint** before starting memory-heavy services. New `wait_for_http_health` helper polls `curl -fsS --max-time 2 "$url"` up to 180 times. The existing `Before=` systemd ordering from PR #76 only waits for the LLM unit to reach `active`, which fires the moment `ExecStart` returns — long before the model is actually loaded. The health-gate closes that gap so whisper / genie-core / Home Assistant don't race the LLM's model load. `[services.llm].url` is read from config with a fallback to `http://127.0.0.1:8080/health`.

**3. Compaction tuned for the 4k runtime** in `crates/genie-core/src/llm/openai_compat.rs`:

- `GENIE_RUNTIME_MAX_BODY_BYTES` lowered `24 KB → 4 KB`. The 24 KB threshold was sized for the 8192-token runtime; at 4096 tokens it would overrun.
- `compact_genie_runtime_system` rebuilds the system prompt structurally instead of passing it through verbatim. Generated sections: a richer prefix that explicitly tells the LLM the tool-call JSON contract (`{"tool":..,"arguments":..}`, no markdown), a filtered tool-list (`compact_genie_runtime_tool_lines` — only includes tools actually referenced in the source prompt, so chat-only deploys don't carry irrelevant `home_control` text), an adaptive rules block (`compact_genie_runtime_rules` — varies by whether HA is connected, web_search is enabled, etc.), and a household-context tail capped at 900 bytes via UTF-8-safe `truncate_utf8`.
- The hardcoded `GENIE_RUNTIME_COMPACT_SYSTEM` blurb is replaced by the richer prefix. Eliminates a class of "model emits prose mentioning a tool name instead of a structured call" failures.

**4. Identity-recall fast-path** in `crates/genie-core/src/tools/quick.rs`. New `memory_recall_query` recognizes `"what is my name"`, `"do you remember my name"`, `"who am i"`, `"what do you remember about X"`, `"search memory for X"`, and similar, then dispatches directly to `memory_recall` without going through the LLM at all. Per the PR body, returns `"Your name is Jared"` in 21 ms on Jetson — bypasses the entire LLM round-trip for the most common identity questions. Two new unit tests pin both the name-form and the search-form. The old `does_not_route_memory_search_to_web` test is correctly *deleted* — the new behavior actively does route memory searches (just to `memory_recall`, not `web_search`).

Knock-on: this fix at the routing layer also robustly addresses issue #85 ("LLM hallucinated `Your name is GeniePod`"). Even on a runtime that doesn't see the tool manifest correctly, the identity question never reaches the LLM in the first place.

Tests:

- `genie_runtime_profile_compacts_runtime_prompt_under_4k_budget` (renamed from `_under_expanded_budget`) flips the asserted behavior to compaction-required at the new threshold, pins that the new structured system prompt survives and noisy text is dropped.
- `start_all_uses_configured_llm_backend` extended with three assertions covering `read_llm_url`, `wait_for_http_health`, and the new `Configured LLM health` echo line.
- `genie_ai_runtime_service_preserves_model_page_cache` extended to assert `GENIEPOD_AI_RUNTIME_CONTEXT=4096`.

End-to-end Jetson verification in PR body: runtime started with `Context: 4096 tokens`, `KV cache: 288 MB`, `FREE: 1104 MB`. Chat at port 3000 works with streaming. Memory recall fires `tool: memory_recall` in 21 ms.

All 8 CI checks green on `18bad34` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, shellcheck, ruff, PR body checklist).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate raising genie-ai-runtime context default to 8192

2 participants