setup: preserve warm genie-ai-runtime model cache - #70
Conversation
|
Jetson acceptance for #69: This confirms removing |
ai-hpc
left a comment
There was a problem hiding this comment.
Clean, narrowly-scoped fix that does exactly what #69 asks for. The original ExecStartPre=/bin/sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches' was over-eager: the comment justified it with "Jetson NvMap needs contiguous blocks", but the kernel's allocator compacts on demand and we don't need to pre-clear the cache to get a successful first load. Removing it makes restart-after-restart hit the warm path (Linux page cache holds the GGUF pages between runtime stops/starts since the file is still mmaped to a recent reader).
The genie-model-cache-status.sh helper is well-designed:
- Uses
mincore(2)via Pythonctypes— the right syscall for "is this page resident?". Returns per-page residency bits. mmapopens withACCESS_COPY(MAP_PRIVATE) so the probe is read-only — won't dirty pages or trigger writes.- Reads
[core].llm_model_pathfromgeniepod.tomlwith section-awareawkthat mirrors PR #63's pattern, so it can't accidentally match anllm_model_path =line outside[core](none exists today, but the discipline is correct). if [ "$(id -u)" -eq 0 ]; then AWK=(awk); else AWK=(sudo awk); fi— auto-sudos for the 0600-locked config so the helper works for non-root operators.- 0-byte file is handled cleanly (prints
0 / 0 MB cold, exit 0 — no division by zero). del bufbeforemm.close()is the right ctypes-cleanup idiom; without itmmap.close()fails with "cannot close exported pointers exist".- 95% / 1% thresholds for warm / cold are sensible; "partial" covers the in-between case during eviction pressure.
The four lock-in tests are smoke-tests rather than behavior tests, but they're exactly the right ones for this change — pin "no ExecStartPre= in the genie-ai-runtime service", pin "issue #69 referenced in the comment", pin "helper uses mincore + reports Resident:", pin "Makefile deploys the helper". A future PR that accidentally re-adds the drop-caches line or breaks the deploy path will fail CI loudly. The test that explicitly asserts the comment mentions "issue #69" is a nice touch — future archaeologists can trace the rationale back to the issue.
The end-to-end measurement on the operator's Jetson (linked in the PR comment / chat) is the proof: model cache residency is 0% → 100% on first start, then 100% across systemctl restart, then a subsequent jetson-llm interactive load reports Model loaded in 1161 ms (2051 MB/s) — the warm path, vs the ~30s cold path documented in the runtime ROADMAP. That's a ~25× restart speedup with zero downside on the typical 7.6 GB Orin Nano + Qwen3-4B Q4_K_M deployment.
One small note for a follow-up (not blocking): the new comment on the ExecStartPre line says what changed but the old "Jetson NvMap needs contiguous blocks" rationale lived for a reason. If memory-pressured deployments later report a cold-load regression due to fragmentation, a paragraph in the service file explaining "we used to drop caches; here's why we no longer think that's necessary" would help the next person tempted to add it back. Not blocking — the linked issue carries enough context.
All 7 checks green on a712973 including the new cargo clippy + test (--no-default-features) job from #62. Going in.
|
Merged at |
|
Additional service-path validation:
{"content":"Hello from GeniePod!","model":"Qwen3 4B Instruct Awq","usage":{"completion_tokens":7,"prompt_tokens":23,"total_tokens":30}}This confirms the normal service path preserves warm model weights and serves chat completions. Remaining board note: |
Fixes #75. Issue evidence on Jetson Orin Nano 8 GB shows the runtime's auto-clamp behavior is order-sensitive: same `-c 4096` request fits only ~1.7k context when the full stack is already resident, but fits 4k / 6k / 8k cleanly when `genie-ai-runtime` loads first. This PR pins the right startup order so the runtime claims its KV cache before memory-heavy services occupy DRAM. Three coordinated changes: 1. `Before=genie-whisper.service genie-whisper-warmup.service homeassistant.service genie-core.service` on `genie-ai-runtime.service`. systemd ordering directive that makes the LLM unit's load complete before the other memory-heavy services start. Combined with PR #72's existing `After=genie-ai-runtime.service genie-llm.service` on `genie-core`, the dependency is now bidirectional. `Before=` is a no-op for units that aren't installed on this host, so this is safe for installs that don't ship homeassistant or whisper. 2. `GENIEPOD_AI_RUNTIME_CONTEXT` default bumped from `2048` to `8192` — the largest context the issue verified loads cleanly with `--int8-kv` on Orin Nano 8 GB. The env knob stays settable via systemd drop-in for smaller Jetsons. 3. `deploy/scripts/start_all.sh` reorders the `UNITS=(...)` array so the configured LLM unit + warmup run before `homeassistant`, `genie-whisper`, `genie-whisper-warmup` in the manual lifecycle path too, mirroring the systemd `Before=`. Tests added to `tool_dispatch_test.rs` lock both invariants: - `start_all_uses_configured_llm_backend` asserts `$configured_llm_unit` appears before `homeassistant.service` and `genie-whisper.service` in the `UNITS=` array. - `genie_ai_runtime_service_preserves_model_page_cache` asserts `GENIEPOD_AI_RUNTIME_CONTEXT=8192` and that the new `Before=` clause is present. Compatibility with PR #70 (warm page cache across restart) is preserved — `Before=` only affects boot-time ordering, not `systemctl restart genie-ai-runtime` alone. End-user verified on the same Jetson the issue was filed against. Worth a follow-up: PR #74's `GENIE_RUNTIME_MAX_BODY_BYTES = 4 KB` body-compaction threshold is now leaving performance on the table at the new 8192-token runtime context (the client compacts prompts the runtime could now handle). Right path is to make the threshold a function of `GENIEPOD_AI_RUNTIME_CONTEXT` or probe runtime capacity at connection time. Not blocking this PR. All 7 CI checks green on `c1cae29` (fmt, clippy, test, aarch64 cross-compile, shellcheck, ruff, `--no-default-features`).
…es, swap, start (#94) Rewrites `deploy/scripts/genie-restart-all.sh` from a per-unit `systemctl restart` loop into a deliberate five-step memory reset, with a `--soft` flag that skips the cache + swap reset for operators who want to preserve the warm LLM cache from PR #70. Default (`--hard`) order: 1. `stop_all.sh` — stops every GeniePod systemd unit. 2. `pkill -x` — best-effort reap of any orphan subprocesses that escaped the cgroup stop (`piper`, `whisper-server`, `whisper-cli`, `jetson-llm-server`, `jetson-llm`, `llama-server`, `deep-filter`, `sox`, `ffmpeg`). Uses `-x` (exact basename) so unrelated processes with these strings in their full command line aren't matched. 3. `sync && echo 3 > /proc/sys/vm/drop_caches` — release page cache. 4. `swapoff -a && swapon -a` — flush the swap file to a clean baseline. `swapoff` requires enough free RAM to absorb the swap contents, which is why step 1 happens first. 5. `start_all.sh` — bring the stack back up. Trade-off vs PR #70 (warm-cache preservation across `systemctl restart genie-ai-runtime`): step 3 deliberately evicts the Qwen3-4B GGUF from page cache, so the first LLM request after `genie-restart-all.sh` hits the ~30 s cold-load path instead of the ~1.3 s warm path. That's intentional — this script exists for the post-`make deploy` case where binaries / config / model path may have changed and the prior warm cache is stale. Operators who only want a soft service refresh pass `--soft`. Robustness: - `set -uo pipefail` without `-e` so individual `pkill` / `swapoff` failures don't wedge the script mid-restart. - `swapoff -a` failures (no swap configured, or not enough free RAM to absorb the swap contents) are logged + skipped rather than fatal. - `drop_caches` write failures (running without sudo) are logged + skipped. - Missing `stop_all.sh` / `start_all.sh` produce a clear warning instead of an obscure exec error. Regression test `genie_restart_all_hard_mode_performs_full_memory_reset` in `crates/genie-core/tests/tool_dispatch_test.rs` pins both the presence of all five steps AND the strict ordering (stop → reap → drop_caches → swap → start). Markers are matched against strings that only appear in the imperative code section (echo lines + `"$STOP_ALL"` / `"$START_ALL"` invocations), not the doc-comment header, so a refactor that just shuffles comment text can't satisfy the test. All 8 CI checks green on `0ee70a8` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, shellcheck, ruff, PR body checklist).
Summary
Fixes #69.
This keeps Qwen3 model weights warm across
genie-ai-runtime.servicerestarts instead of forcing every start down the cold-load path.Changes:
drop_cachesExecStartPrefromgenie-ai-runtime.service./opt/geniepod/bin/genie-model-cache-status.shto report Linux page-cache residency for the configured GGUF usingmincore.make deploy-setup.Roadmap check
Checked
GeniePod/genie-ai-runtimeROADMAP before finalizing this. The runtime already has per-run model-load timing plusscripts/bench_load.sh, and the v1.0 notes call out:So this PR keeps the fix in GenieClaw deployment: do not evict the model from page cache immediately before starting the default runtime.
Validation
bash -n deploy/scripts/genie-model-cache-status.sh deploy/setup-jetson.sh deploy/scripts/start_all.sh deploy/scripts/stop_all.shbash deploy/scripts/genie-model-cache-status.sh deploy/config/geniepod.tomlmake -n deploy-setup JETSON_HOST=192.168.55.1 JETSON_USER=aihpccargo test -p genie-core --test tool_dispatch_test jetson_lifecycle_scripts_are_valid_shellcargo test -p genie-core --test tool_dispatch_test makefile_deploys_lifecycle_helperscargo test -p genie-core --test tool_dispatch_test genie_ai_runtime_service_preserves_model_page_cachecargo test -p genie-core --test tool_dispatch_test model_cache_status_helper_reports_residencygit diff --check