Skip to content

setup: preserve warm genie-ai-runtime model cache - #70

Merged
ai-hpc merged 1 commit into
GeniePod:mainfrom
Crystora:fix/issue-69-runtime-warm-weights
May 18, 2026
Merged

setup: preserve warm genie-ai-runtime model cache#70
ai-hpc merged 1 commit into
GeniePod:mainfrom
Crystora:fix/issue-69-runtime-warm-weights

Conversation

@Crystora

Copy link
Copy Markdown
Contributor

Summary

Fixes #69.

This keeps Qwen3 model weights warm across genie-ai-runtime.service restarts instead of forcing every start down the cold-load path.

Changes:

  • Remove the drop_caches ExecStartPre from genie-ai-runtime.service.
  • Add /opt/geniepod/bin/genie-model-cache-status.sh to report Linux page-cache residency for the configured GGUF using mincore.
  • Deploy the helper through make deploy-setup.
  • Print the helper in the Jetson setup footer.
  • Add static tests covering the service behavior and helper deployment.

Roadmap check

Checked GeniePod/genie-ai-runtime ROADMAP before finalizing this. The runtime already has per-run model-load timing plus scripts/bench_load.sh, and the v1.0 notes call out:

  • cold model load: ~30s, NVMe-bound
  • warm model load: ~1.3s, page-cache hit

So this PR keeps the fix in GenieClaw deployment: do not evict the model from page cache immediately before starting the default runtime.

Validation

  • bash -n deploy/scripts/genie-model-cache-status.sh deploy/setup-jetson.sh deploy/scripts/start_all.sh deploy/scripts/stop_all.sh
  • bash deploy/scripts/genie-model-cache-status.sh deploy/config/geniepod.toml
  • make -n deploy-setup JETSON_HOST=192.168.55.1 JETSON_USER=aihpc
  • cargo test -p genie-core --test tool_dispatch_test jetson_lifecycle_scripts_are_valid_shell
  • cargo test -p genie-core --test tool_dispatch_test makefile_deploys_lifecycle_helpers
  • cargo test -p genie-core --test tool_dispatch_test genie_ai_runtime_service_preserves_model_page_cache
  • cargo test -p genie-core --test tool_dispatch_test model_cache_status_helper_reports_residency
  • git diff --check

@Crystora

Copy link
Copy Markdown
Contributor Author

Jetson acceptance for #69:

== before start ==
Model: /opt/geniepod/models/Qwen3-4B-Q4_K_M.gguf
Size: 2382 MB
Resident: 0 / 2382 MB (0.0%) cold
== after first start ==
Model: /opt/geniepod/models/Qwen3-4B-Q4_K_M.gguf
Size: 2382 MB
Resident: 2382 / 2382 MB (100.0%) warm
== after restart ==
Model: /opt/geniepod/models/Qwen3-4B-Q4_K_M.gguf
Size: 2382 MB
Resident: 2382 / 2382 MB (100.0%) warm

This confirms removing drop_caches from genie-ai-runtime.service preserves Qwen3 GGUF page-cache residency across service restart. CI is green.

@ai-hpc ai-hpc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, narrowly-scoped fix that does exactly what #69 asks for. The original ExecStartPre=/bin/sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches' was over-eager: the comment justified it with "Jetson NvMap needs contiguous blocks", but the kernel's allocator compacts on demand and we don't need to pre-clear the cache to get a successful first load. Removing it makes restart-after-restart hit the warm path (Linux page cache holds the GGUF pages between runtime stops/starts since the file is still mmaped to a recent reader).

The genie-model-cache-status.sh helper is well-designed:

  • Uses mincore(2) via Python ctypes — the right syscall for "is this page resident?". Returns per-page residency bits.
  • mmap opens with ACCESS_COPY (MAP_PRIVATE) so the probe is read-only — won't dirty pages or trigger writes.
  • Reads [core].llm_model_path from geniepod.toml with section-aware awk that mirrors PR #63's pattern, so it can't accidentally match an llm_model_path = line outside [core] (none exists today, but the discipline is correct).
  • if [ "$(id -u)" -eq 0 ]; then AWK=(awk); else AWK=(sudo awk); fi — auto-sudos for the 0600-locked config so the helper works for non-root operators.
  • 0-byte file is handled cleanly (prints 0 / 0 MB cold, exit 0 — no division by zero).
  • del buf before mm.close() is the right ctypes-cleanup idiom; without it mmap.close() fails with "cannot close exported pointers exist".
  • 95% / 1% thresholds for warm / cold are sensible; "partial" covers the in-between case during eviction pressure.

The four lock-in tests are smoke-tests rather than behavior tests, but they're exactly the right ones for this change — pin "no ExecStartPre= in the genie-ai-runtime service", pin "issue #69 referenced in the comment", pin "helper uses mincore + reports Resident:", pin "Makefile deploys the helper". A future PR that accidentally re-adds the drop-caches line or breaks the deploy path will fail CI loudly. The test that explicitly asserts the comment mentions "issue #69" is a nice touch — future archaeologists can trace the rationale back to the issue.

The end-to-end measurement on the operator's Jetson (linked in the PR comment / chat) is the proof: model cache residency is 0% → 100% on first start, then 100% across systemctl restart, then a subsequent jetson-llm interactive load reports Model loaded in 1161 ms (2051 MB/s) — the warm path, vs the ~30s cold path documented in the runtime ROADMAP. That's a ~25× restart speedup with zero downside on the typical 7.6 GB Orin Nano + Qwen3-4B Q4_K_M deployment.

One small note for a follow-up (not blocking): the new comment on the ExecStartPre line says what changed but the old "Jetson NvMap needs contiguous blocks" rationale lived for a reason. If memory-pressured deployments later report a cold-load regression due to fragmentation, a paragraph in the service file explaining "we used to drop caches; here's why we no longer think that's necessary" would help the next person tempted to add it back. Not blocking — the linked issue carries enough context.

All 7 checks green on a712973 including the new cargo clippy + test (--no-default-features) job from #62. Going in.

@ai-hpc
ai-hpc merged commit daf9bfd into GeniePod:main May 18, 2026
7 checks passed
@ai-hpc

ai-hpc commented May 18, 2026

Copy link
Copy Markdown
Contributor

Merged at daf9bfd7ffddfa264539687df7999d8afdabdc8c.
Thanks to @bittoby!

@Crystora

Copy link
Copy Markdown
Contributor Author

Additional service-path validation:

  • genie-model-cache-status.sh reports Qwen3 fully warm: 2382 / 2382 MB (100.0%) warm.
  • genie-ai-runtime.service is active and /health returns status: ok for Qwen3 4B Instruct Awq.
  • OpenAI-compatible chat endpoint works:
{"content":"Hello from GeniePod!","model":"Qwen3 4B Instruct Awq","usage":{"completion_tokens":7,"prompt_tokens":23,"total_tokens":30}}

This confirms the normal service path preserves warm model weights and serves chat completions. Remaining board note: genie-core is still offline/auto-restarting and should be diagnosed separately from #69.

ai-hpc pushed a commit that referenced this pull request May 18, 2026
Fixes #75. Issue evidence on Jetson Orin Nano 8 GB shows the runtime's auto-clamp behavior is order-sensitive: same `-c 4096` request fits only ~1.7k context when the full stack is already resident, but fits 4k / 6k / 8k cleanly when `genie-ai-runtime` loads first. This PR pins the right startup order so the runtime claims its KV cache before memory-heavy services occupy DRAM.

Three coordinated changes:

1. `Before=genie-whisper.service genie-whisper-warmup.service homeassistant.service genie-core.service` on `genie-ai-runtime.service`. systemd ordering directive that makes the LLM unit's load complete before the other memory-heavy services start. Combined with PR #72's existing `After=genie-ai-runtime.service genie-llm.service` on `genie-core`, the dependency is now bidirectional. `Before=` is a no-op for units that aren't installed on this host, so this is safe for installs that don't ship homeassistant or whisper.
2. `GENIEPOD_AI_RUNTIME_CONTEXT` default bumped from `2048` to `8192` — the largest context the issue verified loads cleanly with `--int8-kv` on Orin Nano 8 GB. The env knob stays settable via systemd drop-in for smaller Jetsons.
3. `deploy/scripts/start_all.sh` reorders the `UNITS=(...)` array so the configured LLM unit + warmup run before `homeassistant`, `genie-whisper`, `genie-whisper-warmup` in the manual lifecycle path too, mirroring the systemd `Before=`.

Tests added to `tool_dispatch_test.rs` lock both invariants:

- `start_all_uses_configured_llm_backend` asserts `$configured_llm_unit` appears before `homeassistant.service` and `genie-whisper.service` in the `UNITS=` array.
- `genie_ai_runtime_service_preserves_model_page_cache` asserts `GENIEPOD_AI_RUNTIME_CONTEXT=8192` and that the new `Before=` clause is present.

Compatibility with PR #70 (warm page cache across restart) is preserved — `Before=` only affects boot-time ordering, not `systemctl restart genie-ai-runtime` alone.

End-user verified on the same Jetson the issue was filed against.

Worth a follow-up: PR #74's `GENIE_RUNTIME_MAX_BODY_BYTES = 4 KB` body-compaction threshold is now leaving performance on the table at the new 8192-token runtime context (the client compacts prompts the runtime could now handle). Right path is to make the threshold a function of `GENIEPOD_AI_RUNTIME_CONTEXT` or probe runtime capacity at connection time. Not blocking this PR.

All 7 CI checks green on `c1cae29` (fmt, clippy, test, aarch64 cross-compile, shellcheck, ruff, `--no-default-features`).
ai-hpc added a commit that referenced this pull request May 18, 2026
…es, swap, start (#94)

Rewrites `deploy/scripts/genie-restart-all.sh` from a per-unit `systemctl restart` loop into a deliberate five-step memory reset, with a `--soft` flag that skips the cache + swap reset for operators who want to preserve the warm LLM cache from PR #70.

Default (`--hard`) order:

1. `stop_all.sh` — stops every GeniePod systemd unit.
2. `pkill -x` — best-effort reap of any orphan subprocesses that escaped the cgroup stop (`piper`, `whisper-server`, `whisper-cli`, `jetson-llm-server`, `jetson-llm`, `llama-server`, `deep-filter`, `sox`, `ffmpeg`). Uses `-x` (exact basename) so unrelated processes with these strings in their full command line aren't matched.
3. `sync && echo 3 > /proc/sys/vm/drop_caches` — release page cache.
4. `swapoff -a && swapon -a` — flush the swap file to a clean baseline. `swapoff` requires enough free RAM to absorb the swap contents, which is why step 1 happens first.
5. `start_all.sh` — bring the stack back up.

Trade-off vs PR #70 (warm-cache preservation across `systemctl restart genie-ai-runtime`): step 3 deliberately evicts the Qwen3-4B GGUF from page cache, so the first LLM request after `genie-restart-all.sh` hits the ~30 s cold-load path instead of the ~1.3 s warm path. That's intentional — this script exists for the post-`make deploy` case where binaries / config / model path may have changed and the prior warm cache is stale. Operators who only want a soft service refresh pass `--soft`.

Robustness:

- `set -uo pipefail` without `-e` so individual `pkill` / `swapoff` failures don't wedge the script mid-restart.
- `swapoff -a` failures (no swap configured, or not enough free RAM to absorb the swap contents) are logged + skipped rather than fatal.
- `drop_caches` write failures (running without sudo) are logged + skipped.
- Missing `stop_all.sh` / `start_all.sh` produce a clear warning instead of an obscure exec error.

Regression test `genie_restart_all_hard_mode_performs_full_memory_reset` in `crates/genie-core/tests/tool_dispatch_test.rs` pins both the presence of all five steps AND the strict ordering (stop → reap → drop_caches → swap → start). Markers are matched against strings that only appear in the imperative code section (echo lines + `"$STOP_ALL"` / `"$START_ALL"` invocations), not the doc-comment header, so a refactor that just shuffles comment text can't satisfy the test.

All 8 CI checks green on `0ee70a8` (fmt, clippy, test, aarch64 cross-compile, `--no-default-features`, shellcheck, ruff, PR body checklist).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate slow Qwen3 model loading and warm weight behavior

2 participants