GENIEPOD_AI_RUNTIME_CONTEXT=8192 destabilizes the stack under steady-state load (chat UI lag + swap engagement)

[ai-hpc]

Summary

GENIEPOD_AI_RUNTIME_CONTEXT=8192 (the alpha.9 default landed in PR #76, with --int8-kv) is destabilizing the box under normal operation. Symptoms observed on Jetson Orin Nano 8 GB with the full stack running:

• Chat UI on :3000 becomes unstable — slow / hanging requests, occasional empty responses or 5xx, requests that used to be <1 s now occasionally take many seconds.
• Other operations stutter — :3080 dashboard polls intermittently fail or lag, genie-ctl status slow, voice cycle latency banner shows higher variance than the alpha.7 baseline.
• Swap file is in active use — free -h shows non-zero Swap: used, which is the unambiguous signal that the working set has overflowed RAM. With swap engaged, every memory access has a chance of paging from disk, which compounds latency for everything (CUDA dispatch, HTTP handlers, audio capture).

Why this is a real regression vs PR #76's verification

PR #76 raised GENIEPOD_AI_RUNTIME_CONTEXT from 2048 → 8192 based on issue #75's evidence that the runtime could fit 8192 tokens with INT8 KV when genie-ai-runtime loaded first (Before= whisper / homeassistant / genie-core). That measurement was a cold-boot best case — drop_caches, no resident services, then start genie-ai-runtime alone. Issue #75's table confirmed this explicitly:

ctx	Loaded	KV cache	Runtime budget free	Note
4096	4096	288 MB	1903 MB	(genie-ai-runtime first)
6144	6144	432 MB	992 MB	(genie-ai-runtime first)
8192	8192	576 MB	1616 MB	(genie-ai-runtime first)
4096	1735	(clamped)	(~tight)	after full stack was already running

The 8192-with-1616-MB-free measurement assumed nothing else was holding RAM at runtime startup. In steady-state operation — whisper resident + genie-core's HTTP server + piper + deep-filter + Home Assistant container + dashboards + Telegram adapter + tokio runtimes — the actual free RAM after genie-ai-runtime claims its full 8192-ctx KV pool is materially less than that 1616 MB. Once free RAM drops below the kernel's reclaim watermark, swap engages, and everything degrades.

Reproduction (representative)

On a Jetson Orin Nano 8 GB running alpha.9 main (post-PR #76), after make deploy + bash /opt/geniepod/setup-jetson.sh + reboot to ensure boot order is what PR #76 designed:

ssh aihpc@192.168.55.1
free -h        # baseline; check Swap: used == 0 immediately after boot
sudo systemctl start geniepod.target
sleep 60       # let warmup land, dashboard register, etc.
free -h        # check Swap: used after the stack is fully resident

Reproduces non-zero Swap: used and audible / measurable lag in the chat UI for me.

Acceptance criteria

• Steady-state operation with the full default stack (genie-ai-runtime + genie-whisper + genie-core + genie-api + genie-health + genie-governor + genie-mqtt + homeassistant) on Orin Nano 8 GB does not engage swap during normal use.
• Chat UI at :3000 returns within the alpha.7-baseline latency band on every request, not just the first one.
• The first-reply latency banner from Print first-voice-reply latency banner at startup #19 shows the same ±100 ms median variance bound that PR deploy: pin genie-* services to dedicated CPU cores for voice latency… #48 (CPU pinning / core isolation for voice subsystem latency stability #25) targets after CPU pinning lands.

Suggested directions (pick one or combine)

A. Drop the default context back to a safer steady-state value

Easiest fix: lower GENIEPOD_AI_RUNTIME_CONTEXT from 8192 to 4096 (or 5120 if benchmarking shows it fits). #75's data has 4096-context-cold-boot fitting in 1903 MB of runtime budget free, which has more headroom for the rest of the stack to share. Combined with PR #74's body-compaction threshold (24 KB for typical English chat), 4096 tokens is still plenty for the web-chat path that motivated raising the context in the first place.

Trade-off: voice-loop conversations that span many turns get less LLM context. For a home appliance with command-style voice this is fine; for a long discursive chat it's a regression on multi-turn memory.

B. Make the context size auto-shrink under memory pressure

Probe free (or /proc/meminfo) at genie-ai-runtime.service start and pick the largest context that leaves a safety margin (e.g., 1.5 GB free post-load). setup-jetson.sh could compute a per-host Environment=GENIEPOD_AI_RUNTIME_CONTEXT=<N> drop-in based on MemTotal and the resident size of the rest of the enabled stack.

Trade-off: more complexity, more failure modes. Probably right long-term but heavier to land.

C. Detect swap engagement and report it on the dashboard

Doesn't fix the cause but surfaces the symptom. genie-health could probe /proc/swaps and /proc/meminfo periodically and surface swap_used_kb > 0 as a yellow dashboard warning. Operators would at least see why their box is slow.

Probably worth doing alongside A or B — even after dropping the context, swap-engagement is a useful health signal for future tuning regressions.

Cross-references

• Evaluate raising genie-ai-runtime context default to 8192 #75 — original benchmark that justified 8192 (cold-boot measurement only).
• PR setup: start genie-ai-runtime before memory-heavy services #76 — landed 8192 + --int8-kv + Before= ordering as the new default. My review comment on that PR flagged "Smaller-Jetson safety: an Orin Nano 4 GB or earlier Jetson without --int8-kv support could fail to allocate 8192 ctx at boot." — this issue is the same shape on the 8 GB Jetson under steady-state load.
• PR core: shape genie-ai-runtime chat requests #74 — GENIE_RUNTIME_MAX_BODY_BYTES = 24 KB body-compaction threshold on the client side. Already tracks the post-setup: start genie-ai-runtime before memory-heavy services #76 context size; would need to be revisited if option A drops the context.
• Investigate slow Qwen3 model loading and warm weight behavior #69 / PR setup: preserve warm genie-ai-runtime model cache #70 — page-cache preservation across runtime restart. Relevant because swap engagement may evict warm GGUF pages, which would re-amplify the latency cost of any LLM call.
• PR deploy: pin genie-* services to dedicated CPU cores for voice latency… #48 — CPU pinning. Reduces scheduler jitter but doesn't help when the real bottleneck is paging from disk.

Notes for the operator

If swap is already engaged on your box and you want to test the hypothesis before any code change:

ssh aihpc@192.168.55.1
sudo systemctl stop geniepod.target
sudo swapoff -a && sudo sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' && sudo swapon -a
# Edit the drop-in:
sudo mkdir -p /etc/systemd/system/genie-ai-runtime.service.d
echo -e '[Service]\nEnvironment=GENIEPOD_AI_RUNTIME_CONTEXT=4096' \
  | sudo tee /etc/systemd/system/genie-ai-runtime.service.d/local-ctx.conf
sudo systemctl daemon-reload
sudo systemctl start geniepod.target
sleep 60
free -h
# If Swap: used stays at 0 and the chat UI is responsive, this issue is confirmed.

/opt/geniepod/bin/genie-restart-all.sh (PR #94) does the stop+drop_caches+swapoff+swapon+start sequence as a single command — useful for repeating the experiment cleanly.

[Crystora]

@/tmp/issue107-comment.md