Default to genie-ai-runtime (v0.1.0-alpha.8) once #32 lands

[ai-hpc]

Background

genie-ai-runtime is explicitly the long-term replacement for llama-server in this stack:

• README.md L85 — "genie-ai-runtime: Jetson-only C++ LLM runtime customized from llama.cpp"
• ARCHITECTURE.md L70 — "llama.cpp fork, CUDA kernels, model memory planner: genie-ai-runtime"
• ARCHITECTURE.md L81 — "llama.cpp OpenAI-compatible client → genie-ai-runtime client"

The runtime already ships a drop-in jetson-llm-server binary with the same /v1/chat/completions shape that LlmClient (crates/genie-core/src/llm/client.rs:6) speaks. Per the runtime's own README, it is positioned as "drop-in replacement target for llama-server" with API shape "closely enough" to allow GenieClaw to swap backends via configuration.

Performance — reported on identical Jetson Orin Nano Super hardware:

Metric	llama.cpp	genie-ai-runtime alpha.8
Prefill	17.97 tok/s	38.68 tok/s (+115%)
Time-to-first-token	baseline	−61% vs runtime's own alpha.2

This is the parity-flip ticket.

Dependency

Blocked by #32 (LLM backend abstraction). When #32 lands, default backend selection becomes a single TOML key, and this issue is the one that flips that key.

Concrete proposal

When #32 lands:

1. Change the default value of [services.llm].backend from \"llama_cpp\" to \"genie_ai_runtime\" in deploy/config/geniepod.toml.
2. Pin the runtime version: target release v0.1.0-alpha.8 (or latest at flip time).
3. Default model: Qwen3-4B-Q4_K_M.gguf (the runtime's validated model).
4. Hardware target unchanged — Jetson Orin Nano Super 8 GB. The runtime explicitly does not support x86, discrete GPUs, Windows, or macOS. Dev-machine builds must continue to default to llama_cpp.

Parity gate — verify, don't trust

The runtime's drop-in claim is "closely enough". We should validate before flipping default. Acceptance gate:

• API parity — identical chat and chat_stream responses for a fixed prompt set (10 prompts × 3 temperatures), comparing token-by-token streaming and final string equality.
• Streaming SSE shape — same data: framing, same [DONE] terminator, same delta structure as parsed in client.rs chat_stream.
• Error shape — HTTP 400 / 503 bodies parse identically through llama_error_message() (client.rs:419). If field names differ, normalize them in the new client.
• Voice-loop end-to-end — full record → STT → llm → TTS cycle on Orin Nano with genie-ai-runtime backend, measuring p50 / p95 first-audio latency vs llama.cpp baseline. Expect ≥30% improvement on first-audio based on the +115% prefill claim.
• Memory budget — combined genie-ai-runtime + whisper-server + piper-tts + genie-core fits in 8 GB on Orin Nano. The runtime's own README cites a 7.6 GB iGPU budget; verify with tegrastats under load.
• Failure modes — process death, OOM, GPU memory pressure all surface through the same health() path that genie-health.service polls.
• Governor pressure modes — night_model_swap, stop_optins_mb, and mode-swap paths in genie-governor work identically against the new backend.

New systemd unit

deploy/systemd/genie-ai-runtime.service, modeled on the existing genie-llm.service:

[Unit]
Description=GeniePod AI Runtime (Jetson-tuned LLM)
Documentation=https://github.com/GeniePod/genie-ai-runtime
After=network.target
ConditionPathExists=/opt/geniepod/bin/jetson-llm-server

[Service]
Type=simple
ExecStartPre=/bin/sh -c 'sync && echo 3 > /proc/sys/vm/drop_caches'
ExecStart=/opt/geniepod/bin/jetson-llm-server \\
    -m \${GENIEPOD_LLM_MODEL} \\
    -p 8080
Environment=GENIEPOD_LLM_MODEL=/opt/geniepod/models/qwen3-4b-q4_k_m.gguf
Restart=on-failure
RestartSec=5
TimeoutStartSec=120
ProtectSystem=no
SupplementaryGroups=video render

[Install]
WantedBy=geniepod.target

Plus a matching genie-ai-runtime-warmup.service mirroring genie-llm-warmup.service (one-shot, sends a tiny prompt to force model load + CUDA kernel compile into iGPU before the first user-visible voice cycle).

Deploy pipeline changes

• make deploy drops jetson-llm-server and jetson-llm binaries into /opt/geniepod/bin/ (cross-compiled from genie-ai-runtime source, or fetched from the upstream v0.1.0-alpha.8 release artifact).
• setup-jetson.sh downloads the validated qwen3-4b-q4_k_m.gguf to /opt/geniepod/models/ if not present (alongside the existing Whisper model fetch).
• Two new systemd units (genie-ai-runtime.service, genie-ai-runtime-warmup.service) ship in deploy/systemd/.
• The legacy genie-llm.service and genie-llm-warmup.service remain in the repo; they are not removed, just no longer the default.

Rollback story

Because backend selection is a single TOML key after #32, regression rollback is a one-line revert:

[services.llm]
backend = \"llama_cpp\"   # rollback if genie-ai-runtime regresses

No binary rebuild, no schema migration, no data loss. This is the right shape for a parity flip.

Why this is worth doing

• README.md and ARCHITECTURE.md already commit to this direction; the flip is the natural alpha.8 milestone.
• +115% prefill / −61% TTFT on the same hardware gives us either tighter voice-cycle latency or headroom for a larger model in the same memory budget.
• Once genie-ai-runtime is the default and llama.cpp is optional, every other subsystem (governor pressure modes, warmup, health, mode-swap) can be reasoned about against one canonical runtime.

Acceptance criteria

• All parity-gate bullets above pass
• Default in deploy/config/geniepod.toml is backend = \"genie_ai_runtime\"
• Dev config geniepod.dev.toml remains backend = \"llama_cpp\" (x86 / mac dev machines)
• setup-jetson.sh provisions the runtime binaries and the validated model
• Documentation updated: README How-It-Works diagram, ARCHITECTURE.md §3 backends section
• CHANGELOG entry under alpha.8

Related

• Required-first issue Make llama.cpp an optional LLM backend (introduce LlmBackend abstraction) #32 (LLM backend abstraction)
• Upstream runtime: https://github.com/GeniePod/genie-ai-runtime
• Runtime release: https://github.com/GeniePod/genie-ai-runtime/releases (target v0.1.0-alpha.8)

[ai-hpc]

Closing as stale. Two things moved under this issue's feet:

1. genie-ai-runtime shipped v1.0.0 (2026-05-16) — this issue pinned v0.1.0-alpha.8 and cited alpha-era perf numbers (+115% prefill / −61% TTFT vs alpha.2). The v1.0.0 release sheet shows substantially better numbers — Prefill 38.0 tok/s, Decode 9.9 tok/s, Cold TTFT 877 ms, Warm-turn TTFT 444 ms on Qwen3-4B Q4_K_M / Orin Nano Super — and INT8 KV cache became the default. The acceptance criteria here no longer match the runtime's shipped surface.
2. Make llama.cpp an optional LLM backend (introduce LlmBackend abstraction) #32 (the prerequisite) has fully landed. All five abstraction slices (llm: introduce backend facade #35, voice: report configured llm backend #38, core: expose configured llm backend #39, runtime: resolve llm service from config #40, llm: select backend from config #43) are on main, plus the docs/config cleanup at 339f84f. [services.llm].backend = "genie_ai_runtime" is selectable end-to-end today.

The default-flip work is still wanted — just against v1.0.0 with current numbers. Replacement issue: #52.