feat: remote model server (qmd serve) for shared inference across clients

[jaylfc]

Summary

Adds qmd serve — a lightweight HTTP server that exposes embedding, reranking, and query expansion via a JSON API. Designed for multi-client setups where multiple QMD instances (e.g. agents in LXC containers) share loaded models over the network.

Supports two backends:

• local (default) — loads GGUF models via node-llama-cpp (CPU/Vulkan)
• rkllama — proxies to an rkllama NPU server (RK3588/RK3576)

Problem

QMD requires node-llama-cpp for all embedding/reranking operations, which:

• Fails on ARM64 (no Vulkan SDK, CMake compilation fails)
• Can't share loaded models across multiple agents
• Loads models per-process (wasteful on memory-constrained devices)

This is a superset of the functionality requested in #489 and attempted in #490, #480, and #116.

Solution

Server (qmd serve)

qmd serve --port 7832 --bind 0.0.0.0
qmd serve --backend rkllama --rkllama-url http://localhost:8080

Endpoints:

Endpoint	Method	Description
/embed	POST	Embed a single text
/embed-batch	POST	Batch embed multiple texts
/rerank	POST	Rerank documents by relevance
/expand	POST	Expand a query (lex/vec/hyde)
/tokenize	POST	Count tokens in text
/health	GET	Server status + loaded models
/status	GET	Index health (doc counts, embedding status)
/collections	GET	List collections with doc counts
/search?q=X	GET	FTS5 keyword search
/browse	GET	Paginated chunk listing

Client (RemoteLLM)

Drop-in replacement for LlamaCpp that forwards all calls to a remote qmd serve instance:

export QMD_SERVER=http://192.168.6.123:7832
qmd embed          # uses remote server, no local model loading
qmd query "search" # full hybrid search via remote

Auto-detected: if QMD_SERVER is set, skips local LLM initialization entirely. Zero compilation, instant startup.

Testing

Tested extensively on Orange Pi 5 Plus (RK3588, 16GB) running 3 OpenClaw agents in LXC containers sharing one qmd serve instance:

• Embed: 0.3s per chunk via NPU (was 3-5s on CPU)
• Rerank: 1.8-2.2s via NPU (was 10-15s on CPU)
• Batch embed: All chunks in single HTTP call, reduces overhead
• 100% embedding completion on 900KB+ transcripts with retry logic
• Rankings verified identical to standard QMD on x86+RTX 3060

Key fixes included

• Batch embedding — sends all chunks in one rkllama call (reduces HTTP overhead)
• Error rate threshold — 99% with 4x minimum sample for large docs
• Increased timeouts — 5 min for batch operations on ARM CPU
• KV cache workaround — documents the rkllama KV cache issue that causes embed degradation

Backwards compatible

• No changes to existing CLI behavior when QMD_SERVER is not set
• qmd serve is a new command, doesn't affect existing commands
• RemoteLLM implements the same LLM interface as LlamaCpp

Related: #489, #490, #480, #116

🤖 Generated with Claude Code

[paralizeer]

Great PR @jaylfc — we've been running this on our ARM64 VPS (Oracle Cloud Ampere) since you posted in #489 and it's solid. The RemoteLLM drop-in is clean, and qmd serve solves a real problem for multi-agent deployments.

We did a thorough review and found a few things worth addressing:

Security: default bind should be 127.0.0.1 (not 0.0.0.0)

Both serve.ts (line 345) and qmd.ts (line 3096) default to binding on all interfaces. This exposes the model server to the entire network out of the box. Since most users will run this on localhost or private networks, defaulting to 127.0.0.1 is safer — users who need network access can explicitly pass --bind 0.0.0.0.

- const bind = options.bind ?? "0.0.0.0";
+ const bind = options.bind ?? "127.0.0.1";

Request body size limit

readBody() accumulates unlimited data, which could OOM the server. A simple 50MB cap:

const MAX_BODY_BYTES = 50 * 1024 * 1024;

function readBody(req: IncomingMessage): Promise<string> {
  return new Promise((resolve, reject) => {
    const chunks: Buffer[] = [];
    let totalBytes = 0;
    req.on("data", (chunk: Buffer) => {
      totalBytes += chunk.length;
      if (totalBytes > MAX_BODY_BYTES) {
        req.destroy();
        reject(new Error(`Request body exceeds ${MAX_BODY_BYTES / 1024 / 1024}MB limit`));
        return;
      }
      chunks.push(chunk);
    });
    req.on("end", () => resolve(Buffer.concat(chunks).toString("utf-8")));
    req.on("error", reject);
  });
}

Input type validation

The POST endpoints check for existence (!text) but not types. This means {"text": 42} passes validation but fails with a confusing stack trace. Using typeof text !== "string" catches bad input cleanly:

- if (!text) {
-   json(res, 400, { error: "text is required" });
+ if (typeof text !== "string" || text.length === 0) {
+   json(res, 400, { error: "text must be a non-empty string" });

Same pattern for /embed-batch (validate array elements are strings), /rerank (validate non-empty documents), and /expand.

README cleanup for upstream

The PR still has fork-specific framing ("Fork note", OpenClaw references in the README). For upstream, the "Remote Model Server" section should be presented as a first-class feature, not a fork addition. The OpenClaw-specific systemd example could be generalized.

---

Everything else looks good: parameterized SQL in /browse ✅, retry logic with backoff for SBC/NPU ✅, LLM interface abstraction ✅, graceful shutdown ✅, setDefaultLLM global swap ✅.

We've already applied the security fixes locally and confirmed they work. Happy to test any further changes you push.

[jaylfc]

Thanks for the thorough review @paralizeer — all four points addressed in 69df17d:

1. Default bind → 127.0.0.1 ✅ Both serve.ts and the CLI default changed. Users explicitly opt-in to network exposure with --bind 0.0.0.0.

2. Request body size limit ✅ Added 50MB cap to readBody() — destroys the request if exceeded instead of accumulating to OOM.

3. Input type validation ✅ All POST endpoints now use typeof text !== "string" || text.length === 0 checks. /embed-batch validates array elements are strings. /rerank validates non-empty documents array. /tokenize and /expand also tightened.

4. README cleaned up ✅ Removed fork-specific framing, generalized the OpenClaw systemd example to a generic agent/container integration section, added rkllama NPU backend to the examples.

Appreciate you testing this on your ARM64 setup — good to know it's solid on Oracle Cloud Ampere too.