feat(llm): add remote embedding/reranking via OpenAI-compatible endpoints

[alexei-led]

Problem

QMD is designed as an on-device tool, but its value — BM25 + vector search + query expansion + LLM reranking in a single pipeline — makes it equally useful in cloud and multi-agent environments. The current hard dependency on local GGUF models via node-llama-cpp creates two problems in those settings:

1. Performance. Embedding 300M–0.6B GGUF models on CPU is slow; in cloud environments there is often no GPU at all.
2. Concurrency. node-llama-cpp serialises all inference calls through a single context. Running multiple agents or workers against the same index causes queuing and timeouts — each waits for the previous embed/rerank call to finish before starting.

Both problems go away when the embedding and reranking workload moves to a dedicated remote server (vLLM, TEI, Ollama, LiteLLM, or any OpenAI-compatible endpoint).

This closes #521 and addresses the use case in #229.

---

What this adds

Two new classes, plus the plumbing to activate them via environment variables.

RemoteLLM

A drop-in LLM implementation that calls an OpenAI-compatible HTTP server:

Capability	Endpoint
Embedding	POST /v1/embeddings
Reranking	POST /v1/rerank (cross-encoder format)
Query expansion / generation	POST /v1/chat/completions

Features:

• Independent circuit breakers per endpoint (embed, rerank, chat) — an embed outage does not affect reranking or chat
• Configurable connect/read timeouts and breaker thresholds
• Bearer auth via QMD_REMOTE_API_KEY
• qmd status shows the remote server URLs instead of local model paths
• RemoteLLM.modelExists() logs a warning before returning the optimistic fail-open result when the server can't be reached, so operators know the check was skipped
• Character-based token approximation (~4 chars/token) keeps chunking and overflow protection working without a local tokenizer

HybridLLM

A thin composite that delegates all operations to RemoteLLM. Designed for future extension (e.g. local generate + remote embed), but in the current implementation runs fully remote when the local backend is null.

Changes to existing code

src/llm.ts

• LLM interface: adds embedBatch(), tokenize(), countTokens(), detokenize(), isRemote?, embedModelName?
• LLMSessionManager and withLLMSessionForLlm accept LLM instead of LlamaCpp only
• New exports: getDefaultLLM() / setDefaultLLM() — sit alongside getDefaultLlamaCpp() / setDefaultLlamaCpp() for backward compatibility
• LlamaCpp.tokenize / detokenize use unknown[] to satisfy the interface while still casting internally to LlamaToken[]

src/store.ts (~45 lines changed)

• getLlm() returns LLM and falls back to getDefaultLLM()
• generateEmbeddings: uses formatDoc() — skips the Qwen3 task-prefix for remote backends, which don't need it
• chunkDocumentByTokens: accepts an optional llm? parameter so internal callers pass the store-scoped LLM rather than always pulling from the global singleton

src/cli/qmd.ts (~52 lines changed)

• Reads QMD_REMOTE_EMBED_URL / QMD_REMOTE_RERANK_URL / QMD_REMOTE_GEN_URL at startup and builds a HybridLLM if set
• Guards the YAML models: override in getStore() — skips setDefaultLlamaCpp when remote mode is already active, so a models: block in the config cannot silently revert to local

src/index.ts (~46 lines changed)

• StoreOptions.llm? — inject a backend directly (useful for testing or custom integrations)
• createStore() checks QMD_REMOTE_* env vars automatically when no llm option is provided, so SDK users get the same remote mode as the CLI

---

Usage

# Minimal: TEI for embeddings and reranking
export QMD_REMOTE_EMBED_URL=http://localhost:8080
export QMD_REMOTE_RERANK_URL=http://localhost:8081
qmd embed && qmd query "how does auth work?"

# Separate chat server for query expansion
export QMD_REMOTE_GEN_URL=http://localhost:8082

# Authentication (sent as Bearer token to all endpoints)
export QMD_REMOTE_API_KEY=sk-...

All QMD_REMOTE_* variables are optional — when unset, qmd falls back to the existing local GGUF behaviour with no behavioural change.

Variable	Default
QMD_REMOTE_EMBED_URL	— (required to enable remote mode)
QMD_REMOTE_RERANK_URL	— (required to enable remote mode)
QMD_REMOTE_GEN_URL	QMD_REMOTE_EMBED_URL
QMD_REMOTE_API_KEY	—
QMD_REMOTE_EMBED_MODEL	remote-embedding
QMD_REMOTE_RERANK_MODEL	remote-reranker
QMD_REMOTE_GEN_MODEL	gpt-4o-mini
QMD_REMOTE_CONNECT_TIMEOUT	5000 ms
QMD_REMOTE_READ_TIMEOUT	30000 ms

---

Testing

test/remote-llm.test.ts — 19 tests, all in-process with real HTTP servers (no mocks):

• Embed and rerank routing to separate URLs
• Qwen3 instruct prefix added for query embeddings; legacy title: | text: prefixes stripped on receipt
• Embedding dimension lock and error on mismatch
• Circuit breaker opens after N failures; embed, rerank, and chat circuits are independent
• Half-open state retries after cooldown
• Bearer auth header
• Connect and read timeouts
• generate calls chat completions and returns text
• expandQuery parses typed lines; falls back to lex+vec+hyde on error
• tokenize / countTokens / detokenize character-approximation
• HybridLLM routes all operations to remote without as any casts

---

Backward compatibility

No breaking changes. All new behaviour is opt-in via environment variables. Existing local-only setups are unaffected.

[Copilot]

Pull request overview

Adds an opt-in remote LLM backend so QMD can run embeddings, reranking, and generation against OpenAI-compatible HTTP endpoints (improving performance and concurrency in cloud/multi-agent setups), while keeping local GGUF behavior as the default.

Changes:

• Introduces RemoteLLM (OpenAI-compatible /v1/embeddings, /v1/rerank, /v1/chat/completions) plus HybridLLM wiring.
• Plumbs remote-mode activation via QMD_REMOTE_* env vars through CLI and SDK (createStore()), and updates store logic to use the generalized LLM interface.
• Adds in-process HTTP tests for Remote/Hybrid behavior and updates dist artifacts + changelog.

Reviewed changes

Copilot reviewed 8 out of 41 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/remote-llm.ts	New remote OpenAI-compatible implementation with timeouts + circuit breakers.
src/hybrid-llm.ts	New composite LLM wrapper to route operations to remote (future hybridization).
src/store.ts	Switches store to LLM, adds remote-specific formatting paths, and passes store LLM into chunking.
src/cli/qmd.ts	CLI env-var activation of remote mode + status output adjustments.
src/index.ts	SDK createStore() supports llm injection and auto-remote via env vars.
test/remote-llm.test.ts	Comprehensive in-process tests for routing, breakers, timeouts, auth, token approximation.
CHANGELOG.md	Documents new remote mode and related interface changes.
.gitignore	Stops ignoring dist/ to allow GitHub install without a build step.
dist/*	Updated build outputs to match source changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[alexei-led]

Fixed. Changed getDefaultLlamaCpp() to getDefaultLLM() in chunkDocumentByTokens — this ensures the remote backend is used for tokenization when no explicit override is provided, rather than falling back to load local GGUF models.

[jonesj38]

see #116

[lukeboyett]

FYI — we've been running a much narrower variant of this in prod (lukeboyett/qmd:feat/openai-embed-backend): OpenAI embeddings only, activated by OPENAI_API_KEY alone, no new files, no rerank/expansion changes. ~300 net lines. It's a strict subset of what this PR covers, so no reason to open it upstream if #575 or #517 lands. Happy to share it as prior art if useful for comparing approaches.

[alexei-led]

Whatever approach works for me. I just need to be able to use remote embedding engine