Feature request: Support remote Ollama embeddings via HTTP (OLLAMA_EMBED_URL)

[paralizeer]

Problem

QMD 2.0 uses node-llama-cpp for all embedding and tokenization operations. This requires local compilation via CMake on every run, which fails on platforms without GPU/Vulkan support (e.g., ARM64 VPS, Docker containers, CI runners) and is unusable when the Ollama instance runs on a separate machine (common with Tailscale, Docker networks, or dedicated GPU boxes).

The OLLAMA_EMBED_URL env var exists but only applies to qmd vsearch partially — expandQuery(), chunkDocumentByTokens(), generateEmbeddings(), and the vsearch CLI command all still require node-llama-cpp and trigger CMake compilation.

Use Case

Running QMD on an ARM64 Oracle Cloud VPS with Ollama on a separate machine (connected via Tailscale). The Ollama instance serves qwen3-embedding:0.6b (1024 dims, MTEB #1 multilingual). There is no local GPU and no way to compile node-llama-cpp cleanly.

This is a common setup for anyone using:

• Remote Ollama (Docker, Tailscale, LAN)
• ARM64 servers (Oracle, Ampere, Graviton)
• Headless VPS without GPU drivers

Proposed Solution

When OLLAMA_EMBED_URL is set, bypass all node-llama-cpp / getDefaultLlamaCpp() calls:

1. ollamaEmbed() helper function

async function ollamaEmbed(text: string): Promise<EmbeddingResult> {
  const url = process.env.OLLAMA_EMBED_URL;
  const model = process.env.OLLAMA_EMBED_MODEL || "nomic-embed-text";
  const res = await fetch(`${url}/api/embed`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ model, input: text }),
  });
  const data = await res.json();
  return { embedding: data.embeddings[0], model };
}

2. Patch points (6 locations)

Function	File	What to bypass
getEmbedding()	store.ts	getDefaultLlamaCpp() → ollamaEmbed()
generateEmbeddings()	store.ts	withLLMSessionForLlm → direct Ollama HTTP
expandQuery()	store.ts	LLM query expansion → pass-through [{type:"vec", query}]
chunkDocumentByTokens()	store.ts	llm.tokenize() → char-based estimation (text.length / 3)
embedBatch (2 sites)	store.ts	llm.embedBatch() → ollamaEmbedBatch()
vectorSearch CLI	cli/qmd.ts	withLLMSession() → direct call

3. Environment variables

export OLLAMA_EMBED_URL=http://your-ollama:11434
export OLLAMA_EMBED_MODEL=qwen3-embedding:0.6b  # optional, defaults to nomic-embed-text

Results (tested on ARM64 Oracle VPS)

• Before: Every vsearch/embed/query triggers CMake compilation (fails on ARM64 without Vulkan)
• After: Zero compilation, instant results via HTTP to remote Ollama
• qmd embed --force successfully re-indexes 7,100+ documents
• qmd vsearch returns results in <2s vs hanging on CMake indefinitely

Notes

• qmd search (BM25) is unaffected — works perfectly without any of this
• expandQuery uses LLM for HYDE-style query expansion. When using Ollama path, we skip this and pass the raw query as a vector search. A future improvement could call Ollama /api/generate for query expansion.
• Char-based chunking (text.length / avgCharsPerToken) is a reasonable approximation that avoids requiring a local tokenizer
• The OLLAMA_EMBED_MODEL env var allows users to pick any embedding model available on their Ollama instance

Happy to submit a PR if there is interest.

[jaylfc]

I've implemented remote model support in a fork that covers embeddings, reranking, AND query expansion — not just embeddings.

What it adds:

• qmd serve — runs an HTTP server that loads the models once and exposes /embed, /embed-batch, /rerank, /expand, /tokenize, and /health endpoints
• qmd query --server http://host:7832 or QMD_SERVER=http://host:7832 — any QMD command can use a remote model server instead of loading models locally
• RemoteLLM class implementing the existing LLM interface, so it's a clean drop-in

Use case: Multiple OpenClaw agents in separate LXC containers sharing a single set of models on a 16GB ARM SBC (Orange Pi 5 Plus), saving ~4GB+ of duplicated model weights.

Fork: https://github.com/jaylfc/qmd/tree/feat/remote-llm-provider

Install from fork:

npm install -g github:jaylfc/qmd#feat/remote-llm-provider

Happy to clean this up into a proper PR against upstream if there's interest.

[jaylfc]

Update: The fork now includes several improvements for SBC/NPU deployment:

• rkllama NPU backend (--backend rkllama): routes embed/rerank/expand to RK3588 NPU via rkllama
• Batch embed optimisation: sends all chunks in one rkllama API call instead of individual HTTP requests
• Per-chunk retry with backoff for intermittent NPU failures
• 60-minute session timeout (up from 10min) for large document batches
• Auto-detect QMD_SERVER env var so containers don't try to initialise node-llama-cpp locally
• KV cache clearing in rkllama fork to prevent NPU performance degradation

Benchmarked against standard QMD on x86 with 3060: BM25 identical, vector search rankings correct, reranking accurate via logit scoring. Full results: https://github.com/jaylfc/openclaw-sbc-guide/blob/main/docs/benchmark-results.md

rkllama PR for rerank + concurrent serving: NotPunchnox/rkllama#138

[paralizeer]

@jaylfc just tested your feat/remote-llm-provider branch on our ARM64 VPS (Oracle Cloud Ampere, Ubuntu 24.04). The qmd serve → QMD_SERVER flow works perfectly for our multi-agent setup where we need several QMD instances sharing models loaded on a single host.

This is the most complete implementation across all the remote embedding PRs (#116, #480, #490). Would love to see this upstreamed — happy to help with testing, docs, or co-authoring a PR if you open one.

cc @tobi — between #116 (OpenAI-compatible embeddings) and @jaylfc's fork (native QMD model server), the remote model story is really coming together. Both solve real deployment blockers for ARM64/Docker/headless environments.

[jaylfc]

Thanks for the detailed writeup @paralizeer — this was really helpful in validating the approach.

I've opened an upstream PR that covers this: #509

It implements qmd serve as a standalone model server with a RemoteLLM client that drops in via QMD_SERVER env var — covers embeddings, reranking, query expansion, and batch operations. Also added index endpoints (/search, /browse, /collections, /status) for remote memory access.

Been running this on an Orange Pi 5 Plus (RK3588) with 3 agents in LXC containers sharing one server instance — all working reliably with NPU acceleration via rkllama. Happy to help test if you want to try it on your ARM64 VPS setup.

[jaylfc]

One heads up if you're using rkllama on RK3588 — we found a critical KV cache issue where embed/rerank times degrade from 0.3s to 26s+ over hours if the KV cache isn't cleared between operations. The fix is in our rkllama fork (PR NotPunchnox/rkllama#138) — clears the cache after each embed/rerank call.

Without the fix, everything looks fine initially but performance tanks after a few hundred operations. Took us a while to track down. Worth knowing if you're doing bulk embedding on NPU.

[jaylfc]

Also @paralizeer — if you're running multiple agents on ARM64 sharing a remote Ollama/rkllama instance, you might want to keep an eye on https://github.com/jaylfc/tinyagentos

It's what we're building on top of all this — a self-hosted platform for managing AI agents on low-power hardware. Web dashboard with an app store for agent frameworks (SmolAgents, PocketFlow, OpenClaw etc), model management with hardware-aware recommendations, agent deployment into LXC containers, and system monitoring. Sounds like it fits your setup.

[paralizeer]

@jaylfc TinyAgentOS looks great — the hardware profiling + LXC isolation + QMD serve per agent is exactly the right architecture for multi-agent on constrained hardware.

We reviewed the codebase and opened a PR with security fixes: jaylfc/tinyagentos#1

Main changes: default bind to 127.0.0.1 (was 0.0.0.0 with no auth), agent name validation, framework allowlist on deploy, atomic config writes, and deploy rollback on failure. All 127 tests pass.

We're running a similar setup on Oracle Cloud ARM64 (4 vCPU, 23GB) — three agents sharing services on one host instead of LXC containers, but the same principle. Your qmd serve shared model server is already running in production for us as a systemd service.

Would be interested in collaborating on the OpenClaw catalog entry — the manifest references scripts/install.sh that doesn't exist yet. Happy to contribute that.

[jaylfc]

Thanks @paralizeer — really appreciate you taking the time to review the codebase and open a PR with security fixes. Good catches on the bind address and deploy validation, I'll review that properly.

Great to hear you've got qmd serve running in production on ARM64 already — that's exactly the kind of real-world validation that matters.

Regarding collaboration on TinyAgentOS — I appreciate the offer and it's good to know there's interest. Right now I'm still laying the architectural foundations and working through the core platform design, so I'd like to get those solidified before opening up to external contributions. Once the project is more mature and the patterns are well established, I'll definitely be looking for contributors. I'll keep you posted — feel free to star/watch the repo in the meantime and I'll flag when it's ready for community involvement.

The OpenClaw install script is on the roadmap — good spot on the missing reference.