[FEATURE]: models via API

[qdrddr]

I'd like to be able to use models via API using LiteLLM Client SDK.

With providers like:
Ollama/LM Studio - Embeddings, LLM
OpenRouter - Embeddings, LLM
DeepInfra - OCR, reranker, Embeddings, LLM
LiteLLM Proxy - OCR, reranker, Embeddings, LLM

[selipso]

Same, this is the only feature making me look elsewhere. Want to run this in a Linuxbrew box without a GPU

[jamiequint]

#133

i forked a branch jamiequint/qmd that supports this now

[theol0403]

This makes running qmd on a raspberry pi practical!

[marcinbogdanski]

I did alternative/complimentary PR:

feat(llm): add API backend support with backend-aware session routing

• adds API-backed LLM support (embeddings, query expansion, rerank)
• backend-aware session/chunking paths so API mode does not initialize local Llama/tokenizer paths

Seem repo is exploding, for maintainer convenience, I found these duplicates/complimentary work:

• Add optional OpenRouter provider for embeddings/search pipeline#133
• feat: Add OpenAI embedding and query expansion support#116
• Add OpenAI-compatible remote LLM support and remote sanity-check scri…#106
• feat: add remote embedding providers (Voyage AI, OpenAI-compatible)#65
• possibly more

[JasonGiedymin]

I've closed my PR #106 as the maintainers have other priorities.

The app has improved performance but for those who still require remote calls (such as myself) my repo will still be available and I'll keep it up to date (within reason). Branch: remote-api: https://github.com/JasonGiedymin/qmd-remote/tree/remote-api

Local perf enhancements in latest mainline:
QMD now runs on both Node.js and Bun, with up to 2.7x faster reranking
through parallel GPU contexts. GPU auto-detection replaces the unreliable
gpu: "auto" with explicit CUDA/Metal/Vulkan probing.

Again, if this isn't possible for you (vps, no GPUs) I'll be maintaining my fork up to date (within reason) and you can continue using that (for now).

[qdrddr]

That's a pity.
I understand that the author wants to keep it air-gapped and fast.

Though some times user would prefer to make it a bit longer in favor of SotA models for maximum recall accuracy that you simply couldn't run locally.

Hope @tobi you could reconsider and could merge this PR.

[JasonGiedymin]

If it wasn’t clear I just want to point out that those notes were from the latest mainline release (not my fork). They did a good job improving performance. I just so happen to be running qmd on a raspberry pi so I need to use remote hosted LLMs.

[mcr-ksh]

+1

[alexleach]

Yep, I need this. I'm running qmd in a docker dev container on Mac OS, which doesn't pass through Metal GPU support, so the build of ollama.cpp fails.

I can run the models within Docker Desktop's Docker Model Runtime, which does provide GPU support, by connecting to the OpenAI-compatible endpoint: model-runner.docker.internal:80.

Note, this used to be supported between qmd versions 0.1.0 and 0.6.0, where there was an OLLAMA_URL environment variable. The introduced dependency on node-llama-cpp dropped support for remote endpoints:-

OLLAMA_URL environment variable introduced here:

qmd/llm.ts

Line 167 in bab46da

const DEFAULT_OLLAMA_URL = "http://localhost:11434";

And then, according to CHANGELOG.md it was dropped in version 0.6.0, in commit: d383b5c#diff-a61a4b7ab1e926e9570624e7f2d77137282c090d355651ae52aabd7468cb3240

• Restoring the OLLAMA_URL environment variable would trivially grant the ability to select between local and remote (OpenAI API-compatible) LLM backends.

• It would not be difficult to implement an additional LLM backend class (e.g., OpenAiLlm) that implements the existing LLM interface but proxies calls to the remote endpoint.

• I'm happy to propose/submit a PR for review with this change, but it sounds like PR's are already available and the easiest thing is to switch to one of the forks already provided, for instance by @JasonGiedymin or @jamiequint (looks like there are others).