add VLLM support?

[cboettig]

Problem/Solution

It has been great to see Ollama added as a first-class option in #646, this has made it easy to access a huge variety of models and has been working very well us.

I increasingly see groups and university providers using VLLM for this as well. I'm out of my depth but I understand VLLM is considered better suited when a group is serving a local model to multiple users (e.g. from a local GPU cluster, rather than everyone running an independent Ollama). It gets passing mention in some threads here as well. I think supporting more providers is all to the good and would love to see support for this as a backend similar to the existing Ollama support, though maybe I'm not understanding the details and that is unnecessary? (i.e. it looks like it might be possible to simply use the OpenAI configuration with alternative endpoint to access a VLLM server?)

[cboettig]

It looks like the team at the National Research Platform has a nice work-around for this at the moment using LiteLLM via it's OpenAI-compatible API (https://docs.litellm.ai/docs/proxy/user_keys) This works, though isn't really the same as direct VLLM support, but thought it was worth mentioning.

[flefevre]

Which environnemental variables do I have to set-up in order to use my own litellm instance, such as https://litellm.mylaboratory.gov/
Thanks for your help

[utsumi-fj]

@flefevre LiteLLM seems to support the following routes:

• /openai/deployments/{model}/chat/completions
• /openai/deployments/{model}/embeddings

I could use Language model (Completion model) and Embedding model by specifying the LiteLLM server URL as follows:

Language model

• Completion model: OpenAI :: gpt-4o (This is a dummy model and different from the actual model.)
• Base API URL: http://{ LiteLLM server }/openai/deployments/{ actual model name }

Embedding model

• Embedding mode: OpenAI :: text-embedding-3-large (This is a dummy model and different from the actual model.)
• Base API URL: http://{ LiteLLM server }/openai/deployments/{ actual model name }

The actual model name can be shown by /models or /v1/models API.

[srdas]

@cboettig Here is how you can use Jupyter AI to run models from a vLLM server using openrouter.

Install vLLM on the server. (I installed it locally on my Mac, which requires installation from source for Apple silicon in its own python environment.) For instructions, see: https://docs.vllm.ai/en/latest/getting_started/installation/index.html

Serve up the vLLM model you want from the environment:

You will see the server running on port 8000:

To check which models are running enter this in your browser and verify the model ids:

Start up Jupyter AI and update the AI Settings as follows (notice that we are using OpenRouter as the provider, which is a unified interface for LLMs based on OpenAI's API interface):

You can test that the model is working by running the following in a JupyterLab notebook:

Then, try the chat in Jupyter AI to see that it works:

I will update the documentation for the usage of vLLM through a separate PR.

[zonca]

thanks @srdas, this is extremely useful,
I got this to work with the Jetstream inference service https://docs.jetstream-cloud.org/general/inference-service,
however, I had to hack around a "openrouter_api_key keyword not accepted error", see details here:

https://www.zonca.dev/posts/2025-02-04-jetstream-llm-service-deepseek-jupyterai#patch-for-openrouter_api_key-keyword-not-accepted-error

Do you have a better workaround for my issue?

[srdas]

@zonca I think this may have been fixed by https://github.com/jupyterlab/jupyter-ai/pull/1219/files, which has been merged, so it will be available in the next release, which should come shortly. I believe it will address the issue.

[dlqqq]

@zonca Thanks for documenting that issue with Jupyter AI in a blog post! I'm releasing the v2.29.1 patch release right now, which includes PR #1219 to patch this bug. Can you try out Jupyter AI v2.29.1 in a new Python environment, and see if you still encounter this issue?

[zonca]

@srdas @dlqqq much appreciated thanks! I confirm this is fixed in 2.29.1, I'll also update my blog post.

[dlqqq]

@zonca Thanks for the quick reply! Really glad to hear. Our team will close this issue once we add documentation for vLLM. 🎉

[srdas]

@zonca Thank you so much for verifying that the fix works. That was so quick! Appreciate your maintaining a blog to help others who are using Jupyter AI.

[cboettig]

@srdas This is great, thanks much.

I am not seeing openrouter as an option for embedding model though?