Fetch from model list from /v1/models/ from OpenAI Compatible APIs
#3427
Conversation
- Implement automatic fetching of models from /v1/models endpoint for custom OpenAI-compatible providers - Add proper API key handling from auth.json or environment variables - Include enhanced logging for debugging provider configuration - Enable support for providers configured with @ai-sdk/openai-compatible npm package
airpods69
changed the title
/v1/models/ from OpenAI Compatible APIs
| // For custom config providers, we may need to get the npm field from config | ||
| const npmPackage = providerInfo?.npm || config.provider?.[providerID]?.npm; | ||
| if (npmPackage !== "@ai-sdk/openai-compatible") { | ||
| continue; | ||
| } |
There was a problem hiding this comment.
this isn't gonna work for like any providers
| }, | ||
| limit: { | ||
| context: 4096, // Default context limit | ||
| output: 4096, // Default output limit | ||
| }, |
There was a problem hiding this comment.
This will cause a veryyy poor experience for a lot of people 4096 context window isn't gonna be useful for most agentic work
There was a problem hiding this comment.
I just opened the repo to search if I could get away with 4096 context window length (or by disabling /compact)
And yes, I agree it was pretty ugly. just happened to me. should have also tested out the models itself as well.
| // Create a default model entry with basic information | ||
| fetchedModels[modelData.id] = { | ||
| id: modelData.id, | ||
| name: modelData.id || modelData.name || modelData.object || "Unknown Model", // Use id, name, or object from API |
There was a problem hiding this comment.
isn't model name required, i think it will always be defined
There was a problem hiding this comment.
Hmmmm, It is required but ig a good idea to let it stay here as well? unless I am missing out on something that you can see.
There was a problem hiding this comment.
I mean name field returned from the api will always be defined modelData.name so no need for all the ORs
|
There is something to this for providing a good ux, I'm wondering can we restrict it to:
And then don't override any of their existing custom models for that provider What is teh default context size for LM studio? |
|
Also we have these style preferences I should make a style guide: |
This would be a good idea and what also I had in my mind. I just added the model to Which I think would be fine with people who are going towards using custom providers like nano-gpt? atleast they don't have to worry about it for a while. If they are crossing the limits then they can add the model and the num_ctx value manually? Better than the 4096 limit atleast in my opinion... (just suffered from it lol) Also will keep the style guide in mind, it was a pretty quick PR for something I wanted to try out with the provider cause it has way too many models to select from. The UX factor definitely felt good after using the |
2 participants
This commit introduces automatic model discovery for OpenAI-compatible providers, allowing OpenCode to dynamically fetch available models from the /v1/models endpoint instead of requiring manual model configuration.
Key changes:
This enhancement enables OpenCode to work seamlessly with OpenAI-compatible APIs such as local LLM servers (Ollama, LocalAI), API proxies, and custom implementations without requiring manual model definitions in the configuration. The functionality addresses the need for supporting custom and local AI providers with dynamic model discovery.
Fixes the issue where OpenAI-compatible providers would show no available models unless manually configured in opencode.json.
(the PR is generated by qwen, though I have tested it out locally properly with the provider, used nano-gpt.com/api/v1 for this)