Problem with many OpenAI APIs

[uukelele]

Currently, to use an OpenAI compatible API, user can do something like this:

{
    "api": "openai",
    "url": "https://provider.ai/v1",
    "model": "llm-1-pro"
}

...and store the API key in OPENAI_API_KEY.

This works for now, but it falls apart as soon as we want to use two different OpenAI compatible APIs as we only have one slot for the API key.

There are a couple possible fixes -

1. Have an API_KEY_ALIAS in the model dict:

{
    "api": "openai",
    "url": "https://provider.ai/v1",
    "model": "llm-1-pro",
    "api_key_alias": "PROVIDER_API_KEY"
}

Then the API key can be added as PROVIDER_API_KEY.

2. Store the key in the profile JSON. But this isn't really a good idea because obviously security, it cannot be added to git anymore, and also it doesn't allow environment variables.

3. We could also define a better way to use OpenAI compatible APIs instead of this.

Personally I think the best short-term option would be going with Option 2. But maybe there are other ways?

[uukelele]

I have an idea that could work for #752:

Store a map of 'supported' providers. E.g.:

{
  "ai21": "https://api.ai21.com/studio/v1",
  "anyscale": "https://api.endpoints.anyscale.com/v1",
  "cohere": "https://api.cohere.com/v1",
  "deepinfra": "https://api.deepinfra.com/v1/openai",
  ...
}

Then, using the dictionary, the key (e.g. ai21) would be the model prefix, the value (e.g. https://api.ai21.com/studio/v1) would be the base URL, and to get the API key alias, we could simply do the prefix -> uppercase.

Then we could store it, maybe in like providers.json:

{
    "gpt": {
         // map of GPT compatible APIs, **including** OpenAI
    },
    "anthropic": {
         // map of Anthropic compatible APIs, including Anthropic themselves
    }
}

In fact we could even instead of using a map of supported providers with prefix:url, we could store an array of dictionaries with each dictionary having prefix, url, and api_key_alias fields. Maybe also a default_model.

And then using the keys of providers.json (e.g. gpt, anthropic) we could do a path operation to combine that with src/models/${provider}.js.

[Sweaterdog]

Why not fo option 2, but in keys.json have the user br able to add key names, like for example OPENAI_COMPAT_1 for one of them. The user would then specify which API key the profile uses. And the name could be anything.

[uukelele]

Why not fo option 2, but in keys.json have the user br able to add key names, like for example OPENAI_COMPAT_1 for one of them. The user would then specify which API key the profile uses. And the name could be anything.

So basically option 1?