Allow custom OpenAI ENDPOINT & Model Names - e.g. Sambanova API Cloud

[soufianechami]

I am trying to use Sambanova API which is compatible with OpenAI SDK. Would love to see how fast inference can benefit here and also use open source LLMs.

Below is an example of Sambanova API call:

import os
import openai

client = openai.OpenAI(
    api_key=os.environ.get("SAMBANOVA_API_KEY"),
    base_url="https://api.sambanova.ai/v1",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.1-8B-Instruct",
    messages=[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"Hello"}],
    temperature=0.1,
    top_p=0.1
)

print(response.choices[0].message.content)

[Chenglong-MS]

@soufianechami This should be supported now, using

provider="openai"
api_key="xxxxxxx"
api_base="https://api.sambanova.ai/v1"
model="Meta-Llama-3.1-8B-Instruct"

Thanks!