Allow custom OpenAI ENDPOINT & Model Names - e.g. Sambanova API Cloud #63
Description
I am trying to use Sambanova API which is compatible with OpenAI SDK. Would love to see how fast inference can benefit here and also use open source LLMs.
Below is an example of Sambanova API call:
import os
import openai
client = openai.OpenAI(
api_key=os.environ.get("SAMBANOVA_API_KEY"),
base_url="https://api.sambanova.ai/v1",
)
response = client.chat.completions.create(
model="Meta-Llama-3.1-8B-Instruct",
messages=[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"Hello"}],
temperature=0.1,
top_p=0.1
)
print(response.choices[0].message.content)