feat: implemented direct provider resolve#21
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
@Winston-Hsiao if none of the keys are set, there's an error thrown. if old key is set, then openrouter is used with deprecation warning / migration notice. sk-or- prefix is required for the old api key, otherwise it will result in an error. i hope this clarifies your doubts |
Winston-Hsiao
left a comment
Winston-Hsiao
left a comment
There was a problem hiding this comment.
Hey good work, but not quite the fully flexible multi-provider (Cerebras or OpenRouter) implementation we want. We want to allow users to switch between them—which currently is not the case.
Also we need to make sure to validate when using the new `cfg.Provider = "cerebras" config that is set, that you check if the model is a valid/eligible model to use through Cerebras directly.
Also make sure to avoid just using the bare "cerebras" string, can set a typed enum in a types file somewhere and export that package/enum'd string type
| // Validate that at least one API key is configured | ||
| if !cfg.HasAnyAPIKey() { | ||
| fmt.Fprintln(os.Stderr, "Error: No API key configured. Set one of:") | ||
| fmt.Fprintln(os.Stderr, " CEREBRAS_API_KEY - for direct Cerebras inference") | ||
| fmt.Fprintln(os.Stderr, " OPENROUTER_API_KEY - for OpenRouter multi-provider access") | ||
| fmt.Fprintln(os.Stderr, " Or run: raypaste config set cerebras-api-key <key>") |
There was a problem hiding this comment.
"Or run: ..." language seems a bit odd here
maybe "Set key by running: " instead? (and give both example commands)
| { | ||
| name: "cerebras model with cerebras key goes direct", | ||
| cfg: Config{CerebrasAPIKey: "csk-123", Models: map[string]Model{}}, | ||
| modelAlias: "cerebras-llama-8b", | ||
| wantProvider: "cerebras", | ||
| wantKey: "csk-123", | ||
| }, | ||
| { | ||
| name: "cerebras model with only openrouter key falls back to openrouter", | ||
| cfg: Config{OpenRouterAPIKey: "sk-or-abc", Models: map[string]Model{}}, | ||
| modelAlias: "cerebras-llama-8b", | ||
| wantProvider: "openrouter", | ||
| wantKey: "sk-or-abc", | ||
| }, | ||
| { | ||
| name: "cerebras model with both keys prefers direct", | ||
| cfg: Config{CerebrasAPIKey: "csk-123", OpenRouterAPIKey: "sk-or-abc", Models: map[string]Model{}}, | ||
| modelAlias: "cerebras-llama-8b", | ||
| wantProvider: "cerebras", | ||
| wantKey: "csk-123", | ||
| }, |
There was a problem hiding this comment.
I envisioned we would not have a default/fallback behavior here but instead the user can set their config to choose specifically if they want to use direct to Cerebras OR direct to OpenRouter.
i.e. your current implemented behavior constrains to "cerebras model with both keys prefers direct"
given both keys are set Config{CerebrasAPIKey: "csk-123", OpenRouterAPIKey: "sk-or-abc",
- but what if the user:
- doesn't want to clear their Cerebras API key to route through OpenRouter
- does want to switch back and forth, keeping both keys set and specifically wants to route to one or the other.
| ` + output.Green("default-model") + ` - Default model alias or OpenRouter ID | ||
| ` + output.Green("default-length") + ` - Default output length (short|medium|long) | ||
| ` + output.Green("disable-copy") + ` - Disable auto-copy to clipboard (true|false) | ||
| ` + output.Green("temperature") + ` - Sampling temperature (0.0-2.0) |
There was a problem hiding this comment.
based on one of the other comments regarding switching between available provider options by choice instead of forced default to cerebras openrouter/old fallback
we would likely want a "provider" key representing the selected/active provider that stores either "cerebras" or "openrouter", it can start out initialized as nil for new users, they will be prompted to set it with:
raypaste config set provider if it's nil
for quality of life new users when setting their key initially with raypaste config set cerebras-api-key [key...] can have the provider value set to "cerebras" automatically, likewise goes for if openrouter-api-key is set first instead
We want a flexible (no annoying config) lock-in implementation—which is currently not the case
| switch key { | ||
| case "openrouter-api-key": | ||
| cfg.OpenRouterAPIKey = value | ||
| if err := cfg.SaveTo(cfgFile); err != nil { |
There was a problem hiding this comment.
can add the cfg.Provider = ... here in these setter case blocks
| } | ||
| } else { | ||
| fmt.Println("not set") | ||
| } |
There was a problem hiding this comment.
Avoid excessively nested if, else statements (for easier readability of logical flow).
can clean this up to be more idiomatic go example below:
case "openrouter-api-key":
key := cfg.GetOpenRouterAPIKey()
if key == "" {
fmt.Println("not set")
break
}
masked := maskSecret(key)
if cfg.OpenRouterAPIKey == "" {
fmt.Println(masked + " (from environment)")
} else {
fmt.Println(masked)
}
Somewhat confusing/annoying distinction worth calling out is that we support api key setting in both config (the .yaml file OR reading from the env) seen by following the logic in your new functions which is why we check for "" twice once on the overall resolved key OR on the one set by config which allows the distinctive printing of where the key is sourced from (config or env)
You added these in internal/config/config.go
// GetOpenRouterAPIKey retrieves the OpenRouter API key from config or environment
func (c *Config) GetOpenRouterAPIKey() string {
if c.OpenRouterAPIKey != "" {
return c.OpenRouterAPIKey
}
return os.Getenv("OPENROUTER_API_KEY")
}
// GetCerebrasAPIKey retrieves the Cerebras API key from config or environment
func (c *Config) GetCerebrasAPIKey() string {
if c.CerebrasAPIKey != "" {
return c.CerebrasAPIKey
}
return os.Getenv("CEREBRAS_API_KEY")
}
3 participants
Background
Currently all LLM completions route through OpenRouter, requiring users to have an OpenRouter API key. Given the tight ties to Cerebras-provided inference for speed, we want to support direct Cerebras connections so users can bypass the OpenRouter hop for lower latency. This also lays groundwork for the future Raypaste backend, where RAYPASTE_API_KEY will authenticate against our own servers.
Changes
llama-3.1-8b-instruct) alongside the OpenRouter ID (meta-llama/llama-3.1-8b-instruct). BuildRequest selects the right one based on provider.
Testing