Trim unsupported fields from the body while making groq queries

[swiftugandan]

Groq does not support some of the fields that are set in the body see: https://console.groq.com/docs/openai

The openai js library automatically adds some of these fields, resulting in 400 errors.

Please help trim these out for groq requests

[ankrgyl]

Thanks for filing @swiftugandan. Would you be open to contributing a PR? Happy to walk you through where to make changes in the code.

[swiftugandan]

Yes sure, where should I look?

[ankrgyl]

We likely need to add logic in fetchOpenAI similar to how we call translateParams on Anthropic.

Currently translateParams operates in terms of ModelFormat, so I think we need to create something like translateProviderParams that does something similar in terms of ModelEndpointType. To keep things simple, I would create a map that specifies which params are disallowed by specific endpoint types (in this case, groq), and then check this logic in fetchOpenAI().

BTW, are you passing these params in and hoping that they're stripped, or are they being inserted automatically somewhere? If the latter, maybe to start, we remove these parameters if they're specified, but null, and throw an error if they are non-null?

[swiftugandan]

I think they are being added as default params automatically, let me run the proxy to verify what is being sent

[swiftugandan]

I have successfully run the node proxy, but when I send a groq message, I am getting an error Failed to lookup api key: Failed to validate JWT: Error: JWT string does not consist of exactly 3 parts (header, payload, signature) from

braintrust-proxy/apis/node/src/login.ts

Line 19 in b87c35a

const response = await fetch(`${Env.braintrustApiUrl}/api/secret`, {

I reproduced the error using curl.

➜ braintrust-proxy git:(main) ✗ curl -X POST \ -H "Authorization: Bearer gsk_L..." \ -H "Content-Type: application/json" \ -d '{ "model": "mixtral-8x7b-32768", "org_name": "undefined", "mode": "full" }' \ "https://www.braintrustdata.com/api/secret"

How can I get past this step?

[ankrgyl]

The node proxy does not currently support "api token passthrough", which the cloudlfare one does. Could you debug locally using the cloudflare proxy? Just cd into apis/cloudflare and run:

pnpm install
cp wranger-template.toml wrangler.toml
npx wrangler start

and it should work

[swiftugandan]

it turns out no additional fields are being injected. The problem is with the messages array that is being sent.

groq expects messages: [{"role": "user", "content": "some string"}] but I was sending the OpenAI format messages: [{"role": "user", "content": [{ "type": "text", "text":"some string" }]}] ... do we need to add support for this format?

[ankrgyl]

Very interesting. We could do something similar to what I suggested above, which per-ModelEndpointType we boil down the chat messages into a simpler format.