Skip to content

autoevals npm package doesn't support Factuality with Azure OpenAI #116

@mongodben

Description

@mongodben

using autoevals@0.0.119.

not sure why, but when i try to use the Factuality eval scorer with azure openai config, it doesn't work.

get error that API key doesn't exist. error message seems to indicate that it's trying to use the standard OpenAI:

401 Incorrect API key provided: 83be8c47********************6caa. You can find your API key at https://platform.openai.com/account/api-keys.

Error: 401 Incorrect API key provided: 83be8c47********************6caa. You can find your API key at https://platform.openai.com/account/api-keys.
    at APIError.generate (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/error.js:48:20)
    at OpenAI.makeStatusError (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/core.js:263:33)
    at OpenAI.makeRequest (/Users/ben.p/projects/chatbot/node_modules/autoevals/node_modules/openai/core.js:306:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

here's my relevant code: https://github.com/mongodb/chatbot/pull/628/files#diff-f687edc4d800857499bfc0a098f9178ee614a6fe107d2611ba38dbb47a93a135R159

i wasn't able to fully discern what was going on by looking at the autoevals code but seems to relate to call to the function cachedChatCompletion(), which the Ragas LLM-as-a-judge scorers don't use.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions