Uniform improved `client=` (direct & global) option for Python and JS by ibolmo · Pull Request #121 · braintrustdata/autoevals

ibolmo · 2025-03-04T15:22:49Z

For TS

New client argument to OpenAIClassifiers as well as the global init() method. Similar to how we have in Python (but maybe a little simpler).

For Python

Previously:

openai_obj = openia.OpenAI(...)
client = LLMCLient(
  openai=openai_obj,
  complete=openai_obj.chat.completions.create,
  ...
)

Would break because the prepare_openai would wrap client.openai but leave behind the complete as the unwrapped complete.

Now if you pass customizations we will avoid automatic wrapping to avoid this problem. Users can, however, wrap like so and everything will be wrapped correctly:

openai_obj = wrap_openai(openai.OpenAI())
client = LLMClient(
  openai=openai_obj,
  complete=openai_obj.chat.completions.create,
)

Further, because we've tighten things up, we can now automatically configure the client reducing the code supplied by the user to something like:

init(openai.OpenAI(...))

Before they had to:

openai_obj = openai.OpenAI(...)

client = LLMClient(
  openai=openai_obj,
  complete=openai_obj.chat.completions.create,
  ...
)

init(client)

Also, discovered that the client.openai should have been the openai instance. The only time this is not the case is for v0 support of the OpenAI library.

Discovered that the client.openai should have been the openai instance. Also, that the wrapping of the client should not be automatic as the attached methods may not be the right ones. i.e. if the client.openai is unwrapped, then we wrap in the prepare_openai the attached methods in the client.complete would be the unwrapped ones vs. the new wrapped one.

Simplifies the interface and the code. Need to update the SDK to expose a `unwrap()` method to avoid the hacks.

avoids hacky lookup

py/autoevals/test_oai.py

setup.py

ibolmo · 2025-03-04T15:30:01Z

py/autoevals/oai.py

+        has_customization = self.complete is not None or self.embed is not None or self.moderation is not None
+
+        # avoid wrapping if we have custom methods (the user may intend not to wrap)
+        if not has_customization and not isinstance(self.openai, NamedWrapper):


This caught me off guard when I was reproducing the Azure problems. Now we avoid mix matching objects like:

openai_obj = openai.OpenAI(...) client = LLMClient( openai=openai_obj, complete=openai_obj.chat.completions.create, ... )

Previously the openai_obj would be wrapped and the rest of the code would assume that the methods would be wrapped too.

github-actions · 2025-03-04T16:42:19Z

Braintrust eval report

Autoevals (bra-2135--azure-openai-autoevals-1742245432)

Score	Average	Improvements	Regressions
NumericDiff	75% (+0pp)	-	-
Start	1742245432.23s	-	-
End	1742245433.78s	-	-
Duration	1.51s (-0.08s)	100 🟢	-
Prompt_tokens	279.25tok (+0tok)	-	-
Completion_tokens	19.17tok (+0tok)	-	-
Total_tokens	298.42tok (+0tok)	-	-

py/autoevals/oai.py

luckily discovered this while trying to run an Azure example

…t passing client

ibolmo · 2025-03-05T02:44:35Z

js/llm.test.ts

+} from "./llm.fixtures";
+
+beforeAll(() => {
+  init(


note the use of new init()

ibolmo · 2025-03-05T02:45:56Z

js/oai.test.ts

+  }) => Promise<void>,
+) => {
+  const createSpy = jest.fn();
+  const wrapperMock = (client: any) => {


avoids having to add braintrust as a dependency

is there a reason this is worth avoiding? seems like we would want to pull it in at some point as tests cover new surface area introduced in the sdk?

looks like you added it for python too so for the sake of consistent access to sdk might it be worth adding to the js dependencies as well?

ibolmo · 2025-03-05T02:47:28Z

js/test/setup.ts

@@ -0,0 +1,17 @@
+import { setupServer } from "msw/node";


todo go back to our previous tests and adopt msw in them (separate PR)

ibolmo · 2025-03-05T02:50:36Z

py/autoevals/oai.py

-            embed=openai.embeddings.create,
-            moderation=openai.moderations.create,
+        client = openai.OpenAI()  # Configure with your settings
+        llm = LLMClient(openai=client)  # Methods will be auto-configured


the interface is similar but not the same as the one in JS.

for exact same we would need to support passing exact methods in JS, but I'm betting that'll be an unlikely use case.

We should support this, though:

init(openai.OpenAI())

ibolmo · 2025-03-05T03:35:48Z

js/ragas.ts

  const [expectedEntities, contextEntities] = responses.map(mustParseArgs);

  const score = await ListContains({
-    ...extractOpenAIArgs(args),


wasn't needed (as far as I can tell)

ibolmo · 2025-03-05T03:36:08Z

node/llm.test.ts

@@ -1,145 +0,0 @@
-import { ChatCompletionMessageParam } from "openai/resources";


just got moved to js/llm.test.ts (not 100% sure why it needed to be here)

ibolmo · 2025-03-05T03:36:48Z

setup.py

 extras_require = {
    "dev": [
        "black",
+        "braintrust",


todo whether this is possible, or I need to workaround this

choochootrain

seems legit but will defer to experts for rubber stamp

choochootrain · 2025-03-10T21:12:14Z

js/oai.test.ts

+  }) => Promise<void>,
+) => {
+  const createSpy = jest.fn();
+  const wrapperMock = (client: any) => {


is there a reason this is worth avoiding? seems like we would want to pull it in at some point as tests cover new surface area introduced in the sdk?

py/autoevals/moderation.py

py/autoevals/oai.py

choochootrain · 2025-03-10T21:32:14Z

js/oai.test.ts

+  }) => Promise<void>,
+) => {
+  const createSpy = jest.fn();
+  const wrapperMock = (client: any) => {


looks like you added it for python too so for the sake of consistent access to sdk might it be worth adding to the js dependencies as well?

github-actions · 2025-03-17T21:28:03Z

Braintrust eval report

Autoevals (main-1742246885)

Score	Average	Improvements	Regressions
NumericDiff	75% (+0pp)	-	-
Start	1742246884.95s	-	-
End	1742246886.48s	-	-
Duration	1.49s (-0.02s)	78 🟢	22 🔴
Prompt_tokens	279.25tok (+0tok)	-	-
Completion_tokens	19.17tok (+0tok)	-	-
Total_tokens	298.42tok (+0tok)	-	-

…n and JS" (#126) Reverts #121

ibolmo added 3 commits March 3, 2025 23:58

refactor: auto configure client if args are not provided

dc1dc31

Simplifies the interface and the code. Need to update the SDK to expose a `unwrap()` method to avoid the hacks.

use new unwrap() and local sdk (temp)

1f0dc76

avoids hacky lookup

ibolmo self-assigned this Mar 4, 2025

ibolmo commented Mar 4, 2025

View reviewed changes

py/autoevals/test_oai.py Outdated Show resolved Hide resolved

ibolmo commented Mar 4, 2025

View reviewed changes

setup.py Outdated Show resolved Hide resolved

ibolmo commented Mar 4, 2025

View reviewed changes

ibolmo force-pushed the bra-2135--azure-openai-autoevals branch from 02b17e4 to beb5345 Compare March 4, 2025 16:44

fix v0 async, test coverage, and various improvements

24b75dc

ibolmo force-pushed the bra-2135--azure-openai-autoevals branch from beb5345 to 24b75dc Compare March 4, 2025 16:44

add is_async explanation

7406abd

ibolmo commented Mar 4, 2025

View reviewed changes

py/autoevals/oai.py Outdated Show resolved Hide resolved

drop dependency on unwrap() and fix import problems

219c9c6

luckily discovered this while trying to run an Azure example

ibolmo marked this pull request as ready for review March 4, 2025 17:34

ibolmo added 8 commits March 4, 2025 15:51

add client option to OpenAIAuth deprecated all others

767f101

update tests to use new msw

e21ae2b

add more test coverage and try to wrap as much as possible

203a634

update package-lock.json

73f4f47

add test coverage for init

c17cb47

move node/llm.test.ts to js/llm.test.ts

0d8e170

improve llm test coverage and fix a bug where llm classifiers were no…

24fe660

…t passing client

improve types

3427b83

ibolmo commented Mar 5, 2025

View reviewed changes

ibolmo added 6 commits March 4, 2025 22:40

update pnpm lock

a482458

make sure we use 3.9.x

edf25b0

Fix types

ed70e21

try removing runtime_checkable

05203ad

affix black to match CI

f138e99

add pyright config

0edd002

ibolmo added 4 commits March 5, 2025 11:19

fix types round million

0fd6e2b

DX improvements

b979511

fix types and an old test

4bf757c

vscode worksapce settings

bb2a534

ibolmo changed the title ~~Improved LLMCLient and added test coverage for prepare_openai~~ Uniform improved client= (direct & global) option for Python and JS Mar 5, 2025

ibolmo added 10 commits March 5, 2025 14:44

export init

32a556a

bump version

02b53aa

fix protocol and accept OpenAI clients directly

c8fa427

update setup.py with openai

5688ed6

try to use node:utils

df44b9c

remove extra mention of LLMClient and instead mention Client type

4e0456e

lot's of doc improvements

9747959

avoid escaping example strings

6216f99

let's use an object as input to init() (prettier)

f5010fd

overhaul readme

2ae1b0c

ibolmo requested review from ornellaaltunyan and sachinpad March 6, 2025 18:31

fix example

d4a150a

choochootrain reviewed Mar 10, 2025

View reviewed changes

ibolmo added 4 commits March 10, 2025 17:03

Merge branch 'main' into bra-2135--azure-openai-autoevals

2153749

Merge branch 'main' into bra-2135--azure-openai-autoevals

f745878

update moderation docs

44816b5

fix docs

8c51826

choochootrain approved these changes Mar 15, 2025

View reviewed changes

ibolmo added 2 commits March 17, 2025 14:58

Merge branch 'main' into bra-2135--azure-openai-autoevals

f4e0b57

bump verison

8614b8f

ibolmo merged commit 01230f9 into main Mar 17, 2025
8 checks passed

ibolmo mentioned this pull request Mar 17, 2025

Revert "Uniform improved client= (direct & global) option for Python and JS" #126

Merged

ibolmo added a commit that referenced this pull request Mar 17, 2025

Revert "Uniform improved client= (direct & global) option for Pytho…

65a9ef5

…n and JS" (#126) Reverts #121

		@@ -1,145 +0,0 @@
		import { ChatCompletionMessageParam } from "openai/resources";

Conversation

ibolmo commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

For TS

For Python

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Braintrust eval report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

choochootrain left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Braintrust eval report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

ibolmo commented Mar 4, 2025 •

edited

Loading

github-actions bot commented Mar 4, 2025 •

edited

Loading

github-actions bot commented Mar 17, 2025 •

edited

Loading