vitest evals

[cpinn]

Add ability to create experiments writing tests with vitest

Use existing braintrust datasets or pass their own data to test
Use existing scorers or pass their own custom scorer to tests

Each describe will create an experiment and the tests inside will each be their own span in the experiment.

See golden-ts-vitest-experiment-v* projects for some examples

Example:

import { initDataset } from "braintrust";

bt.describe("My experiment", () => {
  const evalData = initDataset({
    project: "llm-evals",
    dataset: "qa-benchmark",
  }).fetchedData();
 // test will also accept the dataset array and it will be expanded or pass the data individually
  bt.test.each(await evalData)(
    "Q&A evaluation",
    {
      scorers: [
        Factuality, // from autoevals
        ({ output, expected }) => ({ // custom scorer
          name: "conciseness",
          score: output.length <= expected.length * 1.2 ? 1 : 0.7,
        }),
      ],
    },
    async (record) => {
      const { input, expected } = record;
      const answer = await llm.answer(input.question);
      return answer; 
    },
  );
});

[ibolmo]

various questions

[ibolmo]

I'd recommend for users to:

import * as vitest from 'vitest';

const { describe, expect, test, ... } = wrapVitest(vitest);

simpler, less prone to error, and future proof if we add more functions to our support

[ibolmo]

til.. so the context is re-inserted as an argument

[ibolmo]

didn't expect to see this file in here 🤔

[ibolmo]

like we want to extract and use the same getOrCreate inside of test() calls

[ibolmo]

i didn't see a wrapExpect in wrapper.ts. I wonder if each expect is a scorer?

[ibolmo]

i think if you traced(maybeFn || configOrFn, ...) you may have gotten this automatically?

[ibolmo]

you should probably just throw again. the traced() call should handle the error.

[ibolmo]

what happens with concurrent calls i.e.

it.concurrent(
 describe(..., () => {
 })
);

it.concurrent(
 describe(..., () => {
 })
);

did you give currentExperiment a try?

[cpinn]

I modified how the context is created for experiments. I added some additional tests for the concurrent experiments.

[ibolmo]

small suggestions not blocking

[ibolmo]

are the expects scorers?

[ibolmo]

ah yes, could we try multiple expect() calls? I wonder how that would look in the bt logs.

what does the custom message look like in braintrust? i.e.

expect(result, 'equality').toBe(expected)

[ibolmo]

ah so we don't wrap expects.

like i mentioned earlier, it would be interesting to use the expect(..., 'message')... and the message is the index for the output, or maybe we have a counter/stack that we push expect(output) to then the output might be something like

expect(0).toBe(0);
expect('something', 'message').toBe('something');
expect('foo').toBe('bar');

then the event could be

{
  ...
  output: {
    0: output,
    'message': output,
    1: output,
  }, 
  scores: {
    0: 1,
    'message': 1,
    1: 0
  }
 }
}

[cpinn]

ah yes I didn't wrap the expect method just the test methods. I was thinking people cared about getting the scoring which does add its output to individual logs. Thinking about this flow a bit.

[cpinn]

did a compromise and logged named outputs automatically. Users can still log additional outputs as necessary. Will see what the feedback is like.

[ibolmo]

great coverage

[ibolmo]

i wonder if we need the trace() method 🤔

[ibolmo]

i'd expect this to be done automatically 🤔. looking at their docs it doesn't seem like they require them to be explicit

[cpinn]

This is done automatically. The method is exposed for access but doesn't need to be called. I thought I removed it from the examples and tests, might have missed this one.