Do not collect ancestors when GitMetadataSettings.collect is none

[barrettpyke]

Problem

logger.compute_metadata attempts to collect git ancestor commit data if base_exp_id or base_experiment are not None.

This causes an error when clients are attempting to run Evals in an environment (e.g. CI/CD runs in a docker container) that does not have the Git binary installed, this only happens with git repos.

gitutil.py attempts to check if the git module is present before attempting git commands but that just checks that the GitPython module is present which doesn't guarantee the binary.

There doesn't seem to be a way to bypass this call to get_past_n_ancestors.

Steps to Reproduce:

1. Create a script (see below) that runs an Eval in an empty directory.
2. Run git init in the directory from step 1.
3. Run git remote add origin https://github.com/fake/repo.git
4. Run git branch -m main
5. Run git add . && git commit -m "test"
6. Create a Docker container that does not have git installed in the step 1 directory e.g.
   docker run --rm -it \ -v "$PWD":/app \ -w /app \ python:3.12-slim \ sh
7. Run pip install braintrust.
8. Run BRAINTRUST_API_KEY=<your-api-key> python <your-script-name>.py.

Expected Behavior
It should successfully run the Eval.

Observed Behavior
It fails with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/git/cmd.py", line 1279, in execute
    proc = safer_popen(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/repro-script.py", line 37, in <module>
    result = Eval(
             ^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/framework.py", line 981, in Eval
    f = _EvalCommon(
        ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/framework.py", line 739, in _EvalCommon
    experiment = init_experiment(
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/framework.py", line 1198, in init_experiment
    summary = ret.summarize(summarize_scores=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/logger.py", line 3537, in summarize
    state = self._get_state()
            ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/logger.py", line 3376, in _get_state
    self._lazy_metadata.get()
  File "/usr/local/lib/python3.12/site-packages/braintrust/util.py", line 185, in get
    res = self.callable()
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/logger.py", line 1414, in compute_metadata
    args["ancestor_commits"] = list(get_past_n_ancestors())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/gitutil.py", line 86, in get_past_n_ancestors
    ancestor_output = _get_base_branch_ancestor()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/braintrust/gitutil.py", line 73, in _get_base_branch_ancestor
    head = "HEAD" if _current_repo().is_dirty() else "HEAD^"
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/git/repo/base.py", line 962, in is_dirty
    if osp.isfile(self.index.path) and len(self.git.diff("--cached", *default_args)):
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/git/cmd.py", line 1003, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/git/cmd.py", line 1616, in _call_process
    return self.execute(call, **exec_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/git/cmd.py", line 1293, in execute
    raise GitCommandNotFound(redacted_command, err) from err
git.exc.GitCommandNotFound: Cmd('git') not found due to: FileNotFoundError('[Errno 2] No such file or directory: 'git'')
  cmdline: git diff --cached --abbrev=40 --full-index --raw

Solution

I used GitMetadataSettings being set to collect: none to skip this get_past_n_ancestors call. I'm not sure if an explicit flag for this exact operation would be better since GitMetadataSettings collection is a bit different than getting the past ancestor commits to add to an experiment.

Test Plan

Tested locally with the repro script in a container without git binary and it successfully created the Eval.

[cpinn]

could we modify this check to be:

Suggested change

	elif merged_git_metadata_settings.collect != "none":
	elif merged_git_metadata_settings and merged_git_metadata_settings.collect != "none":

merged_git_metadata_settings only gets set in an else statement so this would be more defensive