Add braintrust.auto_instrument() for one-line auto-instrumentation #1265

clutchski · 2026-01-14T16:34:51Z

Adds a new auto_instrument() function that detects installed AI/ML libraries and automatically instruments them for tracing.

Supported integrations:

OpenAI (patch_openai/unpatch_openai)
Anthropic (patch_anthropic/unpatch_anthropic)
LiteLLM (patch_litellm/unpatch_litellm)
Pydantic AI (setup_pydantic_ai)
Google GenAI (setup_genai)
Agno (setup_agno)
Claude Agent SDK (setup_claude_agent_sdk)
DSPy (patch_dspy/unpatch_dspy)

Usage:

import braintrust
braintrust.auto_instrument()

py/src/braintrust/wrappers/test_auto.py

knjiang

mostly questions around maintainablility and discovery of auto-instrumentation.

code LGTM

py/src/braintrust/wrappers/test_auto.py

knjiang · 2026-01-16T21:53:39Z

py/src/braintrust/wrappers/claude_agent_sdk/__init__.py

-    except ImportError as e:
-        logger.error(f"Failed to import Claude Agent SDK: {e}")
-        logger.error("claude-agent-sdk is not installed. Please install it with: pip install claude-agent-sdk")
+    except ImportError:


i dont see tests for claude_agent_sdk and others, i think we can probably add them in as well for test_auto.
i'm wnodering if there's an a way we can also enforce testing auto instrumentation?

I don't know about enforcement but I added the rest!

manugoyal

Looks cool to me

manugoyal · 2026-01-22T00:50:44Z

py/examples/auto_instrument.py

+logger = braintrust.init_logger(project="auto-instrument-demo")
+
+# Now import and use AI libraries normally - all calls are traced!
+# IMPORTANT: Import AI libraries AFTER calling auto_instrument()


Is this requirement really necessary? I think module imports are shared objects,
so if you modify a module in one scope, it'll modify all imports of that module.
E.g.

In [1]: def foo(): ...: import sys ...: setattr(sys, 'manuattr', 'foo') ...: In [2]: import sys In [3]: getattr(sys, 'manuattr') --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[3], line 1 ----> 1 getattr(sys, 'manuattr') AttributeError: module 'sys' has no attribute 'manuattr' In [4]: foo() In [5]: getattr(sys, 'manuattr') Out[5]: 'foo'

- import openai; openai.OpenAI() - works regardless of order (looks up attribute dynamically) - from openai import OpenAI; OpenAI() - only works if imported AFTER patching (creates local binding)

manugoyal · 2026-01-22T00:52:40Z

py/src/braintrust/auto.py

+    try:
+        from braintrust.oai import patch_openai
+
+        patch_openai()


Looks like patch_openai swallows import errors, so if we're trying to detect
whether the patch succeeded, maybe patch_openai should return the boolean
directly?

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

DSPy uses litellm which uses httpx. The standalone VCR in test_utils doesn't capture httpx requests (only requests/urllib3). The pytest-vcr plugin has httpx support, so span verification is done in test_dspy.py. This test now focuses on patching behavior only - verifying that auto_instrument/auto_uninstrument work correctly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

ibolmo · 2026-01-14T23:45:24Z

py/src/braintrust/wrappers/agno/__init__.py

-        logger.error(f"Failed to import Agno: {e}")
-        logger.error("Agno is not installed. Please install it with: pip install agno")
+    except ImportError:
+        # Not installed - this is expected when using auto_instrument()


maybe have the logging conditional if and only if not using auto instrumentation

ibolmo · 2026-01-23T22:16:03Z

py/src/braintrust/auto.py

+
+    Example:
+        ```python
+        import braintrust


i'd rather have each register itself with the library, and then have this iterate through registered instrumentations. This allows 3rd (or first) party libraries to register themselves as well (we'll need this for langchain-py and adk).

ibolmo · 2026-01-23T22:50:10Z

py/setup.py

    package_dir={"": "src"},
    packages=setuptools.find_packages(where="src"),
-    package_data={"braintrust": ["py.typed"]},
+    package_data={"braintrust": ["py.typed", "wrappers/cassettes/*.yaml"]},


should we be including the cassettes? :O

ibolmo · 2026-01-23T22:53:09Z

py/src/braintrust/oai.py

+def _apply_openai_wrapper(client):
+    """Apply tracing wrapper to an OpenAI client instance in-place."""
+    wrapped = wrap_openai(client)
+    for attr in ("chat", "responses", "embeddings", "moderations", "beta"):


we should be using wrapt library to override class methods. i fear about the loss of types and other idiosyncratic pythonic things (signature/introspection, binding, ...) with this setattr approach

yea i just didn't want to add a dependency without really making sur eit was worht itm but we can try. we did this at dd as well.

yeah i think wrapt is very robust and its used in opentelemetry as well. so i think it's worth the bytes plus it's already in the build 😂

ibolmo · 2026-01-23T22:53:34Z

py/src/braintrust/auto.py

+    return results
+
+
+def _uninstrument_openai() -> bool:


each wrapper should have this rather than in this massive auto.py file

ibolmo · 2026-01-23T22:53:54Z

py/src/braintrust/__init__.py

    _internal_with_custom_background_logger,  # noqa: F401 # type: ignore[reportUnusedImport]
 )
 from .oai import (
+    patch_openai,  # noqa: F401 # type: ignore[reportUnusedImport]


i don't think we should expose these

ibolmo

woops forgot to block until my questions/suggestions are reviewed

clutchski marked this pull request as ready for review January 14, 2026 16:34

clutchski requested review from ibolmo and knjiang January 14, 2026 16:35

clutchski force-pushed the matt/auto.py branch from cd05d06 to 80a1ba6 Compare January 14, 2026 20:34

clutchski commented Jan 16, 2026

View reviewed changes

py/src/braintrust/wrappers/test_auto.py Outdated Show resolved Hide resolved

knjiang approved these changes Jan 16, 2026

View reviewed changes

clutchski force-pushed the matt/auto.py branch from c638c2d to 8c6d75e Compare January 21, 2026 23:19

manugoyal approved these changes Jan 22, 2026

View reviewed changes

clutchski added 2 commits January 21, 2026 17:27

add autoinstrumentation

9a2d62c

bump

d2eaa19

clutchski force-pushed the matt/auto.py branch from f3cfcc9 to d2eaa19 Compare January 22, 2026 01:36

package cassettes

07c74dc

clutchski requested a review from ankrgyl as a code owner January 22, 2026 21:18

clutchski and others added 3 commits January 22, 2026 13:45

Add google_genai cassette and filter x-goog-api-key header

5317aa2

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Remove reference to deleted test_auto.py

0b419e9

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

ibolmo reviewed Jan 23, 2026

View reviewed changes

ibolmo requested a review from knjiang January 23, 2026 22:55

ibolmo requested changes Jan 23, 2026

View reviewed changes

Add braintrust.auto_instrument() for one-line auto-instrumentation #1265

Are you sure you want to change the base?

Add braintrust.auto_instrument() for one-line auto-instrumentation #1265

Uh oh!

Conversation

clutchski commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

knjiang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manugoyal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ibolmo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

clutchski commented Jan 14, 2026 •

edited

Loading