Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 30 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,60 +1,57 @@
# Braintrust SDK
# Braintrust Python SDKs

[Braintrust](https://www.braintrust.dev/) is a platform for evaluating and shipping AI products. To learn more about Braintrust or sign up for free,
visit our [website](https://www.braintrust.dev/) or check out the [docs](https://www.braintrust.dev/docs).
[Braintrust](https://www.braintrust.dev/) is a platform for evaluating and shipping AI products. Learn more at [braintrust.dev](https://www.braintrust.dev/) and in the [docs](https://www.braintrust.dev/docs).

This repository contains the Python SDK for Braintrust. The SDK includes utilities to:
This repository contains Braintrust's Python SDKs and integrations, including:

- Log experiments and datasets to Braintrust
- Run evaluations (via the `Eval` framework)
- Manage an on-premises installation of Braintrust
- The main `braintrust` SDK package in [`./py`](./py)
- Integration packages under [`./integrations`](./integrations)
- Examples, tests, and local development tooling for Python SDK development

## Quickstart: Python
## Quickstart

Install the library with pip.
Install the main SDK and scorer package:

```bash
pip install braintrust autoevals
```

Then, create a file named `eval_tutorial.py` with the following code:
Create `tutorial_eval.py`:

```python
from braintrust import Eval
from autoevals import LevenshteinScorer
from braintrust import Eval

Eval(
"Say Hi Bot",
data=lambda: [
{
"input": "Foo",
"expected": "Hi Foo",
},
{
"input": "Bar",
"expected": "Hello Bar",
},
], # Replace with your eval dataset
task=lambda input: "Hi " + input, # Replace with your LLM call
scores=[LevenshteinScorer],
"Say Hi Bot",
data=lambda: [
{"input": "Foo", "expected": "Hi Foo"},
{"input": "Bar", "expected": "Hello Bar"},
],
task=lambda input: "Hi " + input,
scores=[LevenshteinScorer],
)
```

Then, run the following command:
Run it:

```bash
BRAINTRUST_API_KEY=<YOUR_API_KEY> \
braintrust eval eval_tutorial.py
BRAINTRUST_API_KEY=<YOUR_API_KEY> braintrust eval tutorial_eval.py
```

## Integrations
## Packages

Braintrust provides integrations with several popular AI development tools and platforms:

- **LangChain Python**: Integration for logging LangChain Python executions to Braintrust. [Learn more](integrations/langchain-py)
| Package | Purpose | PyPI | Docs |
| --- | --- | --- | --- |
| `braintrust` | Core Python SDK for logging, tracing, evals, and CLI workflows. | [![PyPI - braintrust](https://img.shields.io/pypi/v/braintrust.svg)](https://pypi.org/project/braintrust/) | [py/README.md](py/README.md) |
| `braintrust-langchain` | LangChain callback integration for automatic Braintrust logging. | [![PyPI - braintrust-langchain](https://img.shields.io/pypi/v/braintrust-langchain.svg)](https://pypi.org/project/braintrust-langchain/) | [integrations/langchain-py/README.md](integrations/langchain-py/README.md) |
| `braintrust-adk` | Deprecated Google ADK integration package. New ADK support lives in `braintrust`. | [![PyPI - braintrust-adk](https://img.shields.io/pypi/v/braintrust-adk.svg)](https://pypi.org/project/braintrust-adk/) | [integrations/adk-py/README.md](integrations/adk-py/README.md) |

## Documentation

For more information, check out the [docs](https://www.braintrust.dev/docs):
- Python SDK docs: https://www.braintrust.dev/docs/reference/sdks/python
- Release notes: https://www.braintrust.dev/docs/reference/release-notes

## License

- [Python](https://www.braintrust.dev/docs/reference/sdks/python)
Apache-2.0
44 changes: 10 additions & 34 deletions integrations/adk-py/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# braintrust-adk (DEPRECATED)

[![PyPI version](https://img.shields.io/pypi/v/braintrust-adk.svg)](https://pypi.org/project/braintrust-adk/)

SDK for integrating [Braintrust](https://braintrust.dev) with [Google ADK (Agent Development Kit)](https://github.com/google/adk-python). This package provides automatic tracing and logging of ADK agent executions to Braintrust.

> **This package is deprecated.** The Google ADK integration is now included in the main [`braintrust`](https://pypi.org/project/braintrust/) package.
Expand Down Expand Up @@ -37,10 +39,10 @@ pip install braintrust-adk

## Requirements

- Python >= 3.9
- Python >= 3.10
- Google ADK >= 1.14.1

## Quick Start
## Quickstart

The `braintrust-adk` integration automatically traces your ADK agents' execution, including:

Expand All @@ -49,8 +51,6 @@ The `braintrust-adk` integration automatically traces your ADK agents' execution
- Parallel execution flows
- Multi-step agent reasoning

### Basic Usage

```python
from google.adk.agents import LlmAgent
from braintrust_adk import setup_adk
Expand All @@ -73,9 +73,9 @@ response = agent.send_message("What's the weather like in New York?")
print(response.text)
```

### Advanced Configuration
## Advanced Configuration

#### Using Project ID
### Using Project ID

If you know your Braintrust project ID, you can use it directly:

Expand All @@ -86,7 +86,7 @@ setup_adk(
)
```

#### Custom Tools with Tracing
### Custom Tools with Tracing

Other braintrust functions like `traced` work seamlessly with this integration.

Expand Down Expand Up @@ -118,16 +118,12 @@ response = agent.send_message(
)
```

### Manual Patching
## Manual Patching

The `setup_adk` will automatically patch Google ADK Runner, Agent, and Flow classes to automatically trace all agent interactions. If you prefer to manually patch classes, you can use the `wrap_agent`, `wrap_runner`, and `wrap_flow` functions. Take a look at the [manual example](./examples/manual.py).

Note that, as of writing, `adk web` does not support [custom Runners](https://github.com/google/adk-web/issues/72) and you will need to use `setup_adk` if you would like LLM traces.

## Examples

The `examples/` directory contains complete working examples:

## Viewing Traces in Braintrust

Once you've set up the integration, you can view your traces in the Braintrust dashboard:
Expand All @@ -140,28 +136,8 @@ Once you've set up the integration, you can view your traces in the Braintrust d
- Token usage and latency metrics
- Any errors or warnings

## Development

To contribute to this integration:

```bash
# Clone the repository
git clone https://github.com/braintrustdata/braintrust-sdk-python.git
cd sdk/integrations/adk-py

uv sync

# Run examples
cd examples

# simple programmatic agent call
uv run manual.py

# or use the adk web UI
uv run adk web --port 8888
```

## Related Resources
## Documentation

- [Braintrust Documentation](https://www.braintrust.dev/docs)
- [Braintrust Python SDK Documentation](https://www.braintrust.dev/docs/reference/sdks/python)
- [Google ADK Documentation](https://github.com/google/genai-agent-dev-kit)
30 changes: 11 additions & 19 deletions integrations/langchain-py/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# braintrust-langchain

[![PyPI version](https://img.shields.io/pypi/v/braintrust-langchain.svg)](https://pypi.org/project/braintrust-langchain/)

SDK for integrating [Braintrust](https://braintrust.dev) with [LangChain](https://langchain.com/). This package provides a callback handler to automatically log LangChain executions to Braintrust.

## Installation
Expand All @@ -10,19 +12,17 @@ pip install braintrust-langchain

## Requirements

- Python >= 3.9
- LangChain >= 0.1.0
- Python >= 3.10
- LangChain >= 0.3.27

## Usage
## Quickstart

First, make sure you have your Braintrust API key set in your environment:

```bash
export BRAINTRUST_API_KEY="your-api-key"
```

### Basic Usage

```python
import asyncio
from braintrust import init_logger
Expand Down Expand Up @@ -54,6 +54,8 @@ if __name__ == "__main__":
asyncio.run(main())
```

## Passing Handlers Explicitly

If you'd prefer to pass the callback handler to specific LangChain calls instead of setting it globally, you can do so using the `callbacks` config option:

```python
Expand Down Expand Up @@ -83,18 +85,8 @@ The callback handler supports logging for:

Review the [LangChain documentation](https://python.langchain.com/docs/modules/callbacks/) for more information on how to use callbacks.

## Development

Contributions are welcomed!

```bash
git clone https://github.com/braintrustdata/sdk.git

cd sdk/integrations/langchain-py

pip install -e ".[dev]"
## Documentation

# work on the code

pytest
```
- Braintrust docs: https://www.braintrust.dev/docs
- Braintrust Python SDK docs: https://www.braintrust.dev/docs/reference/sdks/python
- LangChain callback docs: https://python.langchain.com/docs/modules/callbacks/
88 changes: 52 additions & 36 deletions py/README.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,73 @@
## Braintrust
# Braintrust Python SDK

A Python library for logging data to Braintrust. `braintrust` is distributed as
a [library on PyPI](https://pypi.org/project/braintrust/). It is open source and
[available on GitHub](https://github.com/braintrustdata/braintrust-sdk-python/tree/main/py).
[![PyPI version](https://img.shields.io/pypi/v/braintrust.svg)](https://pypi.org/project/braintrust/)

### Quickstart
The official Python SDK for logging, tracing, and evaluating AI applications with [Braintrust](https://www.braintrust.dev/).

Install the library with pip.
## Installation

```bash
pip install braintrust
```

**Performance tip**: For 3-5x faster JSON serialization, install with the optional `performance` extra:

```bash
pip install braintrust[performance]
```

Or install `orjson` separately:
Install the SDK:

```bash
pip install orjson
pip install braintrust
```

The SDK automatically detects and uses orjson if available, with seamless fallback to standard json. See [ORJSON_OPTIMIZATION.md](ORJSON_OPTIMIZATION.md) for details.
## Quickstart

Then, run a simple experiment with the following code (replace `YOUR_API_KEY` with
your Braintrust API key):
Run a simple evaluation:

```python
from braintrust import Eval


def is_equal(expected, output):
return expected == output


Eval(
"Say Hi Bot",
data=lambda: [
{
"input": "Foo",
"expected": "Hi Foo",
},
{
"input": "Bar",
"expected": "Hello Bar",
},
], # Replace with your eval dataset
task=lambda input: "Hi " + input, # Replace with your LLM call
scores=[is_equal],
"Say Hi Bot",
data=lambda: [
{"input": "Foo", "expected": "Hi Foo"},
{"input": "Bar", "expected": "Hello Bar"},
],
task=lambda input: "Hi " + input,
scores=[is_equal],
)
```

# Performance Optimization
Then run:

```bash
BRAINTRUST_API_KEY=<YOUR_API_KEY> braintrust eval tutorial_eval.py
```

## Optional Extras

Install extras as needed for specific workflows:

```bash
pip install "braintrust[cli]"
pip install "braintrust[openai-agents]"
pip install "braintrust[otel]"
pip install "braintrust[temporal]"
pip install "braintrust[all]"
```

Available extras:

- `performance`: installs `orjson` for faster JSON serialization
- `cli`: installs optional dependencies used by the Braintrust CLI
- `openai-agents`: installs OpenAI Agents integration support
- `otel`: installs OpenTelemetry integration dependencies
- `temporal`: installs Temporal integration dependencies
- `all`: installs all optional extras

## Documentation

- Python SDK docs: https://www.braintrust.dev/docs/reference/sdks/python
- Braintrust docs: https://www.braintrust.dev/docs
- Source code: https://github.com/braintrustdata/braintrust-sdk-python/tree/main/py

## License

For 3-5x faster JSON serialization, install `orjson`. The SDK automatically detects and uses orjson if available, with seamless fallback to standard json.
Apache-2.0