JSON serialization error with Gemini 2.5 models in multi-turn evaluation

[AbhiPrasad]

Bug Report

Error Message:

TypeError: Object of type ChatCompletionMessage is not JSON serializable

Location: sdk/py/src/braintrust/functions/invoke.py:173
Affected Versions: Current Braintrust SDK (Python)
Impact: Blocks evaluation of multi-turn conversations when using Gemini 2.5 models with LLM judges

Symptoms

1. Model-specific: Only occurs with Gemini 2.5 models (e.g., gemini-2.5-flash)
2. Multi-turn specific: Single-turn conversations work fine
3. Judge-specific: Only fails when LLM judge scorers are used (local scorers work)
4. Consistent failure: Reproducible 100% of the time with the specific conditions
5. Other models unaffected: GPT-4o-mini and other models work correctly with the same setup

Reproduction

The issue occurs when:

1. Loading a multi-turn conversation dataset
2. Evaluating with gemini-2.5-flash (or other Gemini 2.5 models)
3. Using LLM judge scorers loaded via init_function() (e.g., rag-factuality-llm, rag-relevance-llm, rag-completeness-llm)

Key code path:

# LLM judges loaded from Braintrust
rag_factuality_llm = init_function(project_name=project_name, slug="rag-factuality-llm")

# Evaluation with multi-turn conversation dataset
# → triggers JSON serialization of ChatCompletionMessage at invoke.py:173

The ChatCompletionMessage object returned from the Gemini 2.5 model response is not being serialized to a plain dict/JSON-compatible format before being passed to the LLM judge scorer. This works with other models (e.g., GPT-4o-mini) that may return serializable types by default.

Labels

• python
• vertex-ai

---

Linear issue: https://linear.app/braintrustdata/issue/BRA-2972/json-serialization-error-with-gemini-25-models-in-multi-turn

[AbhiPrasad]

Claude Reproduction:

---

Summary: Issue #38 - JSON Serialization Error with Gemini 2.5 Models

Reproduction Status: CONFIRMED

Root Cause

File: py/src/braintrust/functions/invoke.py:204

The invoke() function sends HTTP requests using requests.post(json=request), which internally calls Python's standard json.dumps(). This standard encoder cannot serialize Pydantic BaseModel objects like ChatCompletionMessage.

When init_function() is used as a scorer in Eval(), the flow is:

1. Task function returns multi-turn output containing ChatCompletionMessage objects
2. Framework calls scorer with output=<task_output> (containing ChatCompletionMessage)
3. init_function's inner f() passes these kwargs directly to invoke(input=kwargs)
4. invoke() builds a request dict and calls proxy_conn().post("function/invoke", json=request)
5. requests.post(json=...) calls json.dumps(request) → TypeError: Object of type ChatCompletionMessage is not JSON serializable

Why It's NOT Gemini-Specific (Technically)

Both Gemini 2.5 and GPT-4o-mini return ChatCompletionMessage objects via the OpenAI-compatible API. The issue affects any model when:

• The task function returns raw ChatCompletionMessage objects in its output (common in multi-turn)
• An LLM judge scorer via init_function() tries to invoke a remote function with that output

The issue says "GPT-4o-mini works" likely because users were extracting .content strings for single-turn, or their task function handled serialization differently for GPT-4o-mini.

Evidence

Real execution traceback (from repro_issue38_precise.py):

File "invoke.py", line 204, in invoke
    resp = proxy_conn().post("function/invoke", json=request, headers=headers, stream=stream)
  → requests/sessions.py → prepare_body → complexjson.dumps(json, allow_nan=False)
  → json/encoder.py → TypeError: Object of type ChatCompletionMessage is not JSON serializable

Test results:

• 13/13 unit tests pass (test_issue38_unit.py) - confirming serialization behavior
• 7/7 integration tests pass (test_issue38_integration.py) - confirming invoke() and init_function failure paths
• Real invoke() call fails with exact error from issue

Key Comparison

Scenario	Standard json.dumps	Braintrust bt_dumps / bt_safe_deep_copy
ChatCompletionMessage in dict	FAILS TypeError	SUCCEEDS (calls model_dump())
Multi-turn with ChatCompletionMessage list	FAILS TypeError	SUCCEEDS
Plain dict messages	Succeeds	Succeeds
String output (single-turn)	Succeeds	Succeeds

The Fix

The Braintrust SDK already has bt_safe_deep_copy() (in bt_json.py) which handles Pydantic models via model_dump(). The internal span logging pipeline uses this correctly. The fix is to apply the same sanitization in invoke() before passing data to requests.post(json=...).

In py/src/braintrust/functions/invoke.py, around line 182-204, sanitize the input field:

from ..bt_json import bt_safe_deep_copy

# Before building the request, sanitize the input
input = bt_safe_deep_copy(input) if input is not None else None

Or alternatively, serialize using bt_dumps() and pass raw bytes to requests.post(data=...) instead of json=.

"""
Reproduction script for GitHub issue #38:
JSON Serialization Error with Gemini 2.5 Models

The issue: When using init_function() as a scorer, the task output (a ChatCompletionMessage)
is passed to invoke(), which sends it via requests.post(json=request). The standard json.dumps
cannot serialize ChatCompletionMessage objects.

This script demonstrates the exact failure path.
"""

import json
import os
import sys
import traceback

# Ensure we use the local braintrust package
sys.path.insert(0, "/workspace/sdk-python/py/src")

# os.environ["BRAINTRUST_API_KEY"] = "API_KEY"

from openai.types.chat import ChatCompletionMessage

# ============================================================
# Part 1: Demonstrate the core serialization failure
# ============================================================
print("=" * 60)
print("Part 1: Core serialization failure")
print("=" * 60)

# Create a ChatCompletionMessage like what Gemini 2.5 returns via the proxy
msg = ChatCompletionMessage(
    role="assistant",
    content="Based on the context provided, the answer is X.",
)

# This is what happens when a multi-turn conversation output contains ChatCompletionMessage
# objects. The task function returns them, and the scorer receives them.
output_with_messages = [
    {"role": "user", "content": "What is X?"},
    msg,  # This is a ChatCompletionMessage, NOT a dict
]

# Simulate what init_function's scorer does:
# invoke(..., input={"input": ..., "output": output_with_messages, "expected": ...})
scorer_input = {
    "input": [
        {"role": "user", "content": "What is X?"},
        {"role": "assistant", "content": "Previous answer"},
        {"role": "user", "content": "Follow up question"},
    ],
    "output": output_with_messages,
    "expected": "Expected answer",
    "metadata": {},
}

# This simulates what happens inside invoke() at line 204:
# resp = proxy_conn().post("function/invoke", json=request, ...)
# The `json=request` parameter causes requests to call json.dumps(request)
request = {
    "input": scorer_input,
    "metadata": None,
    "tags": None,
    "parent": "some-parent-id",
    "stream": False,
    "api_version": 1,
}

print(f"\nAttempting json.dumps on request containing ChatCompletionMessage...")
try:
    result = json.dumps(request)
    print("SUCCESS - request was serialized")
except TypeError as e:
    print(f"FAILURE - TypeError: {e}")
    print(f"\nThis is exactly the error reported in issue #38!")

# ============================================================
# Part 2: Show that the Braintrust custom encoder handles it
# ============================================================
print("\n" + "=" * 60)
print("Part 2: Braintrust custom encoder CAN handle it")
print("=" * 60)

from braintrust.bt_json import bt_dumps, bt_safe_deep_copy

# bt_dumps uses BraintrustJSONEncoder which handles Pydantic models
try:
    result = bt_dumps(request)
    print(f"bt_dumps succeeded: {result[:200]}...")
except Exception as e:
    print(f"bt_dumps failed: {e}")

# bt_safe_deep_copy converts objects before serialization
try:
    safe_request = bt_safe_deep_copy(request)
    result = json.dumps(safe_request)
    print(f"bt_safe_deep_copy + json.dumps succeeded: {result[:200]}...")
except Exception as e:
    print(f"bt_safe_deep_copy + json.dumps failed: {e}")

# ============================================================
# Part 3: Show the invoke() function uses plain json.dumps
# ============================================================
print("\n" + "=" * 60)
print("Part 3: invoke() uses plain json.dumps via requests.post(json=)")
print("=" * 60)

# Look at what requests does with json= parameter
import requests
from unittest.mock import patch, MagicMock

# Patch requests to capture what it does
original_post = requests.Session.post

captured_error = None

def mock_post(self, url, **kwargs):
    global captured_error
    if 'json' in kwargs:
        try:
            # This is what requests does internally - calls json.dumps
            json.dumps(kwargs['json'])
            print("  requests.post json serialization: SUCCESS")
        except TypeError as e:
            captured_error = e
            print(f"  requests.post json serialization: FAILURE - {e}")
    raise ConnectionError("Mock - not actually sending")

print("\nSimulating invoke() -> proxy_conn().post() -> requests.post(json=request)...")
with patch.object(requests.Session, 'post', mock_post):
    try:
        from braintrust.functions.invoke import invoke
        import braintrust
        braintrust.login(api_key=os.environ["BRAINTRUST_API_KEY"])

        invoke(
            project_name="test",
            slug="test-scorer",
            input=scorer_input,
        )
    except ConnectionError:
        pass
    except Exception as e:
        if "not JSON serializable" in str(e):
            print(f"  Confirmed: invoke() fails with: {e}")
        else:
            print(f"  Other error (expected during mock): {type(e).__name__}: {e}")

if captured_error:
    print(f"\n  >>> CONFIRMED: The json serialization in invoke() fails because")
    print(f"  >>> requests.post(json=...) uses standard json.dumps which cannot")
    print(f"  >>> serialize ChatCompletionMessage objects.")

# ============================================================
# Part 4: Demonstrate fix approach
# ============================================================
print("\n" + "=" * 60)
print("Part 4: Proposed fix - use bt_safe_deep_copy before sending")
print("=" * 60)

# Fix: sanitize the input before passing to requests
safe_scorer_input = bt_safe_deep_copy(scorer_input)
safe_request = {
    "input": safe_scorer_input,
    "metadata": None,
    "tags": None,
    "parent": "some-parent-id",
    "stream": False,
    "api_version": 1,
}

try:
    result = json.dumps(safe_request)
    print(f"After bt_safe_deep_copy, json.dumps succeeds!")
    print(f"Serialized length: {len(result)} bytes")
except TypeError as e:
    print(f"Still fails: {e}")

print("\n" + "=" * 60)
print("SUMMARY")
print("=" * 60)
print("""
ROOT CAUSE:
  In invoke.py line 204:
    resp = proxy_conn().post("function/invoke", json=request, ...)

  The `json=request` parameter uses standard json.dumps which cannot
  serialize Pydantic model objects like ChatCompletionMessage.

  When init_function() is used as a scorer, it receives the task output
  directly. If the task returns ChatCompletionMessage objects (common
  with multi-turn conversations), these non-serializable objects end up
  in the request payload.

WHY GEMINI 2.5 SPECIFIC:
  The issue manifests specifically with Gemini 2.5 because the Braintrust
  proxy returns responses as OpenAI-compatible ChatCompletionMessage
  objects. The task function passes these directly to scorers.

WHY MULTI-TURN SPECIFIC:
  Single-turn responses typically extract just the content string.
  Multi-turn conversations often preserve the full message objects
  (including ChatCompletionMessage instances) in the output.

FIX:
  In invoke.py, sanitize the request using bt_safe_deep_copy() or
  bt_dumps() before passing to requests.post(), similar to how the
  span logging pipeline handles it.
""")

[viadezo1er]

As advised, using bt_dumps solves the issue. Works both with and without orjson, the custom encoder successfully deserialize the json.