Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
49e4b60
Add tools and judge rubric data to saved runs and TUI
claude Oct 23, 2025
0cbd7ff
Merge pull request #1 from cdreetz/claude/add-save-flag-support-011CU…
cdreetz Oct 23, 2025
1f0e9fc
Revert "Add tools and judge rubric data to saved runs and TUI"
cdreetz Oct 23, 2025
77640ad
Merge pull request #2 from cdreetz/revert-1-claude/add-save-flag-supp…
cdreetz Oct 23, 2025
8967698
Merge branch 'PrimeIntellect-ai:main' into main
cdreetz Dec 11, 2025
ff4d830
Merge branch 'PrimeIntellect-ai:main' into main
cdreetz Jan 16, 2026
edf76b2
Merge branch 'PrimeIntellect-ai:main' into main
cdreetz Mar 22, 2026
1837bf7
Merge branch 'PrimeIntellect-ai:main' into main
cdreetz Apr 3, 2026
32c9991
Merge branch 'PrimeIntellect-ai:main' into main
cdreetz May 8, 2026
35418c2
Add Braintrust tracing instrumentation for verifiers core
devin-ai-integration[bot] May 9, 2026
3f48932
Fix span nesting bug and ty CI failure
devin-ai-integration[bot] May 9, 2026
5bcfc69
Fix race condition: use contextvars instead of instance attr for span…
devin-ai-integration[bot] May 9, 2026
16648a1
Update uv.lock for braintrust dependency
devin-ai-integration[bot] May 9, 2026
f45c2de
Pass state=state in StatefulToolEnv.env_response call_tool for tool t…
devin-ai-integration[bot] May 9, 2026
a48019e
Add per-rollout spans inside _run_group_states for nested tracing
devin-ai-integration[bot] May 9, 2026
76056d3
Fix CancelledError crash and defer group rollout span completion unti…
devin-ai-integration[bot] May 9, 2026
08962ef
Add run-level tags to Braintrust traces
devin-ai-integration[bot] May 9, 2026
59f1b63
Use ContextVar for run tags and guard behind enabled() check
devin-ai-integration[bot] May 9, 2026
f1af4f0
Initialize response=None to prevent NameError in finally block
devin-ai-integration[bot] May 9, 2026
b0460f6
Remove dead if-guard in except BaseException handler
devin-ai-integration[bot] May 9, 2026
6bb9bcc
Move braintrust tracing to verifiers/envs/experimental/braintrust_tra…
devin-ai-integration[bot] May 9, 2026
c2abc54
Use index-based span tracking in _run_group_states
devin-ai-integration[bot] May 9, 2026
472ea2a
Fix type annotation for bt_rollout_spans list
devin-ai-integration[bot] May 9, 2026
5558e02
Fix ty: use list comprehension for bt_rollout_spans init
devin-ai-integration[bot] May 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,9 @@ browser = [
renderers = [
"renderers>=0.1.6",
]
braintrust = [
"braintrust>=0.0.160",
]
rl = [
"torch>=2.8.0,<2.9.0",
"transformers>=4.56.2",
Expand Down Expand Up @@ -230,6 +233,12 @@ invalid-method-override = "ignore"
invalid-assignment = "ignore"
not-iterable = "ignore"

[[tool.ty.overrides]]
include = ["verifiers/envs/experimental/braintrust_tracing/**"]

[tool.ty.overrides.rules]
unresolved-import = "ignore"

[tool.coverage.run]
source = ["verifiers"]
omit = [
Expand Down
139 changes: 130 additions & 9 deletions uv.lock

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions verifiers/envs/experimental/braintrust_tracing/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""Experimental Braintrust tracing variants of the core environment classes.

Usage::

from verifiers.envs.experimental.braintrust_tracing.stateful_tool_env import StatefulToolEnv

These classes are drop-in replacements for their non-tracing counterparts.
Set ``BRAINTRUST_API_KEY`` and optionally ``VF_BRAINTRUST_PROJECT`` to enable
trace logging to Braintrust.
"""

from verifiers.envs.experimental.braintrust_tracing.environment import Environment
from verifiers.envs.experimental.braintrust_tracing.multiturn_env import MultiTurnEnv
from verifiers.envs.experimental.braintrust_tracing.stateful_tool_env import (
StatefulToolEnv,
)
from verifiers.envs.experimental.braintrust_tracing.tool_env import ToolEnv

__all__ = [
"Environment",
"MultiTurnEnv",
"ToolEnv",
"StatefulToolEnv",
]
Loading
Loading