Comprehensive documentation improvements for scorers and evaluators #163
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Braintrust eval reportAutoevals (json-scorer-docs-1768323434)
|
Qard
changed the title
Qard
force-pushed
the
json-scorer-docs
branch
2 times, most recently
from
95ca6c8 to
e2f72e5
Compare
ankrgyl
approved these changes
Jan 13, 2026
Addresses multiple documentation requests with practical examples and complete reference materials. - Custom JSON scorers (#137): Added detailed JSDoc/docstrings with examples showing how to compose scorers (schema validation + semantic comparison) and create custom validators (API response validation) - Context-based evaluators with Braintrust Eval (#82): Added RAGAS module documentation demonstrating how to pass context through metadata in Eval runs, with practical examples for both TypeScript and Python - Complete scorer reference (#101): Created comprehensive SCORERS.md documenting all 30+ available scorers with parameters, score ranges, interpretation guidelines, and usage examples organized by category (LLM-as-judge, RAG, heuristic, JSON, list) Files modified: - js/json.ts, py/autoevals/json.py: Enhanced with module-level examples - js/ragas.ts, py/autoevals/ragas.py: Added Eval integration examples - SCORERS.md: New 650+ line comprehensive reference
Qard
force-pushed
the
json-scorer-docs
branch
from
e2f72e5 to
421fe0e
Compare
clutchski
approved these changes
Jan 13, 2026
Braintrust eval report
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive documentation improvements addressing multiple user requests for better examples and reference materials.
Changes
1. Custom JSON Scorers (#137)
2. Context-Based Evaluators with Braintrust Eval (#82)
contextthroughmetadatain Eval runs3. Complete Scorer Reference (#101)
SCORERS.mdreference documentationDocumentation Files
js/json.ts- Enhanced with module and JSDoc examplespy/autoevals/json.py- Enhanced with module examplesjs/ragas.ts- Added comprehensive module documentationpy/autoevals/ragas.py- Added Braintrust Eval integration examplesSCORERS.md- New comprehensive scorer reference (650+ lines)TODO.md- Updated to track documentation progressIssues Addressed
Closes #137 - Custom scorer for JSONs
Closes #82 - Better docs for context-based evaluators
Closes #101 - Document supported scores