How does Selectools eval compare to DeepEval, Promptfoo, LangSmith, and others?

[johnnichev]

We built a comprehensive eval framework directly into Selectools — no separate install, no SaaS account. Here's how it compares to the alternatives.

Comparison Matrix

Capability	Selectools	DeepEval	Promptfoo	Braintrust	LangSmith	Ragas
Type	Built-in	Separate lib	CLI (Node.js)	SaaS	SaaS	Library
Separate install	No	Yes	Yes (Node.js)	Yes + account	Yes + account	Yes
Evaluators	22 (12 det + 10 LLM)	50+	~20	~10	~8	6 (RAG)
Tool call assertions	Name + args	Name + args	Schema only	Custom	Custom	No
A/B testing	PairwiseEval	No	Side-by-side	Experiments	Experiments	No
Regression detection	Local (JSON)	Cloud only	CLI + GH	SaaS	SaaS	No
Snapshot testing	SnapshotStore	No	No	No	No	No
JUnit XML	Yes	Via pytest	Yes	No	Via pytest	No
HTML report	Interactive (charts)	Cloud	Self-contained	SaaS UI	SaaS UI	No
Live dashboard	serve_eval()	No	Web UI	SaaS	SaaS	No
SVG badges	generate_badge()	No	No	No	No	No
Test case gen	generate_cases()	No	Adversarial	No	No	Test data
Weighted accuracy	Yes	No	No	No	No	No
Works offline	Yes	Partial	Yes	No	No	Partial
Zero deps	Yes	No	No (Node.js)	No	No	No
Price	Free (included)	Free + SaaS	Free + enterprise	Free + $249/mo	Free + $39/seat	Free

Quick Example

from selectools.evals import EvalSuite, TestCase

suite = EvalSuite(agent=agent, cases=[
    TestCase(input="Cancel account", expect_tool="cancel_sub", expect_no_pii=True),
    TestCase(input="Balance?", expect_contains="balance", expect_latency_ms_lte=500),
])
report = suite.run()
print(report.accuracy)     # 0.95
print(report.latency_p50)  # 142ms
report.to_html("report.html")

What we'd love feedback on

• Which evaluators are you missing?
• What eval workflows do you use today?
• Would you switch from DeepEval/Promptfoo if the evaluator count was higher?

Full docs: Eval Framework