🔬 mtcb ✨

The benchmark for evaluating chunking strategies in RAG pipelines.

Installation • Quick Start • Benchmarks • Usage • Metrics

MTCB (Massive Text Chunking Benchmark) is a standardized evaluation framework for text chunking in RAG systems. It measures how well your chunking and embedding strategy retrieves relevant passages across 9 diverse domains, from legal contracts to scientific papers. Built on top of Chonkie.

📦 Installation

pip install mtcb

🚀 Quick Start

Run the lightweight nano benchmark to evaluate a chunking strategy in minutes:

from mtcb import NanoBenchmark
from chonkie import RecursiveChunker

benchmark = NanoBenchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

🧩 Available Benchmarks

Full Benchmark

The full MTCB benchmark spans 9 domains with ~17k questions across ~3k documents:

Dataset	Domain	Documents	Questions
🧸 Gacha	Classic Literature (Gutenberg)	100	2,878
💼 Ficha	SEC Financial Filings	88	1,331
📝 Macha	GitHub READMEs	445	1,812
💻 Cocha	Multilingual Code	1,000	2,372
📊 Tacha	Financial Tables (TAT-QA)	349	2,065
🔬 Sencha	Scientific Papers (QASPER)	243	1,507
⚖️ Hojicha	Legal Contracts (CUAD)	194	1,568
🏥 Ryokucha	Medical Guidelines (NICE/CDC/WHO)	241	1,351
🎓 Genmaicha	MIT OCW Lecture Transcripts	250	2,037
	Total	2,910	16,921

Nano Benchmark

For fast iteration during development, MTCB provides a lightweight nano benchmark with ~100 questions per dataset. Documents are selected to maximize question density:

Dataset	Domain	Documents	Questions
🧸 nano-gacha	Classic Literature	5	100
💼 nano-ficha	SEC Financial Filings	5	100
📝 nano-macha	GitHub READMEs	19	100
💻 nano-cocha	Multilingual Code	26	100
📊 nano-tacha	Financial Tables	11	100
🔬 nano-sencha	Scientific Papers	13	100
⚖️ nano-hojicha	Legal Contracts	10	100
🏥 nano-ryokucha	Medical Guidelines	12	100
🎓 nano-genmaicha	Lecture Transcripts	7	100
	Total	108	900

🔧 Usage

MTCB works with Chonkie — any chunker that extends chonkie.BaseChunker is supported out of the box.

Full Benchmark

Run the complete benchmark across all 9 domains:

from mtcb import Benchmark
from chonkie import RecursiveChunker

benchmark = Benchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

Individual Evaluators

Run a single domain-specific evaluator:

from mtcb import GachaEvaluator
from chonkie import RecursiveChunker

evaluator = GachaEvaluator(
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
    cache_dir="./cache"
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Custom Datasets

Evaluate on your own corpus using SimpleEvaluator:

from mtcb import SimpleEvaluator
from chonkie import RecursiveChunker

evaluator = SimpleEvaluator(
    corpus=["Your document text here...", "Another document..."],
    questions=["What is X?", "How does Y work?"],
    relevant_passages=["passage that must be in retrieved chunk", "another passage"],
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Dataset Generation

Generate verified QA datasets from your own documents:

from mtcb import DatasetGenerator

generator = DatasetGenerator(deduplicate=True)
result = generator.generate(
    corpus=["Your document text..."],
    samples_per_document=10,
    output_path="./output.jsonl",
)

print(f"Generated {result.total_verified} verified samples")
for sample in result.samples:
    print(f"Q: {sample.question}")
    print(f"A: {sample.answer}")

📊 Metrics

MTCB evaluates retrieval quality using:

• Recall@k: Percentage of questions where the relevant passage appears in the top-k results
• Precision@k: Ratio of relevant chunks in the top-k results
• MRR@k: Mean Reciprocal Rank — how high the first relevant result ranks
• NDCG@k: Normalized Discounted Cumulative Gain — position-weighted relevance scoring

📚 Citation

If you use MTCB in your research, please cite:

@software{mtcb2025,
  author = {Bhavnick Minhas and Shreyash Nigam},
  title = {MTCB: Massive Text Chunking Benchmark},
  url = {https://github.com/chonkie-inc/mtcb},
  version = {0.1.0},
  year = {2025},
}