Skip to content

README.md

Latest commit

 

History

History
183 lines (136 loc) · 6.47 KB

README.md

File metadata and controls

183 lines (136 loc) · 6.47 KB
Raw

MTCB Logo

🔬 mtcb ✨

The benchmark for evaluating chunking strategies in RAG pipelines.

PyPI version License GitHub stars Downloads

Installation • Quick Start • Benchmarks • Usage • Metrics

MTCB (Massive Text Chunking Benchmark) is a standardized evaluation framework for text chunking in RAG systems. It measures how well your chunking and embedding strategy retrieves relevant passages across 9 diverse domains, from legal contracts to scientific papers. Built on top of Chonkie.

📦 Installation

pip install mtcb

🚀 Quick Start

Run the lightweight nano benchmark to evaluate a chunking strategy in minutes:

from mtcb import NanoBenchmark
from chonkie import RecursiveChunker

benchmark = NanoBenchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

🧩 Available Benchmarks

Full Benchmark

The full MTCB benchmark spans 9 domains with ~17k questions across ~3k documents:

Dataset Domain Documents Questions
🧸 Gacha Classic Literature (Gutenberg) 100 2,878
💼 Ficha SEC Financial Filings 88 1,331
📝 Macha GitHub READMEs 445 1,812
💻 Cocha Multilingual Code 1,000 2,372
📊 Tacha Financial Tables (TAT-QA) 349 2,065
🔬 Sencha Scientific Papers (QASPER) 243 1,507
⚖️ Hojicha Legal Contracts (CUAD) 194 1,568
🏥 Ryokucha Medical Guidelines (NICE/CDC/WHO) 241 1,351
🎓 Genmaicha MIT OCW Lecture Transcripts 250 2,037
Total 2,910 16,921

Nano Benchmark

For fast iteration during development, MTCB provides a lightweight nano benchmark with ~100 questions per dataset. Documents are selected to maximize question density:

Dataset Domain Documents Questions
🧸 nano-gacha Classic Literature 5 100
💼 nano-ficha SEC Financial Filings 5 100
📝 nano-macha GitHub READMEs 19 100
💻 nano-cocha Multilingual Code 26 100
📊 nano-tacha Financial Tables 11 100
🔬 nano-sencha Scientific Papers 13 100
⚖️ nano-hojicha Legal Contracts 10 100
🏥 nano-ryokucha Medical Guidelines 12 100
🎓 nano-genmaicha Lecture Transcripts 7 100
Total 108 900

🔧 Usage

MTCB works with Chonkie — any chunker that extends chonkie.BaseChunker is supported out of the box.

Full Benchmark

Run the complete benchmark across all 9 domains:

from mtcb import Benchmark
from chonkie import RecursiveChunker

benchmark = Benchmark()
result = benchmark.evaluate(
    chunker=RecursiveChunker(chunk_size=512),
    embedding_model="voyage-3-large",
    k=[1, 5, 10],
)
print(result)

Individual Evaluators

Run a single domain-specific evaluator:

from mtcb import GachaEvaluator
from chonkie import RecursiveChunker

evaluator = GachaEvaluator(
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
    cache_dir="./cache"
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Custom Datasets

Evaluate on your own corpus using SimpleEvaluator:

from mtcb import SimpleEvaluator
from chonkie import RecursiveChunker

evaluator = SimpleEvaluator(
    corpus=["Your document text here...", "Another document..."],
    questions=["What is X?", "How does Y work?"],
    relevant_passages=["passage that must be in retrieved chunk", "another passage"],
    chunker=RecursiveChunker(chunk_size=1000),
    embedding_model="voyage-3-large",
)

result = evaluator.evaluate(k=[1, 3, 5, 10])
print(result)

Dataset Generation

Generate verified QA datasets from your own documents:

from mtcb import DatasetGenerator

generator = DatasetGenerator(deduplicate=True)
result = generator.generate(
    corpus=["Your document text..."],
    samples_per_document=10,
    output_path="./output.jsonl",
)

print(f"Generated {result.total_verified} verified samples")
for sample in result.samples:
    print(f"Q: {sample.question}")
    print(f"A: {sample.answer}")

📊 Metrics

MTCB evaluates retrieval quality using:

  • Recall@k: Percentage of questions where the relevant passage appears in the top-k results
  • Precision@k: Ratio of relevant chunks in the top-k results
  • MRR@k: Mean Reciprocal Rank — how high the first relevant result ranks
  • NDCG@k: Normalized Discounted Cumulative Gain — position-weighted relevance scoring

📚 Citation

If you use MTCB in your research, please cite:

@software{mtcb2025,
  author = {Bhavnick Minhas and Shreyash Nigam},
  title = {MTCB: Massive Text Chunking Benchmark},
  url = {https://github.com/chonkie-inc/mtcb},
  version = {0.1.0},
  year = {2025},
}