Validate the computed chunk_overlap (a float ≥ 1.0 silently drops all content)

[aaravanmay]

TokenChunker.__init__ only checks chunk_overlap >= chunk_size for int inputs. A float overlap is converted to int(chunk_overlap * chunk_size), which can be >= chunk_size while skipping that int-only guard — so the step chunk_size - chunk_overlap goes <= 0, range() yields nothing, and chunk() returns []: the whole document is silently dropped, with no error. The docstring already documents ValueError: If ... chunk_overlap >= chunk_size.

For example, on the current release:

from chonkie import TokenChunker
c = TokenChunker(tokenizer="character", chunk_size=10, chunk_overlap=1.5)
c.chunk("a" * 100)   # -> []  (the entire document is dropped, no error)

This validates the computed self.chunk_overlap (ints and floats alike) right after it's assigned:

if self.chunk_overlap >= self.chunk_size or self.chunk_overlap < 0:
    raise ValueError("chunk_overlap must be >= 0 and less than chunk_size")

Includes a regression test (character tokenizer, no model download) that fails on main and passes with the change, plus a test that a valid fractional overlap (e.g. 0.1) still chunks normally.

Found while building a small fault-injection tester for agent tools.

Summary by CodeRabbit

• Bug Fixes

  • Improved validation of chunk overlap parameters to prevent invalid configurations that could result in no chunks being produced.

• Tests

  • Added regression tests to verify proper validation of overlap parameters across various input scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8fd12fc6-41b8-4081-9efb-b1837035af1a

📥 Commits

Reviewing files that changed from the base of the PR and between ac3a74f and 383cdbe.

📒 Files selected for processing (2)

• src/chonkie/chunker/token.py
• tests/test_token_overlap_validation.py

---

📝 Walkthrough

Walkthrough

TokenChunker now validates the computed chunk_overlap value immediately after it is calculated, rejecting overlaps that are negative or greater than or equal to chunk_size. This prevents silent chunking failures when fractional overlaps are converted to integers. Regression tests confirm the validation rejects invalid float overlaps while allowing valid fractional ones.

Changes

Chunk overlap validation

Layer / File(s)	Summary
Overlap validation in TokenChunker initialization src/chonkie/chunker/token.py	After computing chunk_overlap, validation ensures the value is not negative and is strictly less than chunk_size, catching edge cases where float inputs converted to integers would produce invalid overlaps.
Regression tests for overlap validation tests/test_token_overlap_validation.py	Module-level docstring documents the prior failure mode and new expected behavior. Two pytest tests verify that invalid float overlaps (e.g., 1.5) raise ValueError during construction, while valid fractional overlaps (e.g., 0.1) still produce chunks.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

🐰 A token's overlap, once fractionally small,
Could vanish to zero and chunk not at all!
But now validation stands guard at the gate,
Rejecting the broken, allowing the great!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main fix: adding validation for computed chunk_overlap to prevent silent content loss when float values >= 1.0 are converted to integers that exceed chunk_size.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request adds validation to ensure that computed chunk overlaps in TokenChunker do not exceed the chunk size or fall below zero, preventing silent content dropping. It also includes regression tests for these cases. The review feedback identifies a subtle bug where negative float overlaps (like -0.05) truncate to 0 and bypass the validation, and suggests checking the raw input directly. Additionally, it recommends adding test cases to cover negative overlaps.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

There is a subtle correctness issue here with negative float overlaps. If a user passes a negative float overlap like -0.05 with chunk_size=10, the expression int(-0.05 * 10) evaluates to int(-0.5), which truncates to 0 in Python. As a result, self.chunk_overlap becomes 0, and the check self.chunk_overlap < 0 will evaluate to False, silently bypassing the validation. Checking the raw chunk_overlap input directly for negative values resolves this issue.

Suggested change

	# Validate the COMPUTED overlap, not just the int input: a float chunk_overlap >= 1.0
	# is turned into int(chunk_overlap * chunk_size), which can be >= chunk_size and was
	# skipped by the int-only guard above. That makes the step (chunk_size - chunk_overlap)
	# zero or negative, so range() yields nothing and chunk() silently returns [] - the
	# whole document is dropped. Enforce the docstring's contract for ints and floats alike.
	if self.chunk_overlap >= self.chunk_size or self.chunk_overlap < 0:
	raise ValueError("chunk_overlap must be >= 0 and less than chunk_size")
	# Validate the computed overlap and ensure the input overlap is non-negative.
	# A negative float like -0.05 can truncate to 0 via int(), bypassing the < 0 check on self.chunk_overlap.
	if chunk_overlap < 0 or self.chunk_overlap >= self.chunk_size:
	raise ValueError("chunk_overlap must be >= 0 and less than chunk_size")

[gemini-code-assist]

It is highly recommended to add test cases verifying that negative overlaps (both integer and float) are correctly rejected during initialization.

Suggested change

	def test_float_overlap_ge_one_is_rejected():
	with pytest.raises(ValueError):
	TokenChunker(tokenizer="character", chunk_size=10, chunk_overlap=1.5)
	def test_float_overlap_ge_one_is_rejected():
	with pytest.raises(ValueError):
	TokenChunker(tokenizer="character", chunk_size=10, chunk_overlap=1.5)
	def test_negative_overlap_is_rejected():
	with pytest.raises(ValueError):
	TokenChunker(tokenizer="character", chunk_size=10, chunk_overlap=-0.5)
	with pytest.raises(ValueError):
	TokenChunker(tokenizer="character", chunk_size=10, chunk_overlap=-1)

[aaravanmay]

Following up on this PR — I found it with faultline, an open-source tester I built that feeds agents/pipelines deliberately broken inputs and fails CI when they silently produce wrong output (no LLM judge, deterministic). I've already written a 30-line suite for chonkie that would have caught this overlap bug automatically. Want me to open it as a separate PR with a non-blocking GitHub Action so you can watch it for a week before deciding? Zero config on your side — if it's noisy, close it, no hard feelings.