refactor: extract shared tokenization helper to service/_tokens.py - #25
Merged
Conversation
The NFKD-fold + lowercase + split-on-non-alphanumeric routine was copied across three modules — cross-source title dedup (service/search.py), find_paper Jaccard matching (service/discovery.py), and the deterministic test reranker (reranker/fake.py) — each with its own _NON_ALNUM_RE and four near-identical inline copies of the fold. A divergence in any one would silently break dedup or matching. Centralize on service/_tokens.py with normalize_unicode(), tokenize() (stopwords supplied per call site, since search and discovery use different sets), and the shared NON_ALNUM_RE. No behavior change — the existing search/discovery/reranker tests are the regression guard. Adds tests/unit/test_tokens.py covering the helper directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The NFKD-fold + lowercase + split-on-non-alphanumeric routine was copied across three modules, each with its own
_NON_ALNUM_REand several near-identical inline copies of the fold:service/search.py—_title_key, cross-source title dedup (title + first-author surname)service/discovery.py—_title_tokens/has_significant_tokens/_surname,find_paperJaccard matchingreranker/fake.py—_tokens, deterministic test scoringA divergence in any one would silently break dedup or matching, since they all have to fold identically.
Change
New
service/_tokens.pyexposing:normalize_unicode(text) -> str— the NFKD encode/decode foldtokenize(text, *, stopwords=frozenset()) -> list[str]— fold, lowercase, split, drop empties + stopwords (each call site passes its own stopword set;searchanddiscoverydeliberately use different ones)NON_ALNUM_RE— compiled once at module loadAll three call sites now import from it; their local
_NON_ALNUM_RE,re, andunicodedataimports are gone. No behavior change.Closes #7.
Test plan
ruff check src tests— clean (no unused imports, import order verified)mypy src— cleanpytest -q --strict-markers— 524 passed, 14 skippedtests/unit/test_tokens.py(11 cases: diacritic folding, undecomposable-non-ASCII drop, number/hyphen splitting, stopword filtering, order preservation, empty/punctuation-only input)test_search_service.py/test_discovery_service.py/test_reranker.pypass unchanged — the regression guard for "no behavior change"