feat: PDF full-text ingestion via pdfplumber - #29
Merged
Conversation
Search results carry only title + abstract, so the section-aware chunker and analyze_paper never saw real body text. This adds an optional ingest step that fetches a paper's PDF and stores the extracted body on Paper.full_text — which _embedding_text, the chunker, and the codec already consume. New pdf/ package: a PdfFetcher protocol and HttpPdfFetcher, which streams the download under a 25 MiB cap, rejects non-PDFs by content-type and %PDF magic bytes, extracts page text with pdfplumber in a worker thread (capped at 500k chars — well above the analyzer's 60k prompt cap because the chunker wants the whole body), and treats <200 extracted chars as scanned/image-only. A separate PdfTextCache keyed by paper id (no TTL) avoids re-downloading; an empty cache entry records 'no extractable text'. The fetcher never raises for a PDF-side problem — it returns None and ingest proceeds on title + abstract. LibraryService gains an optional pdf_fetcher; ingest_paper and bulk_ingest fill full_text best-effort (bulk bounded to 4 concurrent downloads). The server auto-enables it when pdfplumber is importable, with RESEARCH_MCP_DISABLE_PDF=1 to opt out. pdfplumber (MIT) ships as the [pdf] extra; reportlab (dev) builds real fixture PDFs so the tests exercise genuine extraction offline. Closes #4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Search results carry only title + abstract, so the section-aware chunker and
analyze_papernever saw real body text. This adds an optional ingest step that fetches a paper's PDF and stores the extracted body onPaper.full_text— which_embedding_text,SectionAwareChunker, and the FAISS codec already consume (no changes needed there).New
pdf/package — aPdfFetcherprotocol andHttpPdfFetcher:%PDFmagic bytes (catches HTML landing pages).<200extracted chars as scanned/image-only and declines (OCR is out of scope).PdfTextCachekeyed bypaper.id(no TTL, atomic tmp+rename) avoids re-downloading; an empty cache entry records "no extractable text".None, and ingest proceeds on title + abstract.Wiring —
LibraryServicegains an optionalpdf_fetcher;ingest_paperandbulk_ingestfillfull_textbest-effort (bulk bounded to 4 concurrent downloads). The server auto-enables it when pdfplumber is importable, withRESEARCH_MCP_DISABLE_PDF=1to opt out (mirrorsRESEARCH_MCP_DISABLE_PUBMED).Dependencies — pdfplumber (MIT, on pdfminer.six) is the
[pdf]extra. pyMuPDF was rejected: it's AGPL, incompatible with this MIT project. reportlab (dev) builds real fixture PDFs so tests exercise genuine extraction offline; CI installs[pdf]so the real-extraction path is covered.Closes #4.
Test plan
ruff check src tests/mypy src— cleanpytest -q --strict-markers— 552 passed, 14 skippedtests/unit/test_pdf_fetcher.py(reportlab fixture + httpxMockTransport): extract, disk-cache hit, HTTP error, HTML landing page, non-PDF magic bytes, byte cap, scanned→None (+cached), char cap, missing-pdfplumber→Nonetests/unit/test_library_service.py: ingest fillsfull_text; graceful-degrade when the fetcher returnsNoneor raises; no-fetcher leaves it unset; no-refetch whenfull_textalready set; bulk fills eacharxiv:1706.03762, extracted 35,539 chars containing "attention" / "transformer" / "bleu" / "encoder"