OCR Text & PDF Search Engine

A full-text search engine for OCR'd PDF collections. Text is parsed into paragraphs and indexed in Elasticsearch, with a Flask backend serving search results and a simple HTML frontend for browsing and previewing the matched PDF page.

Large PDF archives (scanned books, reports, records) are often unsearchable — the text is there thanks to OCR, but there is no way to find which file, and which page, mentions what we are looking for without opening everything one by one. This project closes that gap: point it at a folder of OCR'd text, and it gives you a search box that jumps straight to the matching page in the original PDF.

This is an MVP — the smallest version that proves the core idea works, not a production-ready system. A few scope decisions reflect that:

• Paragraph-level indexing, not full-document. Each paragraph is indexed separately and tagged with its page number, so a search result points to the exact page.
• Fuzzy + phrase matching over a custom index. Elasticsearch handles matching and highlighting out of the box, which let the effort go into getting search quality right (including handling typos) instead of building indexing infrastructure from scratch.
• Local dev setup, not hardened for production. SSL verification is disabled by default (see Notes below) — running against a local Elasticsearch instance.
• No data shipped with the code. PDFs and OCR text are git-ignored.

How it works

• ingest.py — Parses OCR .txt files from a folder, splits them into paragraphs, tags each with a page number, and bulk-indexes them into Elasticsearch under the book_paragraphs index.
• main.py — Flask backend. Exposes a /search endpoint (phrase + fuzzy matching with highlighting) and serves the original PDFs from a pdfs/ folder.
• index.html — Frontend. Search box on the left, results on the left panel, PDF preview (jumping to the matched page) on the right.

Setup

1. Install dependencies

   pip install -r requirements.txt
   

2. Set up Elasticsearch Have a local Elasticsearch instance running at https://localhost:9200 with security enabled (default elastic user).

3. Configure your password Create a file named .env in the project folder with:

   ELASTIC_PASSWORD=your_real_password
   

   .env is git-ignored and is never uploaded to the repo — keep it local only.

4. Add your data

   • Place OCR'd text files in all_ocr_texts/ (one .txt per book, page breaks marked as Page N on their own line, paragraphs separated by a blank line).
   • Place the matching PDFs in pdfs/, using the same base filename as each .txt file.

5. Index your books

   python ingest.py
   

6. Run the backend

   python main.py
   

   This starts the Flask server at http://localhost:8000.

7. Open the frontend Open index.html in your browser and start searching.

Notes

• This is set up for local development — verify_certs=False disables SSL certificate checking, which is fine for a local Elasticsearch instance but should not be used against a production/public server.
• pdfs/ and all_ocr_texts/ are git-ignored by default since they tend to be large files; you'll need to populate them yourself after cloning.

Challenges I ran into

• Relevance scoring was harder than expected. Elasticsearch's default scoring didn't always put the most useful paragraph first — sometimes a paragraph that just repeated a common word ranked above the one that actually answered the query. Had to spend time experimenting with query type (phrase match vs fuzzy match) and boosting to get results that actually matched what a user would consider "correct."
• Fuzzy matching was a double-edged sword. It helped catch OCR errors (misread characters, broken words), but turned up too loose at times — unrelated paragraphs would match just because a word was a couple of edits away from the search term. Had to balance fuzziness against precision so the top results stayed trustworthy.

Possible next steps

• Auth / access control if this ever handles more than one user's documents
• Better relevance tuning based on actual queries instead of default fuzzy matching
• An ingest pipeline that runs OCR itself, instead of assuming pre-OCR'd text
• Logging what people search for (and don't find), to see what the index is missing