Skip to content

Add comprehensive semantic search implementation guide - #3002

Open
amirbiron wants to merge 3 commits into
mainfrom
claude/semantic-search-guide-2MljO
Open

Add comprehensive semantic search implementation guide#3002
amirbiron wants to merge 3 commits into
mainfrom
claude/semantic-search-guide-2MljO

Conversation

@amirbiron

@amirbiron amirbiron commented Jan 30, 2026

Copy link
Copy Markdown
Owner
  • 11-step implementation guide for hybrid text+vector search
  • MongoDB Atlas Vector Search integration with Gemini embeddings
  • Code chunking strategy (220 lines with 40-line overlap)
  • RRF (Reciprocal Rank Fusion) for merging search results
  • Background worker for incremental embedding updates
  • API endpoints and frontend integration examples
  • Migration script for existing data
  • Full test examples

https://claude.ai/code/session_01VVd75CJr4ZNZnE8Nj6HmAj


Note

Low Risk
Documentation-only change; no runtime code paths are modified.

Overview
Adds a new, comprehensive Hebrew implementation guide (docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md) describing how to add semantic/hybrid code search to CodeBot.

The guide documents proposed DB schema/index changes, an async Gemini-based embedding service, code chunking, a hybrid MongoDB Atlas text+vector query with RRF ranking fixes/thresholds, a background embedding worker, an API endpoint, frontend wiring, migration steps, and example tests/config/env vars.

Written by Cursor Bugbot for commit 35a6798. This will update automatically on new commits. Configure here.

- 11-step implementation guide for hybrid text+vector search
- MongoDB Atlas Vector Search integration with Gemini embeddings
- Code chunking strategy (220 lines with 40-line overlap)
- RRF (Reciprocal Rank Fusion) for merging search results
- Background worker for incremental embedding updates
- API endpoints and frontend integration examples
- Migration script for existing data
- Full test examples

https://claude.ai/code/session_01VVd75CJr4ZNZnE8Nj6HmAj
@github-actions

github-actions Bot commented Jan 30, 2026

Copy link
Copy Markdown

🧯 Dangerous deletes guard report

Policy: see .cursorrules — dangerous deletions are blocked unless wrapped safely.

Summary:

  • Flagged findings (blocking): 0
    0
  • Excluded matches (not blocking): 15
  • Total matches (all files): 125

Flagged findings (file:line:snippet):
(none)

Excluded matches (by path pattern)
./Dockerfile:42:    rm -rf /var/lib/apt/lists/*
./Dockerfile:121:    rm -rf /var/lib/apt/lists/*
./webapp/static/js/md_preview.bundle.js.map:4:  "sourcesContent": ["// Markdown-it plugin to render GitHub-style task lists; see\n//\n// https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments\n// https://github.com/blog/1825-t … [truncated]
./README.md:842:find . -name "__pycache__" -exec rm -rf {} +
./docs/DOCUMENTATION_GUIDE.md:453:rm -rf _build
./docs/Makefile:24:	rm -rf $(BUILDDIR)
./node_modules/mermaid/dist/chunks/mermaid.core/chunk-KS23V3DP.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequence  … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.esm.min/chunk-4HFYJGYH.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequen … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.esm.min/chunk-4HFYJGYH.mjs:1:var r={name:"mermaid",version:"11.12.0",description:"Markdown-ish syntax for generating flowcharts, mindmaps, sequence diagrams, class diagrams, gantt charts, git graph … [truncated]
./node_modules/mermaid/dist/chunks/mermaid.esm/chunk-2M32CCKP.mjs.map:4:  "sourcesContent": ["{\n  \"name\": \"mermaid\",\n  \"version\": \"11.12.0\",\n  \"description\": \"Markdown-ish syntax for generating flowcharts, mindmaps, sequence d … [truncated]
./node_modules/mermaid/dist/mermaid.min.js:1524:`,"getStyles"),c1e=RQe});var h1e={};dr(h1e,{diagram:()=>NQe});var NQe,f1e=N(()=>{"use strict";$ge();a1e();l1e();u1e();NQe={parser:Fge,db:n1e,renderer:o1e,styles:c1e}});var m1e,g1e=N(()=>{"use  … [truncated]
./node_modules/mermaid/dist/mermaid.js.map:4:  "sourcesContent": ["/**\n* Default values for dimensions\n*/\nconst defaultIconDimensions = Object.freeze({\n\tleft: 0,\n\ttop: 0,\n\twidth: 16,\n\theight: 16\n});\n/**\n* Default values for tr … [truncated]
./node_modules/mermaid/dist/mermaid.min.js.map:4:  "sourcesContent": ["/**\n* Default values for dimensions\n*/\nconst defaultIconDimensions = Object.freeze({\n\tleft: 0,\n\ttop: 0,\n\twidth: 16,\n\theight: 16\n});\n/**\n* Default values fo … [truncated]
./node_modules/katex/package.json:153:    "build": "rimraf dist/ && mkdirp dist && cp README.md dist && rollup -c --failAfterWarnings && webpack && node update-sri.js package dist/README.md",
./node_modules/katex/src/fonts/Makefile:139:	rm -rf pfa ff otf ttf woff woff2

@github-actions

github-actions Bot commented Jan 30, 2026

Copy link
Copy Markdown

⏱️ Performance report

(No performance test durations collected. Mark tests with @pytest.mark.performance.)

@github-actions

github-actions Bot commented Jan 30, 2026

Copy link
Copy Markdown

📖 Documentation Preview

The documentation has been built successfully!

To view locally:

  1. Download the artifacts
  2. Extract the zip file
  3. Open index.html in your browser

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a58cda4ec4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
@sentry

sentry Bot commented Jan 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md
Bug fixes:
- Add $setWindowFields stages to compute vectorRank/textRank before RRF
  (was using undefined variables, causing all results to get same score)
- Lower min_score from 0.5 to 0.005 (RRF max score is ~0.036)
- Convert EmbeddingService to async (httpx.AsyncClient)
  (was blocking event loop in worker)
- Update flags when content unchanged to prevent infinite retry loop
- Delete old chunks before checking if new ones exist
  (prevents stale chunks persisting when embeddings fail)

https://claude.ai/code/session_01VVd75CJr4ZNZnE8Nj6HmAj

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 4 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md
Comment thread docs/SEMANTIC_SEARCH_IMPLEMENTATION_GUIDE.md Outdated
Additional bug fixes:
- Add missing await for query embedding generation
- Add missing await for snippet metadata embedding
- Include chunkIndex in $group and $project stages
  (was returning chunk 0 for all results)
- Add ge=10 constraint for CHUNK_SIZE_LINES config
- Use max(1, chunk_size // 2) for step fallback
  (prevents infinite loop when chunk_size is very small)

https://claude.ai/code/session_01VVd75CJr4ZNZnE8Nj6HmAj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants