Add: bm25 (Go) — FastEmbed Qdrant/bm25 sparse encoder#2
Conversation
|
Hey @harsh04 thnx for your contribution, the Go bm25 port is clearly useful work and a clean writeup |
… link Per review: add a runnable Go hybrid-search demo under bm25-go-hybrid/ and point the Integrations row at it, matching the repo's in-repo-demo pattern.
|
Totally fair — thanks for the quick, clear review. Reworked it into an actual in-repo demo: It's the first Go demo in the list, so happy to adjust the folder placement, naming, or format to match whatever convention you'd prefer. |
|
@harsh04 Thnx, this is the in-repo demo i wanted, left a few inline comments related to dense model, expected output and match folder name. Sort those and will merge it |
|
|
||
| // denseEmbed is a stand-in. Replace with a real embedding model and keep the | ||
| // dimension/distance consistent with the collection below. | ||
| func denseEmbed(text string) []float32 { |
There was a problem hiding this comment.
this makes the dense vectors somehow random, hashing bytes into buckets isn't semantic. so the dense arm doesn't actually contribute and the ranking is BM25 only , maybe Gemini-embeddings/ example would give a u a real dense model or choose whatever you want for this
| The dense embedder here is a tiny stand-in — swap `denseEmbed` for a real model | ||
| (Gemini, OpenAI, etc.) and bump `denseDim`. | ||
|
|
||
| ## Run |
There was a problem hiding this comment.
can we add the expected output here? (printed hits) so readers see what success looks like
| log.Fatal(err) | ||
| } | ||
|
|
||
| for _, query := range []string{"when can I check in", "is there wifi"} { |
There was a problem hiding this comment.
these queries share words with the docs, so bm25 alone finds them. maybe add one with different words but same meaning so the dense side actually matters (after fixing the dense model)
| | --- | --- | --- | --- | | ||
| | ✅ | **Chonkie** | Advanced text chunking with Qdrant handshake | [Tutorial](chonkie/Chonkie_Qdrant_Handshake.ipynb) | | ||
| | 🚧 | **FastEmbed** | Fast, lightweight embedding library | Coming Soon | | ||
| | ✅ | **bm25 (Go)** | Dense + sparse (BM25) hybrid search in Go, using a FastEmbed-compatible sparse encoder | [Demo](bm25-go-hybrid/) | |
There was a problem hiding this comment.
folder name is bm25-go-hybrid but here is bm25 (Go), can you match them
…utput, folder-name match - Replace the FNV stand-in with real Gemini embeddings (gemini-embedding-001, 1536-dim, L2-normalized), so the dense arm actually contributes - Add queries that share no words with their target docs, so the dense arm is demonstrably doing the work (not BM25 alone) - Document GEMINI_API_KEY setup and add real expected output to the demo README - Rename the integration table row to match the folder (bm25-go-hybrid)
|
Thanks for the detailed review — all four sorted:
Verified end-to-end against a live Qdrant. Ready for another look whenever you have a sec 🙏 |
|
all four confirmed. run it myself, the outputs matches the readme. Great work! |
2 participants
Adds bm25 (Go) to the Integrations table.
A byte-for-byte Go port of FastEmbed's
Qdrant/bm25sparse encoder, so Go services can build BM25 sparse vectors that match a FastEmbed-indexed Qdrant corpus — the sparse half of hybrid search. Term-frequencies only (Qdrant applies IDF server-side via the sparse modifier). 16 languages verified against FastEmbed, MIT licensed.Placed next to the existing FastEmbed row as the Go/sparse counterpart.