Problem
The README headlines a "+0.19 precision@1" improvement, which reads as a universal expectation. In practice the gain depends entirely on how much recall the base embedder leaves on the table.
Your own demo notebook already shows a delta around 0 with a base recall@10 near 0.98 (a saturated base). A new user who tries the adapter on a corpus where it cannot help may wrongly conclude it is broken, when really the precondition just was not met.
Proposed solution
Two small docs changes:
-
Report base recall next to the adapted recall (and the delta) in the README/notebook headline, so the precondition (how much headroom the base leaves) is visible at a glance.
-
A short "When this helps / when it won't" note. From what I can tell it helps when the corpus is reasonably homogeneous and the base embedder is systematically under-calibrated for that domain (e.g. a generic embedder on specialized jargon); it won't move the needle on a heterogeneous corpus, or when the base already saturates.
Alternatives considered
No response
Additional context
I reproduced the pipeline with a local embedder (BGE-M3 via sentence-transformers) on a heterogeneous personal note corpus (~250 markdown notes across very different domains), with an honest held-out split by note and the KB chunked to 512 chars (recall measured at note level). Across every honest configuration the trainer converged to the identity adapter (best-on-validation stayed the untrained one), i.e. no net effect. It looked like a pincer:
- On the heterogeneous corpus there is headroom (base recall@3 around 0.44), but a single linear map cannot capture it (different note clusters would need corrections in different directions, which is not linear), so it degrades held-out notes and model selection keeps identity.
- On a small homogeneous slice (about 10 same-genre docs) a global shift would be the right shape, but the base already saturates (recall@3 = 1.00), so there is nothing to correct.
Caveats, so please weight this accordingly: my corpus is small and idiosyncratic, my lay-register queries are synthetic (I wrote them), and the homogeneous slice was tiny (about 10 notes), so its saturation could be a small-corpus effect as much as a homogeneity effect. I do not currently have a large, genuinely homogeneous in-domain corpus, which is exactly where I'd expect the adapter to shine, so I could not test the favorable case properly. Treat the heterogeneous result as the solid finding and the homogeneous one as a suggestive probe. If you have or can point me to a homogeneous benchmark, I'd gladly run the same harness there.
Thanks for open-sourcing this; the code and the negative-mining notes were genuinely useful to read.
Pre-flight
Problem
The README headlines a "+0.19 precision@1" improvement, which reads as a universal expectation. In practice the gain depends entirely on how much recall the base embedder leaves on the table.
Your own demo notebook already shows a delta around 0 with a base recall@10 near 0.98 (a saturated base). A new user who tries the adapter on a corpus where it cannot help may wrongly conclude it is broken, when really the precondition just was not met.
Proposed solution
Two small docs changes:
Report base recall next to the adapted recall (and the delta) in the README/notebook headline, so the precondition (how much headroom the base leaves) is visible at a glance.
A short "When this helps / when it won't" note. From what I can tell it helps when the corpus is reasonably homogeneous and the base embedder is systematically under-calibrated for that domain (e.g. a generic embedder on specialized jargon); it won't move the needle on a heterogeneous corpus, or when the base already saturates.
Alternatives considered
No response
Additional context
I reproduced the pipeline with a local embedder (BGE-M3 via sentence-transformers) on a heterogeneous personal note corpus (~250 markdown notes across very different domains), with an honest held-out split by note and the KB chunked to 512 chars (recall measured at note level). Across every honest configuration the trainer converged to the identity adapter (best-on-validation stayed the untrained one), i.e. no net effect. It looked like a pincer:
Caveats, so please weight this accordingly: my corpus is small and idiosyncratic, my lay-register queries are synthetic (I wrote them), and the homogeneous slice was tiny (about 10 notes), so its saturation could be a small-corpus effect as much as a homogeneity effect. I do not currently have a large, genuinely homogeneous in-domain corpus, which is exactly where I'd expect the adapter to shine, so I could not test the favorable case properly. Treat the heterogeneous result as the solid finding and the homogeneous one as a suggestive probe. If you have or can point me to a homogeneous benchmark, I'd gladly run the same harness there.
Thanks for open-sourcing this; the code and the negative-mining notes were genuinely useful to read.
Pre-flight