Max BM25 score / BM25 normalization

[svilupp]

• Added BM25 normalization kwarg to RAGTools.jl to enable 0-1 query-specific normalization of BM25 scores for easier filtering and comparison. See ?RT.bm25 and ?RT.max_bm25_score for more information.

Note: this also enables easier reciprocal rank fusion between BM25 and cosine similarity.

Example

documents = [["this", "is", "a", "test"], ["this", "is", "another", "test"], ["foo", "bar", "baz"]]
dtm = document_term_matrix(documents)
query = ["this"]
scores = bm25(dtm, query)
scores_norm = bm25(dtm, query; normalize = true)

## Make it more accurate for your dataset/index
normalize_max_tf = 3 # assume max term frequency is 3 (what is likely for your dataset? depends on chunk size, preprocessing, etc.)
normalize_min_doc_rel_length = minimum([x for x in RT.doc_rel_length(dtm) if x > 0]) |> Float32
scores_norm = bm25(dtm, query; normalize = true, normalize_max_tf, normalize_min_doc_rel_length)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.07%. Comparing base (9b783ec) to head (c221abe).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #267      +/-   ##
==========================================
+ Coverage   92.04%   92.07%   +0.03%     
==========================================
  Files          49       49              
  Lines        4826     4846      +20     
==========================================
+ Hits         4442     4462      +20     
  Misses        384      384              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.