notes.md

feature list:

1. bm25_score
2. bm25(title)
3. bm25(main_content)
4. bm25(urls_text)
5. Minimum Hamming distance between query and document chunk embeddings
6. attribute(quality_score)
7. attribute(log_page_rank_normalized)
8. queryTermCount
9. BM25 rank percentile over the full BM25 candidate list
10. ANN rank percentile over the full ANN candidate list

Query selection

Use real fast production search queries. Normalize repeated whitespace only. Keep unique queries; observed frequency is low enough to ignore weighting. Keep queries between 1 and 15 words. Exclude:

• site: and inurl: syntax
• Quoted queries
• Uppercase Boolean operators AND, OR, NOT Select reproducibly using ORDER BY xxhash64(...). The one-day window is for testing; the final dataset should contain around 50k queries.

Document sampling

• Sample rank buckets independently from BM25 and ANN.
• Cover top results, ranks around 500, medium-depth, and deep candidates.
• Deduplicate and top up to the desired per-query count.
• Use fewer corpus-random negatives because they are generally too easy.

E.g. Top 1–100: 15 Ranks 100–500: 8 Ranks 500–1,000: 8 Ranks 1,000–5,000: 4 Ranks 5,000–20,000: 2 maybe insist more on the long tail 8 - 8 - 4 - 2, e.g. Top 1–100: 5 Ranks 100–500: 5 Ranks 500–1,000: 5 Ranks 1,000–5,000: 10 Ranks 5,000–20,000: 10

actually the idea is to sample documents at any position that a relevance judge would find relevant cf : "The ideal long-term strategy is model-driven sampling: collect deep documents that the current model promotes or relevant documents it incorrectly leaves below 500."