You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a relevance engineer, I would like to evaluate my cases at a number of different values of k for any given metric. To do so, I currently need to define a new custom scorer for each value of k.
Describe the solution you'd like
Using either the Number of Results to Show value as k, or specifying a separate parameter, k, pass that value in the the scorers in the same fashion as qoption is passed in for individual queries.
This would allow implementing a single scorer for a metric, eg:
RR@k
const k = get_k()
let rank = 0;
eachDoc(function(doc, i) {
if (rank === 0 && hasDocRating(i) && (docRating(i)) > 1) { // Make binary 0,1 are irrelevant, 2 and above is relevant.
rank = i+1; // remember the rank of the first relevant document
}
}, k);
const score = rank > 0 ? 1.0 / rank : 0.0;
setScore(score);
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
As a relevance engineer, I would like to evaluate my cases at a number of different values of k for any given metric. To do so, I currently need to define a new custom scorer for each value of k.
Describe the solution you'd like
Using either the Number of Results to Show value as k, or specifying a separate parameter, k, pass that value in the the scorers in the same fashion as qoption is passed in for individual queries.
This would allow implementing a single scorer for a metric, eg:
RR@k
The text was updated successfully, but these errors were encountered: