TY  - JOUR
A1  - Sawade, Christoph
A1  - Bickel, Steffen
A1  - von Oertzen, Timo
A1  - Scheffer, Tobias
A1  - Landwehr, Niels
T1  - Active evaluation of ranking functions based on graded relevance
JF  - Machine learning
N2  - Evaluating the quality of ranking functions is a core task in web search and other information retrieval domains. Because query distributions and item relevance change over time, ranking models often cannot be evaluated accurately on held-out training data. Instead, considerable effort is spent on manually labeling the relevance of query results for test queries in order to track ranking performance. We address the problem of estimating ranking performance as accurately as possible on a fixed labeling budget. Estimates are based on a set of most informative test queries selected by an active sampling distribution. Query labeling costs depend on the number of result items as well as item-specific attributes such as document length. We derive cost-optimal sampling distributions for the commonly used performance measures Discounted Cumulative Gain and Expected Reciprocal Rank. Experiments on web search engine data illustrate significant reductions in labeling costs.
KW  - Information retrieval
KW  - Ranking
KW  - Active evaluation
Y1  - 2013
U6  - https://doi.org/10.1007/s10994-013-5372-5
SN  - 0885-6125
VL  - 92
IS  - 1
SP  - 41
EP  - 64
PB  - Springer
CY  - Dordrecht
ER  - 
TY  - JOUR
A1  - Bickel, Steffen
A1  - Brückner, Michael
A1  - Scheffer, Tobias
T1  - Discriminative learning under covariate shift
N2  - We address classification problems for which the training instances are governed by an input distribution that is allowed to differ arbitrarily from the test distribution-problems also referred to as classification under covariate shift. We derive a solution that is purely discriminative: neither training nor test distribution are modeled explicitly. The problem of learning under covariate shift can be written as an integrated optimization problem. Instantiating the general optimization problem leads to a kernel logistic regression and an exponential model classifier for covariate shift. The optimization problem is convex under certain conditions; our findings also clarify the relationship to the known kernel mean matching procedure. We report on experiments on problems of spam filtering, text classification, and landmine detection.
Y1  - 2009
UR  - http://jmlr.csail.mit.edu/
SN  - 1532-4435
ER  - 
TY  - JOUR
A1  - Bickel, Steffen
A1  - Brueckner, Michael
A1  - Scheffer, Tobias
T1  - Discriminative learning under covariate shift
N2  - We address classification problems for which the training instances are governed by an input distribution that is allowed to differ arbitrarily from the test distribution-problems also referred to as classification under covariate shift. We derive a solution that is purely discriminative: neither training nor test distribution are modeled explicitly. The problem of learning under covariate shift can be written as an integrated optimization problem. Instantiating the general optimization problem leads to a kernel logistic regression and an exponential model classifier for covariate shift. The optimization problem is convex under certain conditions; our findings also clarify the relationship to the known kernel mean matching procedure. We report on experiments on problems of spam filtering, text classification, and landmine detection.
Y1  - 2009
UR  - http://jmlr.csail.mit.edu/
SN  - 1532-4435
ER  -