The search result changed since you submitted your search request. Documents might be displayed in a different sort order.
  • search hit 72 of 669
Back to Result List

Active evaluation of ranking functions based on graded relevance

  • Evaluating the quality of ranking functions is a core task in web search and other information retrieval domains. Because query distributions and item relevance change over time, ranking models often cannot be evaluated accurately on held-out training data. Instead, considerable effort is spent on manually labeling the relevance of query results for test queries in order to track ranking performance. We address the problem of estimating ranking performance as accurately as possible on a fixed labeling budget. Estimates are based on a set of most informative test queries selected by an active sampling distribution. Query labeling costs depend on the number of result items as well as item-specific attributes such as document length. We derive cost-optimal sampling distributions for the commonly used performance measures Discounted Cumulative Gain and Expected Reciprocal Rank. Experiments on web search engine data illustrate significant reductions in labeling costs.

Export metadata

Additional Services

Search Google Scholar Statistics
Metadaten
Author details:Christoph Sawade, Steffen Bickel, Timo von Oertzen, Tobias SchefferORCiD, Niels LandwehrORCiDGND
DOI:https://doi.org/10.1007/s10994-013-5372-5
ISSN:0885-6125
Title of parent work (English):Machine learning
Publisher:Springer
Place of publishing:Dordrecht
Publication type:Article
Language:English
Year of first publication:2013
Publication year:2013
Release date:2017/03/26
Tag:Active evaluation; Information retrieval; Ranking
Volume:92
Issue:1
Number of pages:24
First page:41
Last Page:64
Organizational units:Mathematisch-Naturwissenschaftliche Fakultät / Institut für Informatik und Computational Science
Peer review:Referiert
Institution name at the time of the publication:Mathematisch-Naturwissenschaftliche Fakultät / Institut für Informatik
Accept ✔
This website uses technically necessary session cookies. By continuing to use the website, you agree to this. You can find our privacy policy here.