TY  - JOUR
A1  - Meijering, Jurian V.
A1  - Kern, Kristine
A1  - Tobi, Hilde
T1  - Identifying the methodological characteristics of European green city rankings
JF  - Ecological indicators : integrating monitoring, assessment and management
N2  - City rankings that aim to measure the environmental sustainability of European cities may contribute to the evaluation and development of environmental policy of European cities. The objective of this study is to identify and evaluate the methodological characteristics of these city rankings. First, a methodology was developed to systematically identify methodological characteristics of city rankings within different steps of the ranking development process. Second, six city rankings (European Energy Award, European Green Capital Award, European Green City Index, European Soot-free City Ranking, RES Champions League, Urban Ecosystem Europe) were examined. Official websites and any methodological documents found on those websites were content analyzed using the developed methodology. Interviews with representatives of the city rankings were conducted to acquire any additional information. Results showed that the city rankings varied greatly with respect to their methodological characteristics and that all city rankings had methodological weaknesses. Developers of city rankings are advised to use the methodology developed in this study to find methodological weaknesses and improve their ranking. In addition, developers ought to be more transparent about the methodological characteristics of their city rankings. End-users of city rankings are advised to use the developed methodology to identify and evaluate the methodological characteristics of city rankings before deciding to act on ranking results. (c) 2014 Elsevier Ltd. All rights reserved.
KW  - Methodology
KW  - Ranking
KW  - Index
KW  - Indicator
KW  - City
KW  - Environmental sustainability
Y1  - 2014
U6  - https://doi.org/10.1016/j.ecolind.2014.02.026
SN  - 1470-160X
SN  - 1872-7034
VL  - 43
SP  - 132
EP  - 142
PB  - Elsevier
CY  - Amsterdam
ER  - 
TY  - JOUR
A1  - Sawade, Christoph
A1  - Bickel, Steffen
A1  - von Oertzen, Timo
A1  - Scheffer, Tobias
A1  - Landwehr, Niels
T1  - Active evaluation of ranking functions based on graded relevance
JF  - Machine learning
N2  - Evaluating the quality of ranking functions is a core task in web search and other information retrieval domains. Because query distributions and item relevance change over time, ranking models often cannot be evaluated accurately on held-out training data. Instead, considerable effort is spent on manually labeling the relevance of query results for test queries in order to track ranking performance. We address the problem of estimating ranking performance as accurately as possible on a fixed labeling budget. Estimates are based on a set of most informative test queries selected by an active sampling distribution. Query labeling costs depend on the number of result items as well as item-specific attributes such as document length. We derive cost-optimal sampling distributions for the commonly used performance measures Discounted Cumulative Gain and Expected Reciprocal Rank. Experiments on web search engine data illustrate significant reductions in labeling costs.
KW  - Information retrieval
KW  - Ranking
KW  - Active evaluation
Y1  - 2013
U6  - https://doi.org/10.1007/s10994-013-5372-5
SN  - 0885-6125
VL  - 92
IS  - 1
SP  - 41
EP  - 64
PB  - Springer
CY  - Dordrecht
ER  -