Quit While Ahead: Evaluating Truncated Rankings
Fei Liu
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Alistair Moffat
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Timothy Baldwin
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Xiuzhen Jenny Zhang
School of Computer Science and Information Technology,
RMIT University,
Victoria 3001, Australia.
Status
Proc. 39th Ann. Int. ACM SIGIR Conf. on
Research and Development in Information Retrieval,
Pisa, Italy, July 2015, pages 953-956.
Abstract
Many types of search tasks are answered through the
computation of a ranked list of suggested answers.
We re-examine the usual assumption that answer lists should be as
long as possible, and suggest that when the number of matching items
is potentially small -- perhaps even zero -- it may be more helpful
to "quit while ahead", that is, to truncate the answer ranking
earlier rather than later.
To capture this effect, metrics are required which are attuned to the
length of the ranking, and can handle cases in which there are no
relevant documents.
In this work we explore a generalized approach for representing
truncated result sets, and propose modifications to a number of
popular evaluation metrics.
Full text
http://doi.acm.org/10.1145/2911451.2914737.