Quit While Ahead: Evaluating Truncated Rankings


Fei Liu
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Alistair Moffat
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Timothy Baldwin
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Xiuzhen Jenny Zhang
School of Computer Science and Information Technology, RMIT University, Victoria 3001, Australia.


Status

Proc. 39th Ann. Int. ACM SIGIR Conf. on Research and Development in Information Retrieval, Pisa, Italy, July 2015, pages 953-956.

Abstract

Many types of search tasks are answered through the computation of a ranked list of suggested answers. We re-examine the usual assumption that answer lists should be as long as possible, and suggest that when the number of matching items is potentially small -- perhaps even zero -- it may be more helpful to "quit while ahead", that is, to truncate the answer ranking earlier rather than later. To capture this effect, metrics are required which are attuned to the length of the ranking, and can handle cases in which there are no relevant documents. In this work we explore a generalized approach for representing truncated result sets, and propose modifications to a number of popular evaluation metrics.

Full text

http://doi.acm.org/10.1145/2911451.2914737.