How Precise Does Document Scoring Need To Be?

Ziying Yang
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Alistair Moffat
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Andrew Turpin
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Status

Proc. 12th Asian Information Retrieval Societies Conference, Beijing, December 2016, pages 279-291, LNCS volume 9994.

Abstract

We explore the implications of tied scores arising in the document similarity scoring regimes that are used when queries are processed in a retrieval engine. Our investigation has two parts: first, we evaluate past TREC runs to determine the prevalence and impact of tied scores, to understand the alternative treatments that might be used to handle them; and second, we explore the implications of what might be thought of as "deliberate" tied scores, in order to allow for faster search. In the first part of our investigation we show that while tied scores had the potential to be disruptive to TREC evaluations, in practice their effect was relatively minor. The second part of our exploration helps understand why that was so, and shows that quite marked levels of score rounding can be tolerated, without greatly affecting the ability to compare between systems. The latter finding offers the potential for approximate scoring regimes that provide faster query processing with little or no loss of effectiveness.

Full text

http://dx.doi.org/10.1007/978-3-319-48051-0_21 .