Rank-Biased Precision for Measurement of Retrieval Effectiveness

Alistair Moffat
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Justin Zobel
School of Computer Science and Information Technology, RMIT University, Melbourne 3001
and
NICTA Victoria Research Laboratory, The University of Melbourne, Victoria 3010, Australia.

Status

ACM Trans. Information Systems, 27(1), paper 2, 1-27, December 2008.

Abstract

A range of methods for measuring the effectiveness of information retrieval systems have been proposed. These are typically intended to provide a quantitative single-value summary of a document ranking relative to a query. However, many of these measures have failings. For example, recall is not well founded as a measure of satisfaction, since the user of an actual system cannot judge recall. Average precision is derived from recall, and suffers from the same problem. In addition, average precision lacks key stability properties that are needed for robust experiments. In this paper we introduce a new effectiveness metric, rank-biased precision, that avoids these problems. Rank-biased precision is derived from a simple model of user behavior; is robust if answer rankings are extended to greater depths; and allows accurate quantification of experimental uncertainty, even when only partial relevance judgments are available.

Full text

http://doi.acm.org/10.1145/1416950.1416952

Software

William Webber has created a trec_eval-style implementation, that code is available as rbp_eval-0.2.tar.gz. (Usual disclaimers apply.)