Against Recall: Is It Persistence, Cardinality, Density, Coverage, or Totality?

Justin Zobel
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Alistair Moffat
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Laurence A. F. Park
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Status

SIGIR Forum, 43(1):3-15, June 2009.

Abstract

The concept of recall has been one of the key elements of system measurement throughout the history of information retrieval, despite the fact that there are many unanswered questions as to its value. In this essay, we review those questions and explore several further issues that affect the usefulness of recall. In particular, we ask whether it is reasonable to expect to be able to measure recall; whether some researchers are conflating the concepts of recall and answer set cardinality; and whether it is plausible that a user would rely on a belief that a system is ``high recall'' to deeply explore an answer list. Combined with earlier observations about the unknowability of recall, and the lack of a plausible user model in which recall is a measure of satisfaction, we conclude that use of recall as a measure of the effectiveness of ranked querying is indefensible.

Full text

http://www.sigir.org/forum/2009J/2009j-sigirforum-zobel.pdf