Against Recall: Is It Persistence, Cardinality, Density, Coverage, or Totality?
Justin Zobel
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Alistair Moffat
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Laurence A. F. Park
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Status
SIGIR Forum, 43(1):3-15, June 2009.
Abstract
The concept of recall has been one of the key elements of system
measurement throughout the history of information retrieval, despite
the fact that there are many unanswered questions as to its value.
In this essay, we review those questions and explore several further
issues that affect the usefulness of recall.
In particular, we ask whether it is reasonable to expect to be able
to measure recall; whether some researchers are conflating the
concepts of recall and answer set cardinality; and whether it
is plausible that a user would rely on a belief that a system is
``high recall'' to deeply explore an answer list.
Combined with earlier observations about the unknowability of recall,
and the lack of a plausible user model in which recall is a measure
of satisfaction, we conclude that use of recall as a measure of the
effectiveness of ranked querying is indefensible.
Full text
http://www.sigir.org/forum/2009J/2009j-sigirforum-zobel.pdf