Modeling Relevance as a Function of Retrieval Rank
Xiaolu Lu
School of Computer Science and Information Technology,
RMIT University,
Victoria 3001, Australia.
Alistair Moffat
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Shane Culpepper
School of Computer Science and Information Technology,
RMIT University,
Victoria 3001, Australia.
Status
Proc. 12th Asian Information Retrieval Societies Conference,
Beijing, December 2016, pages 3-15, LNCS volume 9994.
Abstract
Batched evaluations in IR experiments are commonly built using
relevance judgments formed over a sampled pool of documents.
However, judgment coverage tends to be incomplete relative to the
metrics being used to compute effectiveness, since collection size
often makes it financially impractical to judge every document.
%the problem is old
As a result, a considerable body of work has arisen exploring the
question of how to fairly compare systems in the face of unjudged
documents.
Here we consider the same problem from another perspective, and
investigate the relationship between relevance likelihood and
retrieval rank, seeking to identify plausible methods for estimating
document relevance and hence computing an inferred gain.
A range of models are fitted against two typical TREC datasets, and
evaluated both in terms of their goodness of fit relative to the full
set of known relevance judgments, and also in terms of their
predictive ability when shallower initial pools are presumed, and
extrapolated metric scores are computed based on models developed
from those shallow pools.
Full text
http://dx.doi.org/10.1007/978-3-319-48051-0_1
.