Task and Query Difficulty in Performance Prediction
Paul Thomas
Microsoft Bing,
Canberra
Australia
Falk Scholer
School of Computer Science and Information Technology,
RMIT University,
Victoria 3001, Australia.
Peter Bailey
Microsoft Bing, Canberra, Australia.
Alistair Moffat
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Status
Proc. 22nd Australasian Document Computing Symp., Brisbane,
December 2017, pages 11:1-11:4.
Abstract
Prediction evaluations have not distinguished effects due to query
wording from effects due to the underlying information need, nor from
effects due to performance of the retrieval system itself.
Here we use five rankers, 100 tasks, and 28,869 queries to
distinguish these three effects over six pre-retrieval predictors.
We see that task effects dominate those due to query or ranker; that
many "query performance predictors" are in fact predicting task
difficulty; and that this makes it difficult to use these algorithms.
Full text
http://doi.org/10.1145/3166072.3166079
.