Task and Query Difficulty in Performance Prediction

Paul Thomas
Microsoft Bing, Canberra Australia

Falk Scholer
School of Computer Science and Information Technology, RMIT University, Victoria 3001, Australia.

Peter Bailey
Microsoft Bing, Canberra, Australia.

Alistair Moffat
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Status

Proc. 22nd Australasian Document Computing Symp., Brisbane, December 2017, pages 11:1-11:4.

Abstract

Prediction evaluations have not distinguished effects due to query wording from effects due to the underlying information need, nor from effects due to performance of the retrieval system itself. Here we use five rankers, 100 tasks, and 28,869 queries to distinguish these three effects over six pre-retrieval predictors. We see that task effects dominate those due to query or ranker; that many "query performance predictors" are in fact predicting task difficulty; and that this makes it difficult to use these algorithms.

Full text

http://doi.org/10.1145/3166072.3166079 .