Judgment Pool Effects Caused by Query Variations

Alistair Moffat
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.


Proc. 21st Australasian Document Computing Symp., Caulfield, Australia, December 2016, pages 65-68.


Batch-mode retrieval evaluation relies on suitable relevance judgments being available. Here we explore the implications on pool size of adopting a "query variations" approach to collection construction. Using the resources provided as part of the UQV100 collection [Bailey et al., SIGIR 2016] and a total of five different systems, we show that pool size is as much affected by the number of query variations involved as it is by the number of contributing systems, and that systems and users are independent effects. That is, if both system and query variation are to be accommodated in retrieval experimentation, the cost of performing the required judgments compounds.

Full text