Judgment Pool Effects Caused by Query Variations
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Proc. 21st Australasian Document Computing Symp., Caulfield, Australia,
December 2016, pages 65-68.
Batch-mode retrieval evaluation relies on suitable relevance
judgments being available.
Here we explore the implications on pool size of adopting a "query
variations" approach to collection construction.
Using the resources provided as part of the UQV100 collection [Bailey
et al., SIGIR 2016] and a total of five different systems, we show
that pool size is as much affected by the number of query variations
involved as it is by the number of contributing systems, and that
systems and users are independent effects.
That is, if both system and query variation are to be accommodated in
retrieval experimentation, the cost of performing the required