Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.
School of Computer Science and Information Technology, RMIT University, Victoria 3001, Australia.
CSIRO and Australian National University, Canberra Australia
Conventionally, pooling uses system variations to provide diverse documents to be judged for a topic; different user queries are not considered. We explore the ramifications of user query variability on pooling, and demonstrate that conventional test collections do not cover this source of variation. The effect of user query variation on the size of the judging pool is just as strong as the effect of retrieval system variation. We conclude that user query variation should be incorporated early in test collection construction, and cannot be considered effectively post hoc.