Assessing the Cognitive Complexity of Information Needs

Alistair Moffat
Department of Computing and Information Systems, The University of Melbourne, Victoria 3010, Australia.

Peter Bailey
Microsoft Research, Canberra, Australia.

Falk Scholer
School of Computer Science and Information Technology, RMIT University, Victoria 3001, Australia.

Paul Thomas
CSIRO, Canberra Australia

Status

Proc. 19th Australasian Document Computing Symp., Melbourne, December 2014, pages 97-100.

Abstract

Information retrieval systems can be evaluated in laboratory settings through the use of user studies, and through the use of test collections and effectiveness metrics. In a larger investigation we are exploring the extent to which individual user differences and behaviours can affect the scores generated by a retrieval system.

Our objective in the first phase of that project is to define information need statements corresponding to a range of TREC search tasks, and to categorise those statements in terms of task complexity. The goal is to reach a position from which we can determine whether user actions while searching are influenced by the way the information need is expressed, and by the fundamental nature of the information need. We describe the process used to create information need statements, and then report inter- and intra-assessor agreements across four annotators. We conclude that assessing the relative cognitive complexity of tasks is a complex activity, even for experienced annotators.

Full text

http://dx.doi.org/10.1145/2682862.2682874 .

Data Resource

http://dx.doi.org/10.4225/08/55D0B6A098248 .