Towards Nuanced System Evaluation Based on Implicit User Expectations
Paul Thomas
CSIRO,
Canberra
Australia
Peter Bailey
Microsoft, Australia
Alistair Moffat
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
Falk Scholer
School of Computer Science and Information Technology,
RMIT University,
Victoria 3001, Australia.
Status
Proc. 11th Asian Information Retrieval Societies Conference,
Brisbane, December 2015, to appear.
Abstract
Information retrieval systems are often evaluated through the use of
effectiveness metrics.
In the past, the metrics used have corresponded to fixed models of
user behavior, presuming, for example, that the user will view a
pre-determined number of items in the search engine results page, or
that they have a constant probability of advancing from one item in
the result page to the next.
Recently, a number of proposals for models of user behavior have
emerged that are parameterized in terms of the number of relevant
documents (or other material) a user expects to be required to
address their information need.
That recent work has demonstrated that T, the user's a priori
utility expectation, is correlated with the underlying nature of the
information need; and hence that evaluation metrics should be
sensitive to T.
Here we examine the relationship between the query the user issues,
and their anticipated T, seeking syntactic and other clues to guide
the subsequent system evaluation.
That is, we wish to develop mechanisms that, based on the query
alone, can be used to adjust system evaluations so that the
experience of the user of the system is better captured in the
system's effectiveness score, and hence can be used as a more
refined way of comparing systems.
This paper reports on a first round of experimentation, and describes
the progress (albeit modest) that we have achieved towards that goal.
Full text
http://dx.doi.org/10.1007/978-3-319-28940-3_26
.
Data Resource
http://dx.doi.org/10.4225/08/55D0B6A098248
.