Towards Nuanced System Evaluation Based on Implicit User Expectations
Department of Computing and Information Systems,
The University of Melbourne,
Victoria 3010, Australia.
School of Computer Science and Information Technology,
Victoria 3001, Australia.
Proc. 11th Asian Information Retrieval Societies Conference,
Brisbane, December 2015, to appear.
Information retrieval systems are often evaluated through the use of
In the past, the metrics used have corresponded to fixed models of
user behavior, presuming, for example, that the user will view a
pre-determined number of items in the search engine results page, or
that they have a constant probability of advancing from one item in
the result page to the next.
Recently, a number of proposals for models of user behavior have
emerged that are parameterized in terms of the number of relevant
documents (or other material) a user expects to be required to
address their information need.
That recent work has demonstrated that T, the user's a priori
utility expectation, is correlated with the underlying nature of the
information need; and hence that evaluation metrics should be
sensitive to T.
Here we examine the relationship between the query the user issues,
and their anticipated T, seeking syntactic and other clues to guide
the subsequent system evaluation.
That is, we wish to develop mechanisms that, based on the query
alone, can be used to adjust system evaluations so that the
experience of the user of the system is better captured in the
system's effectiveness score, and hence can be used as a more
refined way of comparing systems.
This paper reports on a first round of experimentation, and describes
the progress (albeit modest) that we have achieved towards that goal.