SIGIR'98 papers: Measures of Relative Relevance and Ranked Half-Life:
Performance Indicators for Interactive IR
Measures of Relative Relevance and Ranked Half-Life:
Performance Indicators for Interactive IR
Pia Borlund
Department of IR Theory,
Royal School of Library and Information Science,
DK-2300 Copenhagen S., Denmark.
Peter Ingwersen
Department of IR Theory,
Royal School of Library and Information Science,
DK-2300 Copenhagen S., Denmark.
Abstract
This paper introduces the concepts of the relative relevance (RR)
measure and a new performance indicator of the positional strength of
the retrieved and ranked documents. The former is seen as a measure of
associative performance computed by the application of the Jaccard
formula. The latter is named the Ranked Half-Life (RHL) indicator and
denotes the degree to which relevant documents are located on the top of
a ranked retrieval result. The measures are proposed to be applied in
addition to the traditional performance parameters such as precision
and/or recall in connection with evaluation of interactive IR systems.
The RR measure describes the degree of agreement between the types of
relevance applied in evaluation of information retrieval (IR) systems in
a non-binary assessment context. It is shown that the measure has
potential to bridge the gap between subjective and objective relevance,
as it makes it possible to understand and interpret the relation between
these two main classes of relevance used in interactive IR experiments.
The relevance concepts are defined, and the application of the measures
is demonstrated by interrelating three types of relevance assessments:
algorithmic; intellectual topicality and; situational assessments.
Further, the paper shows that for a given set of queries at given
precision levels the RHL indicator adds to the understanding of
comparisons of IR performance.
SIGIR'98
24-28 August 1998
Melbourne, Australia.
sigir98@cs.mu.oz.au.