Parameter Sensitivity in Rank-Biased Precision
Yuye Zhang
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Laurence A. F. Park
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Alistair Moffat
Department of Computer Science and Software Engineering,
The University of Melbourne,
Victoria 3010, Australia.
Status
Proc. 13th Australasian Document Computing Symposium,
Hobart, Australia, December 2008, pages 61-68.
Abstract
Rank-Biased Precision (RBP) is a retrieval evaluation metric that
assigns an effectiveness score to a ranking by computing a
geometricly weighted sum of document relevance values, with the
monotonicly decreasing weights in the geometric distribution
determined via a persistence parameter p.
Despite exhibiting various advantageous traits over well known
existing measures such as Average Precision, RBP has the drawback of
requiring the designer of any experiment to choose a value for p.
Here we present a method that allows retrieval systems evaluated
using RBP with different p values to be compared.
The proposed approach involves calculating two critical bounding
relevance vectors for the original RBP score, and using those vectors
to calculate the range of possible RBP scores for any other value of
p.
Those bounds may then be sufficient to allow the outright superiority
of one system over the other to be established.
In addition, the process can be modified to handle any RBP residuals
associated with either of the two systems.
We believe the adoption of the comparison process described in this
paper will greatly aid the uptake of RBP in evaluation experiments.