Term Impacts as Normalized Term Frequencies for BM25 Similarity Scoring

Vo Ngoc Anh
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Raymond Wan
Bioinformatics Center, Kyoto University, Gokasho, Uji, Kyoto 611-0011, Japan.

Alistair Moffat
Department of Computer Science and Software Engineering, The University of Melbourne, Victoria 3010, Australia.

Status

Proc. 15th Int. Symp. String Processing and Information Retrieval, Melbourne, Australia, November 2008, pages 51-62. LNCS volume 5280.

Abstract

The BM25 similarity computation has been shown to provide effective document retrieval. In operational terms, the formulae which form the basis for BM25 employ both term frequency and document length normalization. This paper considers an alternative form of normalization using document-centric impacts, and shows that the new normalization simplifies BM25 and reduces the number of tuning parameters. Motivation is provided by a preliminary analysis of a document collection that shows that impacts are more likely to identify documents whose lengths resemble those of the relevant judgments. Experiments on TREC data demonstrate that impact-based BM25 is as good as or better than the original term frequency-based BM25 in terms of retrieval effectiveness.

Full text

http://dx.doi.org/10.1007/978-3-540-89097-3_7