SIGIR'98 posters: Lessons from BMIR-J2: A Test Collection for Japanese IR Systems
Lessons from BMIR-J2: A Test Collection for Japanese IR Systems
Tsuyoshi Kitani
Laboratory for Information Technology,
NTT Data Corporation,
Kawasaki 210-0913, Japan.
Yasushi Ogawa (Ricoh),
Tetsuya Ishikawa (ULIS),
Haruo Kimoto (NTT),
Ikuo Keshi (SHARP),
Jun Toyoura (Mitsubishi Electric),
Toshikazu Fukushima (NEC),
Kunio Matsui (Fujitsu Laboratories),
Yoshihiro Ueda (Fuji Xerox),
Tetsuya Sakai (Toshiba),
Takenobu Tokunaga (Tokyo Institute of Technology),
Hiroshi Tsuruoka (ERI, Univ. of Tokyo),
Hidekazu Nakawatase (NTT),
Teru Agata (Keio Univ.)
Abstract
BMIR-J2 is the first complete Japanese test collection available
for use in evaluating information retrieval systems. It contains sixty
queries and the IDs of 5080 newspaper articles in the fields of economics
and engineering. The queries are classified into five categories, based
on the functions the system is likely to use to interpret them
correctly and retrieve relevant texts. This collection has two
levels of relevance, topically relevant and partially relevant.
Also discussed are design issues such as collection types and size.
This collection and the principles derived in designing it should be
helpful in the future development of new test collections.
SIGIR'98
24-28 August 1998
Melbourne, Australia.
sigir98@cs.mu.oz.au.