← smac.pub home

A ‘Pointwise-Query, Listwise-Document’ based QPP Approach

link bibtex code 22 citations short conference paper

Authors: Suchana Datta, Sean MacAvaney, Debasis Ganguly, Derek Greene

Appeared in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022)

Links/IDs:
DOI 10.1145/3477495.3531821 DBLP conf/sigir/DattaMGG22 ACM 3477495.3531821 Google Scholar 7wWfoDgAAAAJ:r0BpntZqJG4C Semantic Scholar caea6b65a85d5a1eb7b8ad44a2d43664be7c7172 Enlighten 268513 smac.pub sigir2022-qpp

Abstract:

The task of Query Performance Prediction (QPP) in IR involved pre- dicting the relative effectiveness of a search system for a given input query. Supervised approaches for QPP, such as NeuralQPP are often trained on pairs of queries in order to capture their relative retrieval performance. However, pointwise approaches, such as the recently proposed BERT-QPP [1], are generally preferable for efficiency reasons. In this paper, we propose a novel end-to-end neural cross-encoder-based approach that is trained pointwise on individual queries, but listwise over the top-k retrieved set of docu- ments (split into chunks). In contrast to prior work, the network is then trained to predict the number of relevant documents in each chunk for a given query. These are then aggregated into a query per- formance prediction, rather than predicting performance directly. Our experiments demonstrate that training a network to predict the number of relevant documents in the top-k list turns out to be significantly more effective than a cross-encoder network which predicts a target retrieval effectiveness measure, such as BERT-QPP. Results show that our proposed approach is able to substantially improve QPP effectiveness by up to 30% on the TREC-DL’20 dataset, and by nearly 9% for the MS MARCO Dev set over BERT-QPP, a state-of-the-art supervised QPP model.

BibTeX @inproceedings{datta:sigir2022-qpp, author = {Datta, Suchana and MacAvaney, Sean and Ganguly, Debasis and Greene, Derek}, title = {A ‘Pointwise-Query, Listwise-Document’ based QPP Approach}, booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval}, year = {2022}, doi = {10.1145/3477495.3531821} }