link bibtex code 21 citations short conference paper
Appeared in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2022)
Abstract:
The task of Query Performance Prediction (QPP) in IR involved pre- dicting the relative effectiveness of a search system for a given input query. Supervised approaches for QPP, such as NeuralQPP are often trained on pairs of queries in order to capture their relative retrieval performance. However, pointwise approaches, such as the recently proposed BERT-QPP [1], are generally preferable for efficiency reasons. In this paper, we propose a novel end-to-end neural cross-encoder-based approach that is trained pointwise on individual queries, but listwise over the top-k retrieved set of docu- ments (split into chunks). In contrast to prior work, the network is then trained to predict the number of relevant documents in each chunk for a given query. These are then aggregated into a query per- formance prediction, rather than predicting performance directly. Our experiments demonstrate that training a network to predict the number of relevant documents in the top-k list turns out to be significantly more effective than a cross-encoder network which predicts a target retrieval effectiveness measure, such as BERT-QPP. Results show that our proposed approach is able to substantially improve QPP effectiveness by up to 30% on the TREC-DL’20 dataset, and by nearly 9% for the MS MARCO Dev set over BERT-QPP, a state-of-the-art supervised QPP model.
BibTeX @inproceedings{datta:sigir2022-qpp, author = {Datta, Suchana and MacAvaney, Sean and Ganguly, Debasis and Greene, Derek}, title = {A ‘Pointwise-Query, Listwise-Document’ based QPP Approach}, booktitle = {Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval}, year = {2022}, doi = {10.1145/3477495.3531821} }