← smac.pub home

Reproducing Personalised Session Search over the AOL Query Log

pdf bibtex code 10 citations reproducibility paper

Authors: Sean MacAvaney, Craig Macdonald, Iadh Ounis

Appeared in: Proceedings of the 44th European Conference on Information Retrieval Research (ECIR 2022)

DOI 10.1007/978-3-030-99736-6_42 DBLP conf/ecir/MacAvaneyMO22 arXiv 2201.08622 Google Scholar 7wWfoDgAAAAJ:-f6ydRqryjwC Semantic Scholar 7fdc00b9c0515788aaa40d886cff4f17b3d9a389 Enlighten 259616 smac.pub ecir2022-aolia


Despite its troubled past, the AOL Query Log continues to be an important resource to the research community---particularly for tasks like search personalisation. When using the query log these ranking experiments, the document corpus is often an afterthought. Recent work typically uses a corpus containing versions of the documents collected long after the log was produced. Given that web documents are prone to change over time, we study the differences present between a version of the corpus containing documents as they appeared in 2017 (which has been used by several recent works) and a new version we construct that includes documents close to as they appeared at the time the query log was produced (2006). We demonstrate that this new version of the corpus has a far higher coverage of documents present in the original log (93%) than the 2017 version (55%). Among the overlapping documents, the content often differs substantially. Given these differences, we re-conduct session search experiments that originally used the 2017 corpus and find that when using our corpus for training or evaluation, system performance improves. We place the results in context by introducing recent adhoc ranking baselines. We also confirm the navigational nature of the queries in the AOL corpus by showing that including the URL substantially improves performance across a variety of models. Our version of the corpus can be easily reconstructed by other researchers and is included in the ir-datasets package.

BibTeX @inproceedings{macavaney:ecir2022-aolia, author = {MacAvaney, Sean and Macdonald, Craig and Ounis, Iadh}, title = {Reproducing Personalised Session Search over the AOL Query Log}, booktitle = {Proceedings of the 44th European Conference on Information Retrieval Research}, year = {2022}, url = {https://arxiv.org/abs/2201.08622}, doi = {10.1007/978-3-030-99736-6_42} }