← smac.pub home

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

pdf bibtex 1 citation workshop paper to appear

Authors: Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, Luca Soldaini

Appearing in: The First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval (LLM4Eval@SIGIR 2024)

DBLP journals/corr/abs-2403-15246 arXiv 2403.15246 Google Scholar 7wWfoDgAAAAJ:fPk4N6BV_jEC smac.pub llm4eval2024-followir


Modern Large Language Models (LLMs) are capable of following long and complex instructions that enable a diverse amount of user tasks. However, despite Information Retrieval (IR) models using LLMs as the backbone of their architectures, nearly all of them still only take queries as input, with no instructions. For the handful of recent models that do take instructions, it's unclear how they use them. We introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions. FollowIR builds off the long history of the TREC conferences: as TREC provides human annotators with instructions (also known as narratives) to determine document relevance, so should IR models be able to understand and decide relevance based on these detailed instructions. Our evaluation benchmark starts with three deeply judged TREC collections and alters the annotator instructions, re-annotating relevant documents. Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework. Our results indicate that existing retrieval models fail to correctly use instructions, using them for basic keywords and struggling to understand long-form information. However, we show that it is possible for IR models to learn to follow complex instructions: our new FollowIR-7B model has significant improvements (over 13%) after fine-tuning on our training set.

BibTeX @inproceedings{weller:llm4eval2024-followir, author = {Weller, Orion and Chang, Benjamin and MacAvaney, Sean and Lo, Kyle and Cohan, Arman and Van Durme, Benjamin and Lawrie, Dawn and Soldaini, Luca}, title = {FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions}, booktitle = {The First Workshop on Large Language Models (LLMs) for Evaluation in Information Retrieval}, year = {2024}, url = {https://arxiv.org/abs/2403.15246} }