SuiteEval: Simplifying Retrieval Benchmarks

pdf bibtex demonstration paper

Authors: Andrew Parry, Debasis Ganguly, Sean MacAvaney

Appeared in: 48th European Conference on Information Retrieval (ECIR 2026)

Abstract:

Information retrieval evaluation often suffers from fragmented practices -- varying dataset subsets, aggregation methods, and pipeline configurations -- that undermine reproducibility and comparability, especially for foundation embedding models requiring robust out-of-domain performance. We introduce SuiteEval, a unified framework that offers automatic end-to-end evaluation, dynamic indexing that reuses on-disk indices to minimise disk usage, and built-in support for major benchmarks (BEIR, LoTTE, MS MARCO, NanoBEIR, and BRIGHT). Users only need to supply a pipeline generator. SuiteEval handles data loading, indexing, ranking, metric computation, and result aggregation. New benchmark suites can be added in a single line. SuiteEval reduces boilerplate and standardises evaluations to facilitate reproducible IR research, as a broader benchmark set is increasingly required.

BibTeX @inproceedings{parry:ecir2026-suiteeval, author = {Parry, Andrew and Ganguly, Debasis and MacAvaney, Sean}, title = {SuiteEval: Simplifying Retrieval Benchmarks}, booktitle = {48th European Conference on Information Retrieval}, year = {2026}, url = {https://arxiv.org/abs/2602.18107}, doi = {10.1007/978-3-032-21321-1_27}, pages = {187--192} }