pdf bibtex code 1 citation short conference paper
Appeared in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023)
Abstract:
One advantage of neural language models is that they are meant to generalise well in situations of synonymity i.e. where two words have similar or identical meanings. In this paper, we investigate and quantify how well various ranking models perform in a clear-cut case of synonymity: when words are simply expressed in different surface forms due to regional differences in spelling conventions (e.g., color vs colour). We explore the prevalence of American and British English spelling conventions in datasets used for the pre-training, training and evaluation of neural retrieval methods, and find that American spelling conventions are far more prevalent. Despite these biases in the training data, we find that dense retrievers are in general robust in this case of synonymity generalisation while lexical and neural re-ranking models exhibit higher variability, making it challenging to characterise their behaviour. We also explore document normalisation as a possible mitigation strategy and observe that all models are affected by document normalisation. While they all experience a drop in performance when normalised to a different spelling convention than that of the query, we observe varied behaviour when the document is normalised to share the query spelling convention with the lexical models showing improvements, while the dense retrievers remain unaffected and the neural re-rankers exhibit contradictory behaviour
BibTeX @inproceedings{chari:sigir2023-spelling, author = {Chari, Andreas and MacAvaney, Sean and Ounis, Iadh}, title = {On the Effects of Regional Spelling Conventions in Retrieval Models}, booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval}, year = {2023}, url = {https://arxiv.org/abs/2308.00480}, doi = {10.1145/3539618.3592030} }