Search for: All records

Transformer-based rankers have shown the state-of-the-art performance, but their self-attention operation is mostly unable to process long sequences. One of the common approaches to train these rankers is to heuristically select some segments of each document, such as the first segment, as training data. However, these segments may not contain the query-related parts of documents. To address this problem, we propose the query-driven segment selection from long documents to build training data for transformer-based rankers. The segment selector provides relevant samples with more accurate labels and non-relevant samples which are harder to be predicted. The experimental results show that the basic BERT-based ranker trained with the proposed segment selector significantly outperforms that trained by the heuristically selected segments, and performs equally to the state-of-the-art model with localized self-attention that can process longer input sequences. We also demonstrate that training with our segment selector, there is not much gain from feeding input sequences larger than 200 words. Our findings open up new opportunities to design efficient transformer-based rankers.