Despite their success, Transformer LMs have pre-set input length limits, typically 512, due to complexity considerations [4,15,23]. As a result, re-ranking long documents beyond this length has been a challenging problem. Prior research designed techniques to accommodate the neural LM. For example, some attempts to strategically limit the input length, which however prohibits full interaction among all query and document tokens. Others have turned to more efficient Transformers that use lower complexity but also lower capacity sparse self-attention. In this paper, we present an alternative method, re-ranking documents with a modular reranker and a new form of attention operation, query to document cross attention.

Modular re-rankers such as MORES [7] provide an alternative framework to common re-rankers. With MORES, query and document are independently encoded. A Transformer Interaction Module then makes relevance estimates using light-weight query-todocument attention. In this paper, we introduce a modification to this framework, MORES+, for long document re-ranking. We break a candidate document into passage-like chunks and encode them separately using identical Transformer encoders. A joint query-toall-chunk attention then models interaction between query and the full document. The model can flexibly control the attention weights to pick up important information. This design roughly resembles the human reading comprehension process: after browsing the article and the question, the reader will refer back to the relevant pieces in the original article to come up with answers.

We present experiments with classical few-shot document IR datasets, ClueWeb09 and Robust04, as well as a recent large-scale dataset, MS-MARCO document ranking. We found that the MORES+ re-ranker produces better ranking quality on detailed natural language queries where interaction between query and document is more sophisticated.

The rest of the paper is organized as follows. Section 2 provides a background on document re-ranking methods with Transformers. Section 3 describes the MORES modular re-ranker and how we generalize it to MORES+ for document re-ranking. Section 4 presents experimental results with MORES+.

The revolution in pre-trained Transformer language models [4,17] had a large impact on IR [3]. Encoder-only LMs like BERT [4] are the current state-of-the-art for text re-ranking. They can a) generate contextualized representations [17] to help enhance the language understanding capability of the re-rankers and b) perform deep interactions between query and document [3]. Other forms of pretrained LMs also exist. Encoder-decoder LMs are pre-trained on sequence-to-sequence tasks and have previously been used in IR for tasks like document expansion [12,18].

Despite their advantages, Transformer LM re-rankers cannot be naively used for long document ranking since most pre-trained LMs have a maximum input length constraint of 512. Prior research circumvented this by splitting documents into passage-like short text chunks. The passages are first separately judged and passage-level results are further pooled to generate the final prediction. Scorepooling systems simply take max/sum/first of the chunk scores [3]. These systems are typically named after their underlying LM and pooling method. For example, BERT-maxp [3] refers to a BERTbased system with max pooling. A generalization is representation aggregation systems like PARADE [13]. They take the low dimension CLS vector representations from each passage and learn an additional aggregation model to combine these representations. The aggregation model can any neural model, e.g. a Transformer in PARADE-Transformer [13]. These pooling systems make it possible to apply deep LMs to document re-ranking. However, they prohibit direct query and full document interaction. The interactions always happen at the passage level and the passage scores or low dimension representations are bottlenecks in the model.

Alternatively, some research has explored the use of long-text Transformers [2] that use sparse attention patterns [10]. Sparse attention comes with the cost of lower capacity than vanilla attention [2].

A third class of approaches tries to first select the important short chunk of document, the "key block", and feed it into a vanilla Transformer [11,14]. These models' success depends on a) the quality of the selected block, and more importantly b) the assumption that a smaller key block exists to represent the full long document.

In this section, we first give some preliminaries on deep re-rankers based on the Transformer LM. We then briefly discuss the MORES modular re-ranker and how we generalize it for document reranking.

Recent state-of-the-art systems in text re-ranking are based on encoder-only pre-trained Transformer LMs. They cast re-ranking as a sequence pair relation prediction task. Query, document, and a special CLS token are concatenated and fed into the Transformer LM. Query and document go through a series of Transformer layers together, interact, and produce contextualized representations 𝐻 . 𝐻 = LM(concatenate(CLS; query; document))

A final score is computed as a dot product between the representation of the CLS vector 𝐻 𝑐𝑙𝑠 and a projection vector v proj .

Effectively, the Transformer model here serves as a deep non-convex similarity function between query and document. These models, however, have two disadvantages: 1) the aforementioned LM input length constraint, which is typically 512, and, 2) query and document are entangled in a black-box fashion. The former means these models do not natively support long documents. The latter makes it hard to design an easy fix. In this paper, we instead turn to modular re-rankers [7].

The Modular Transformer-based re-ranker, or in short MORES, was introduced by Gao et al. [7]. In MORES, query (prepended with a CLS) and candidate documents are first separately encoded by two Transformer encoders.

𝐻 q = Encoder qry (query)

These representations 𝐻 q and 𝐻 d remain independent until they are fed into an Interaction Module (IM).

The IM is a third Transformer that performs self-attention over the query representation 𝐻 q . In addition, query-to-document cross attention is performed in each layer. Given an intermediate query representation representation 𝑞, the cross-attention generates 𝑞 ′ .

This operation interacts query and document and is of complexity linear to query/document length. The final CLS vector is projected to generate a relevance prediction score.

The flexible design of MORES allows us to further modify the modules to support long documents. Given a long document that does not fit into a typical LM re-ranker, we first break it into chunks, document → {𝑐 1 , .., 𝑐 𝑛 }, and encode each chunk independently.

We modify the Interaction Module to take all or the chunks' representations at once,

where a single joint query-to-all-chunk cross attention operation is performed over concatenated all chunks.

This joint attention allows flexible query-document token-level interaction: the query can freely attend to segments in the entire document without chunk boundary limitation. This operation is a generalization of the cross-attention operation in Equation 6. Since the full document is only being attended to here, this operation remains a complexity linear to document length. We call this new design MORES+, indicating a modification over MORES to support long document re-ranking.

For familiar readers, we note this design shares a similar spirit with Fusion-in-Decoder [9] which uses sequence-to-sequence model to combine multiple retrieved contexts for question answering.

Here we compare the time complexities of score-pooling systems, representation aggregation systems, and MORES+. We use BERTmaxp [3] and PARADE-Transformer [13] as concrete examples. We consider a long document of length 𝐷, a query of length 𝑄, and a chunk size 𝐶 for all systems. This gives us O (𝐷/𝐶) chunks. We Query Dependent Query Independent Total BERT-maxp 1: Complexity analysis of attention operations in score-pooling system BERT-maxp, representation aggregation system PARADE-Transformer and our proposed MORES+. This considers a length 𝐷 document, a length 𝑄 query, and length 𝐶 chunks.

focus our analysis here on the (self and cross) attention operations' cost of the Transformers. Note that the other major cost factor is the feed-forward network, which is point-wist and remains a cost linear to input length across all systems.

With both score pooling systems (BERT-maxp) and representation aggregation systems (PARADE-Transformer), each document chunk is concatenated with the query and separately fed into a Transformer. Self attention costs O ((𝐶 + 𝑄) 2 ) each. A total of

The aggregator in score aggregation systems has an extra cost. In PARADE-Transformer, self-attention costs O (𝐷 2 /𝐶 2 ) for latent vectors from O (𝐷/𝐶) chunks.

In MORES+, each of the documents is encoded independently to the query. Self attention costs O (𝐶 2 ) for each chunk, a total of O (𝐶𝐷). Query encoding costs O (𝑄 2 ). Finally, the interaction module performs cross attention from a length 𝑄 query to O (𝐷/𝐶) of the length 𝐶 chunks, or equivalently 𝐷 document tokens. This costs O (𝑄𝐷) time.

Note that similar to MORES, MORES+ encodes documents independent to the query. These query independent representations can therefore be pre-computed offline. In comparison, both BERTmaxp and PARADE-Transformer start from the concatenated query and document chunks. They, therefore, do not permit any precomputation. In Table 1, we record the complexities of all three systems. Note that a common situation is that a document is a few times longer than a chunk and a chunk is much longer than a query, i.e. 𝐷 > 𝐶 >> 𝑄. With pre-computation, it is possible to move a costly O (𝐶𝐷) term offline. On the other hand, when everything is run online, MORES+ has complexity similar to compared systems.

Encoder-only LMs like BERT [4] and RoBERTa [15] do not perform cross attention and therefore do not contain weights that can initialize the interaction module. In the original MORES paper, the authors propose to borrow self-attention weights but also found the approach to be non-optimal [7]. Observing the emergence of pretrained sequence-to-sequence encoder-decoder LMs [12,18], we propose to instead initialize using their weights. In particular, we initialize with BART [12], using its encoder for our document encoder module and its decoder for our interaction module. The decoderto-encoder weights are used as MORES+'s query-to-document attention weights. To accommodate this new initialization scheme, we reduce the query encoder module down to a single embedding layer and have queries encoded jointly in the interaction process. In other words, the flows of computation in BART are preserved.

We first verify the effectiveness of MORES+ systems on classical few-shot IR datasets. We use Robust04 and Clueweb09 and report nDCG@20, following Dai and Callan [3]. We consider two versions of queries: short keyword queries (title) and natural language query (description). An example provided by Dai and Callan [3] is shown in Table 2.

We also adopt the MS-MARCO document ranking dataset [1]. MS-MARCO document ranking is a recent popular large-scale supervised dataset constructed from Bing with substantially more training queries. Its leaderboard has received submissions from a variety of production-ready systems. Participants are given Train/Dev sets judgments and the submissions are evaluated based on the hidden Eval set performance. MS-MARCO helps demonstrate MORES+'s ability to handle contemporary real-world web search tasks. Queries in MS-MARCO document ranking are of natural language form. We report the official metric MRR@100 on Dev and Eval (leaderboard) query sets.

We show the three datasets' statistics in Table 3. Note that all three have documents on average longer than the typical LM input length of 512. They help us test the effectiveness of long document re-rankers.

Our MORES+ models are implemented based on BART code in the Huggingface transformer library [20]. The document encoder and interaction module both have 12 layers. We set each input chunk size to 512, similar to the maximum input sequence length of BART during pre-training [12]. We use a maximum of 3 chunks on Robust04 and MS-MARCO and a maximum of 6 chunks on ClueWeb09, preserving up to ∼ 1.5k tokens and ∼ 3k tokens in each input respectively. The models are trained with localized contrastive estimation (LCE) loss [8]. Following previous work [10,13], we first pre-train MORES+ on MS-MARCO passage ranking data to warm up the model. 3We use the commonly adopted Indri initial rankings released by Dai and Callan [3] on Robust04 and Clueweb09. Since one of our major baselines, the previous state-of-the-art system PARADE uses a different set of BM25+RM3 initial rankings on Robust04, we also use MORES+ to re-rank BM25+RM3 initial rankings on Robust04 released by Yang et al. [22]. We perform 5-fold cross-validation on Robust04 and ClueWeb09 as described in Dai and Callan [3].

On MS-MARCO, we follow top leaderboard systems and use a dense initial stage retriever. Our dense retriever is fine-tuned from a pre-trained coCondenser [6]. It combines a special dense pre-training architecture Condenser [5] and contrastive learning to pre-train a tailored model for dense retrieval. It allows us to perform a simple low-cost fine-tuning. For the leaderboard submission, we ensemble 4 models that are trained with different initialization.

On the classical few-shot IR datasets, we compare with score pooling systems BERT-firstP/maxP/sumP [3], and latent representation aggregation system PARADE-CNN/Transformer [13]. We also consider long-text Transformers. QDS-Transformer [10] uses a special attention pattern for ranking. A few universal long-text Transformer baselines, Sparse-Transformer, Longformer, and Transformer-XH are also taken from [10]. A recent method BeST [11] that uses query-based key-block selection is also considered. We also borrow some earlier classical baseline systems from Dai and Callan [3] including SDM, RankSVM and Coor-Ascent as well as pre-BERT neural models DRMM and Conv-KNRM. Due to the high training cost of these long-text models, we directly copy previously reported numbers.

On MS-MARCO, we report the performance of the current top leaderboard systems, as shown in Table 5. Many of them are or are based on real-world production-ready systems [21,24].

Table 4 shows the performance of the baseline systems and MORES+ on Robust04 and ClueWeb09. We see that MORES+ achieves the new state-of-the-art in ranking effectiveness.

On Title queries, many neural re-rankers, despite being of higher capacity, do not show a clear advantage over lighter classical systems. Recall that these Title queries are typically of one or a few words, which are simple but ambiguous. As argued by Dai and Callan [3], they typically require less of the language understanding that neural models excel at. Similarly, these keyword-like queries do not require sophisticated query document interaction. In our experiments, we see on Robust04, the best PARADE systems, which pool passage latent vectors into global relevance scores, have similar performance to MORES+ 4 .

On the other hand, when moving over to Description queries that are written in natural language, neural models start to show [22] instead of Indri initial rankings [3].

their advantages on language understanding. In particular, models that are based on pre-trained LMs have a clear performance margin over classical systems. Meanwhile, the direct interaction between the query and the full long document also proves to be of decent utility. We see that on both Robust04 and ClueWeb09 Description queries, MORES+ has a clear advantage over all previous methods, including score-pooling systems and latent representation pooling systems. As many search engines/portals are taking up more user-friendly natural language form queries, we believe the improvements brought by MORES+ will have further future impacts.

We also see that MORES+ can outperform sparse Transformer based models. Both sparse Transformer re-rankers and MORES+ attempt to preserve granular query-document interaction. Sparse Transformers re-rankers do so by making the self-attention pattern sparse, effectively lowering its cost at the cost of capacity. As discussed by Li et al. [13], these unique patterns of sparse attention are not optimal for re-ranking. In comparison, MORES+ uses another form of query-to-document attention which shows better effectiveness here. On the other hand, sparse Transformers are more efficient than other models. Note that, as discussed in subsection 3.4, modular re-rankers support document representation pre-computations, which can substantially speed up re-ranking by tens of times. We focus on effectiveness impacts in this paper and leave the exploration of efficiency to future work.

In addition, MORES+ also outperforms the key block method BeST. We can conceptually think of the cross attention operation in MORES+ as a soft information selection process. It can freely attend to useful pieces in the document, by assigning higher attention weights. This soft process does not put hard constraints nor require the pieces to be contiguous. In other words, MORES+ can more flexibly select "key information" beyond "key block".

In Table 5, we show performance on the MS-MARCO document ranking dataset. We see that, at the time when this paper is written, MORES+ achieves 2nd best performance on the Eval set and the best on the Dev set. Early work [8] on the MS-MARCO document ranking has shown that a) it requires stronger initial retrievers than BM25, but b) stronger retriever rankings are harder to re-rank. This means, for our discussion, it is important to verify the re-ranker's improvement when paired with a strong retriever. Here we see that MORES+ can successfully re-rank candidates produced by one of the state-of-the-art dense retrievers, achieving competitive results.

When making further comparisons with other runs, we would like to remind our readers that other leaderboard systems are typically heavily engineered systems that combine various techniques. In comparison, ours is a simple two-stage system with a pair of dense retrievers and a modular re-ranker. In the last row of Table 5, we show Dev set performance of the MORES+ system without ensemble. It can still achieve competitive results and outperform several top leaderboard systems.

In this section, we consider an ablation study where we input smaller numbers of chunks to MORES+. Ideally, as the number of chunks increases, a good model should be able to pick up more information and generate better final relevance judgments. It should not be confused by extra noise in the input if the added chunks are not relevant. On the other hand, even with less sufficient information, the model should not fail catastrophically but retain a certain level of performance. In practice, reducing the number of input chunks can help lower the total computation amount. It helps lower search latency when search volume increases.

In Figure 1, we plot number of input chunks versus nDCG@20 on Robust04 and ClueWeb09 using description queries. Note here a special case is the single chunk situation, where MORES+ reduces to the original MORES system. This corresponds to the left-most data point in each plot. Here, MORES+ shows monotonic improvements with respect to the number of input chunks while remaining decently effective with a lower number of chunks.

In this paper, we present MORES+, a new approach for re-ranking long documents. It deviates from previous long document systems that rely on an encoder-only Transformer and is based on a modular re-ranker framework, MORES. We introduce a granular tokenlevel query document interaction method using query to document chunk cross attention. We use this method to augment the original MORES framework for long document re-ranking.

Our experiments demonstrate the effectiveness of MORES+ on both classical IR datasets with few training examples and contemporary large-scale supervised datasets. We found MORES+'s new form of token-level interaction is especially beneficial for natural language queries. It achieves a new state-of-the-art on Robost04 and ClueWeb09 using Description queries, with large improvements over baselines. On MS-MARCO document ranking, MORES+ also shows competitive performance, with second-best Eval and best Dev set performance.

We believe MORES+ opens new possibilities in document reranking using modular re-rankers. Broadly, it reminds people that instead of sticking with the original Transformer encoder architecture used in models like BERT, document ranking can also be solved with innovations in architecture. On the other hand, MORES+ shows competitive performance to real-world systems. Future work can explore the possibility of using MORES+ in real search tasks.