Recent question answering systems perform well on benchmark datasets (Seo et al., 2017;Devlin et al., 2019;Guu et al., 2020), but these models often lack the ability to verify whether an answer is correct or not; they can correctly reject some unanswerable questions (Rajpurkar et al., 2018;Kwiatkowski et al., 2019;Asai and Choi, 2021), but are not always well-calibrated to spot spurious answers under distribution shifts (Jia and Liang, 2017;Kamath et al., 2020). Natural language inference (NLI) (Dagan et al., 2005;Bowman et al., 2015) suggests one way to address this shortcoming: logical entailment provides a more rigorous
Decontextualization of the answer sentence
Answer is correct, but information about Michael being the bad guy is missing in the premise
Not entailed, answer rejected Figure 1: An example from the Natural Questions dataset demonstrating how to convert a (question, context, answer) triplet to a (premise, hypothesis) pair. The underlined text denotes the sentence containing the answer Ted Danson, which is then decontextualized by replacing The series with The series The Good Place. Although Ted Danson is the right answer, an NLI model determines that the hypothesis is not entailed by the premise due to missing information. notion for when a hypothesis statement is entailed by a premise statement. By viewing the answer sentence in context as the premise, paired with the question and its proposed answer as a hypothesis (see Figure 1), we can use NLI systems to verify that the answer proposed by a QA model satisfies the entailment criterion (Harabagiu and Hickl, 2006;Richardson et al., 2013).
Prior work has paved the way for this application of NLI. Pieces of our pipeline like converting a question to a declarative sentence (Wang et al., 2018;Demszky et al., 2018) and reformulating an answer sentence to stand on its own (Choi et al., 2021) have been explored. Moreover, an abundance of NLI datasets (Bowman et al., 2015;Williams et al., 2018) and related fact verification datasets (Thorne et al., 2018) provide ample resources to train reliable models. We draw on these tools to enable NLI models to verify the answers from QA systems, and critically investigate the benefits and pitfalls of such a formulation.
Mapping QA to NLI enables us to exploit both NLI and QA datasets for answer verification, but as Figure 1 shows, it relies on a pipeline for mapping a (question, answer, context) triplet to a (premise, hypothesis) NLI pair. We implement a strong pipeline here: we extract a concise yet sufficient premise through decontextualization (Choi et al., 2021), which rewrites a single sentence from a document such that it can retain the semantics when presented alone without the document. We improve a prior question conversion model (Demszky et al., 2018) with a stronger pre-trained seq2seq model, namely T5 (Raffel et al., 2020). Our experimental results show that both steps are critical for mapping QA to NLI. Furthermore, our error analysis shows that these two steps of the process are quite reliable and only account for a small fraction of the NLI verification model's errors.
Our evaluation focuses on two factors. First, can NLI models be used to improve calibration of QA models or boost their confidence in their decisions? Second, how does the entailment criterion of NLI, which is defined somewhat coarsely by crowd annotators (Williams et al., 2018), transfer to QA? We train a QA model on Natural Questions (Kwiatkowski et al., 2019, NQ) and test whether using an NLI model helps it better generalize to four out-of-domain datasets from the MRQA shared task (Fisch et al., 2019). We show that by using the question converter, the decontextualization model, and the automatically generated NLI pairs from QA datasets, our NLI model improves the calibration over the base QA model across five different datasets. 1 For example, in the selective QA setting (Kamath et al., 2020), our approach improves the F1 score of the base QA model from 81.6 to 87.1 when giving answers on the 20% of questions it is most confident about. Our pipeline further identifies the cases where there exists an information mismatch between the premise and the hypothesis. We find that existing QA datasets encourage models to return answers when the context does not actually contain sufficient information, suggesting that fully verifying the answers is a challenging endeavor. 1 The converted NLI datasets, the question converter, the decontextualizer, and the NLI model are available at https://github.com/jifan-chen/ QA-Verification-Via-NLI 2 Using NLI as a QA Verifier
Using entailment for QA is an old idea; our highlevel approach resembles the approach discussed in Harabagiu and Hickl (2006). Yet, the execution of this idea differs substantially as we exploit modern neural systems and newly proposed annotated data for passage and question reformulation. Richardson et al. (2013) explore a similar pipeline, but find that it works quite poorly, possibly due to the low performance of entailment systems at the time (Stern and Dagan, 2011). We believe that a combination of recent advances in natural language generation (Demszky et al., 2018;Choi et al., 2021) and strong models for NLI (Liu et al., 2019) equip us to re-evaluate this approach.
Moreover, the focus of other recent work in this space has been on transforming QA datasets into NLI datasets, which is a different end. Demszky et al. (2018) and Mishra et al. (2021) argue that QA datasets feature more diverse reasoning and can lead to stronger NLI models, particularly those better suited to strong contexts, but less attention has been paid to whether this agrees with classic definitions of entailment (Dagan et al., 2005) or short-context NLI settings (Williams et al., 2018).
Our work particularly aims to shed light on information sufficiency in question answering. Other work in this space has focused on validating answers to unanswerable questions (Rajpurkar et al., 2018;Kwiatkowski et al., 2019), but such questions may be nonsensical in context; these efforts do not address whether all aspects of a question have been covered. Methods to handle adversarial SQuAD examples (Jia and Liang, 2017) attempt to do this (Chen and Durrett, 2021), but these are again geared towards detecting specific kinds of mismatches between examples and contexts, like a changed modifier of a noun phrase. Kamath et al. (2020) frame their selective question answering techniques in terms of spotting out-of-domain questions that the model is likely to get wrong rather than more general confidence estimation. What is missing in these threads of literature is a formal criterion like entailment: when is an answer truly sufficient and when are we confident that it addresses the question?
Our pipeline consists of an answer candidate generator, a question converter, and a decontextualizer, which form the inputs to the final entailment model.
In this work, we focus our attention on extractive QA (Hermann et al., 2015;Rajpurkar et al., 2016), for which we can get an answer candidate by running a pre-trained QA model. 2 We use the Bert-joint model proposed by Alberti et al. (2019) for its simplicity and relatively high performance.
Question Conversion Given a question q and an answer candidate a, our goal is to convert the (q, a) pair to a declarative answer sentence d which can be treated as the hypothesis in an NLI system (Demszky et al., 2018;Khot et al., 2018). While rulebased approaches have long been employed for this purpose (Cucerzan and Agichtein, 2005), the work of Demszky et al. (2018) showed a benefit from more sophisticated neural modeling of the distribution P (d | q, a). We fine-tune a seq2seq model, T5-3B (Raffel et al., 2020), using the (a, q, d) pairs annotated by Demszky et al. (2018).
While the conversion is trivial on many examples (e.g., replacing the wh-word with the answer and inverting the wh-movement), we see improvement on challenging examples like the following NQ question: the first vice president of India who became the president later was? The rule-based system from Demszky et al. (2018) just replaces who with the answer Venkaiah Naidu. Our neural model successfully appends the answer to end of the question and gets the correct hypothesis.
Decontextualization Ideally, the full context containing the answer candidate could be treated as the premise to make the entailment decision. But the full context often contains many irrelevant sentences and is much longer than the premises in single-sentence NLI datasets (Williams et al., 2018;Bowman et al., 2015). This length has several drawbacks. First, it makes transferring models from the existing datasets challenging. Second, performing inference over longer forms of text requires a multitude of additional reasoning skills like coreference resolution, event detection, and abduction (Mishra et al., 2021). Finally, the presence of extraneous information makes it harder to evaluate the entailment model's judgments for correctness; in the extreme, we might have to judge whether a fact about an entity is true based on its entire Wikipedia article, which is impractical.
We tackle this problem by decontextualizing the sentence containing the answer from the full context to make it stand alone. Recent work (Choi et al., 2021) proposed a sentence decontextualization task in which a sentence together with its context are taken and the sentence is rewritten to be interpretable out of context if feasible, while preserving its meaning. This procedure can involve name completion (e.g., Stewart → Kristen Stewart), noun phrase/pronoun swap, bridging anaphora resolution, and more.
More formally, given a sentence S a containing the answer and its corresponding context C, decontextualization learns a model P (S d | S a , C), where S d is the decontextualized form of S a . We train a decontextualizer by fine-tuning the T5-3B model to decode S d from a concatenation of (S a , C) pair, following the original work. More details about the models we discuss here can be found in Appendix B.
Our experiments seek to validate the utility of NLI for verifying answers primarily under distribution shifts, following recent work on selective question answering (Kamath et al., 2020). We transfer an NQ-trained QA model to a range of datasets and evaluate whether NLI improves answer confidence.
Datasets We use five English-language spanextractive QA datasets: Natural Questions (Kwiatkowski et al., 2019, NQ), TriviaQA (Joshi et al., 2017), BioASQ (Tsatsaronis et al., 2015), Adversarial SQuAD (Jia and Liang, 2017, SQuADadv), and SQuAD 2.0 (Rajpurkar et al., 2018). For TriviaQA and BioASQ, we use processed versions from MRQA (Fisch et al., 2019). These datasets cover a wide range of domains including biology (BioASQ), trivia questions (TriviaQA), real user questions (NQ), and human-synthetic challenging sets (SQuAD2.0 and SQuAD-adv). For NQ, we filter out the examples in which the questions are narrative statements rather than questions by the rulebased system proposed by Demszky et al. (2018). We also exclude the examples based on tables because they are not compatible with the task formulation of NLI.foot_1
Where was Dyrrachium located? (Answerable) What naval base fell to the Normans? (Unanswerable)
Adriatic Dyrrachium
Dyrrachium was located in Adriatic.
The naval base Dyrrachium fell to the Normans.
Dyrrachium -one of the most important naval bases of the Adriatic -fell again to Byzantine hands.
Dyrrachium -one of the most important naval bases of the Adriatic -fell again to Byzantine hands.
Entail Not Entail Base QA Model We train our base QA model (Alberti et al., 2019) with the NQ dataset.
To study robustness across different datasets, we fix the base QA model and investigate its capacity to transfer. We chose NQ for its high quality and the diverse topics it covers.
Base NLI Model We use the RoBERTa-based NLI model trained using Multi-Genre Natural Language Inference (Williams et al., 2018, MNLI) from AllenNLP (Gardner et al., 2018) for its broad coverage and high accuracy.
QA-enhanced NLI Model As there might exist different reasoning patterns in the QA datasets which are not covered by the MNLI model (Mishra et al., 2021), we study whether NLI pairs generated from QA datasets can be used jointly with the MNLI data to improve the performance of an NLI model. To do so, we run the QA instances in the NQ training set through our QA-to-NLI conversion pipeline, resulting in a dataset we call NQ-NLI, containing (premise, hypothesis) pairs from NQ with binary labels. As answer candidates, we use the predictions of the base QA model. If the predicted answer is correct, we label the (premise, hypothesis) as positive (entailed), otherwise negative (not entailed). To combine NQ-NLI with MNLI, we treat the examples in MNLI labeled with "entailment" as positive and the others as negative.
We take the same number of examples as of NQ-NLI from MNLI and shuffle them to get a mixed dataset which we call NQ-NLI+MNLI. We use these dataset names to indicate NLI models trained on these datasets. Some basic statistics for each dataset after processing with our pipeline are shown in Appendix A.
In this section, we explore to what extent either off-the-shelf or QA-augmented NLI models work as verifiers across a range of QA datasets.
We start by testing how well a pre-trained MNLI model, with an accuracy of 90.2% on held-out MNLI examples, can identify unanswerable questions in SQuAD2.0. We run our pre-trained QA model on the unanswerable questions to produce answer candidates, then convert them to the NLI pairs through our pipeline, including question conversion and decontextualization. We run the entailment model trained on MNLI to see how frequently it is able to reject the answer by predicting either "neutral" or "contradiction". For questions with annotated answers, we also generate the NLI pairs with the gold answer and see if the entailment model trained on MNLI can accept the answer.
The MNLI model successfully rejects 78.5% of the unanswerable examples and accepts 82.5% of the answerable examples. Two examples taken from SQuAD2.0 are shown in Figure 2. We can see the MNLI model is quite sensitive to the information mismatch between the hypothesis and the premise. In the case where there is no information about Normans in the premise, it rejects the answer. Without seeing any data from SQuAD2.0, MNLI can already act as a strong verifier in the unanswerable setting where it is hard for a QA model to generalize (Rajpurkar et al., 2018).
To analyze the effectiveness of the NLI models in a more systematic way, we test whether they can improve calibration of QA models or improve model performance in a "selective" QA setting (Kamath et al., 2020). That is, if our model can choose to answer only the k percentage of examples it is most confident about (the coverage), what F1 can it achieve? We first rank the examples by the confidence score of a model; for our base QA models, this score is the posterior probability of the answer span, and for our NLI-augmented models, it is the posterior probability associated with the "entailment" class. We then compute F1 scores at different coverage values. 2021) since it follows their procedure. All of the models are initialized using RoBERTa-large (Liu et al., 2019) and trained using the same configurations.
NLI+QA We explore combining complementary strengths of the NLI posteriors and the base QA posteriors. We take the posterior probability of the two models as features and learn a binary classifier y = logistic(w 1 p QA + w 2 p NLI ) as the combined entailment model and tune the model on 100 heldout NQ examples. +QA denotes this combination with any of our NLI models.
To compare with NLI+QA, we train another identical QA model, Bert-joint, using the same configurations and ensemble the two QA models using the same way as NLI+QA.
Selective QA Kamath et al. (2020) train a calibrator to make models better able to selectively answer questions in new domains. The calibrator is a binary classifier with seven features: passage length, the length of the predicted answer, and the top five softmax probabilities output by the QA model. We use the same configuration as (Kamath
Figure 3 shows the macro-averaged results over the five QA datasets. Please refer to Appendix C for per dataset breakdown.
Our NQ-NLI+QA system, which combines the QA models' posteriors with an NQ-NLI-trained system, already shows improvement over using the base QA posteriors. Surprisingly, additionally training the NLI model on MNLI (NQ-NLI+MNLI+QA) gives even stronger results. The NLI models appear to be complementary to the QA model, improving performance even on out-of-domain data. We also see that our our NQ-NLI+MNLI+QA outperforms Mishra et al. (2021)+QA by a large margin. By inspecting the performance breakdown in Appendix C, we see the gap is mainly on SQuAD2.0 and SQuADadv. This is because these datasets often introduce subtle mismatches by slight modification of the question or context; even if the NLI model is able to overcome other biases, these are challenging contrastive examples from the standpoint of the NLI model. This observation also indicates that to better utilize the complementary strength of MNLI, the proposed decontextualization phase in our pipeline is quite important.
Selective QA shows similar performance to using the posterior from QA model, which is the most important feature for the calibrator.
Combining NLI model with the base QA models' posteriors is necessary for this strong performance.
Figure 4 shows the low performance achieved by the NLI models alone, indicating that NLI models trained exclusively on NLI dataset (FEVER-NLI, MNLI) cannot be used by themselves as effective verifiers for QA. This also indicates a possible domain or task mismatch between FEVER, MNLI, and the other QA datasets. NQ-NLI helps bridge the gap between the QA datasets and MNLI. In Figure 4, both NQ-NLI and NQ-NLI+MNLI achieve similar performance to the original QA model. We also find that training using both NQ-NLI and MNLI achieves slightly better performance than training using NQ-NLI alone. This suggests that we are not simply training a QA model of a different form by using the NQ-NLI data; rather, the NQ-NLI pairs are compatible with the MNLI pairs, and the MNLI examples are useful for the model.
We present an ablation study on our pipeline to see how each component contributes to the final performance. For simplicity, we use the off-the-shelf MNLI model since it does not involve training using the data generated through the pipeline. Figure 5 shows the average results across five datasets and Figure 6 presents individual performance on three datasets.
We see that both the question converter and the decontextualizer contribute to the performance of the MNLI model. In both figures, removing either module harms the performance for all datasets. On NQ and BioASQ, using the full context is better than the decontextualized sentence, which hints that there are cases where the full context provides necessary information. We have a more comprehensive analysis in Section 6.2.
Moreover, we see that MNLI outperforms the base QA posteriors on SQuAD2.0 and SQuADadv. Figure 6(a) also shows that the largest gap between the QA and NLI model is on NQ, which is unsurprising since the QA model is trained on NQ. These results show how the improvement in the last section is achieved: the complementary strengths of MNLI and NQ datasets lead to the best overall performance.
We perform manual analysis on 300 examples drawn from NQ, TriviaQA, and SQuAD2.0 datasets where NQ-NLI+MNLI model produced an error. We classify errors into one of 7 classes, described in Section 6.1 and 6.2. All of the authors of this paper conducted the annotation. The annotations agree with a Fleiss' kappa value of 0.78, with disagreements usually being between closely related categories among our 7 error classes, e.g., annotation error vs. span shifting, wrong context vs. insufficient context, as we will see later. The breakdown of the errors in each dataset is shown in Table 1.
We see that across the three different datasets, the number of errors attributed to our pipeline approach is below 10%. This demonstrates that the question converter and the decontextualization model are quite effective to convert a (question, answer, context) triplet to a (premise, hypothesis) NLI pair. For the question converter, errors mainly happen in two scenarios as shown in Figure 7.
(1) The question converter gives an answer of the wrong type to a question. For example, the question asks "How old...", but the answer returned is "Mike Pence" which does not fit the question. The question converter puts Mike Pence back into the question and yields an unrelated statement. Adding a presupposition checking stage to the question converter could further improve its performance (Kim et al., 2021). ( 2) The question is long and syntactically complex; the question converter just copies a long question without answer replacement.
For the decontextualization model, errors usually happen when the model fails to recall one of the required modifications. As shown in the example in Figure 7, the model fails to replace The work with its full entity name The Art of War.
Most of the errors are attributed to the entailment model. We investigate these cases closely and ask ourselves if these really are errors. We categorize them into the following categories.
Entailment These errors are truly mistakes by the entailment model: in our view, the pair of sentences should exhibit a different relationship than what was predicted.
Wrong Context The QA model gets the right answer for the wrong reason. The example in Figure 8 shows that John Von Neumann is the annotated answer but it is not entailed by the premise because no information about CPU is provided. Although the answer is correct, we argue it is better for the model to reject this case. This again demonstrates one of the key advantages of using an NLI model as a verifier for QA models: it can identify cases of information mismatch like this where the model didn't retrieve suitable context to show to the user of the QA system.
Insufficient Context (out of scope for decontextualization) The premise lacks essential information that could be found in the full context, typically later in the context. In Figure 8, the answer Roxette is in the first sentence. However, we do not know that she wrote the song It Must Have Been Love until we go further in the context. The need to add future information is beyond the scope of the decontextualization (Choi et al., 2021).
The predicted answer of the QA model overlaps with the gold answer and it is acceptable as a correct answer. For example, a question asks What Missouri town calls itself the Live Music Show Capital? Both Branson and Branson, Missouri can be accepted as the right answer.
Annotation Error Introduced by the incomplete or wrong annotations -some acceptable answers are missing or the annotated answer is wrong.
From Table 1, we see that "wrong context" cases consist of 25% and 40% of the errors for NQ and TriviaQA, respectively, while they rarely happen on SQuAD2.0. This is because the supporting snippets for NQ and TriviaQA are retrieved from Wikipedia and web documents, so the information contained may not be sufficient to support the question. For SQuAD2.0, the supporting document is given to the annotators, so no such errors happen. This observation indicates that the NLI model can be particularly useful in the open-domain setting where it can reject answers that are not well supported. In particular, we believe that this raises a question about answers in TriviaQA. The supporting evidence for the answer is often insufficient to validate all aspects of the question. What should a QA model do in this case: make an educated guess based on partial evidence, or reject the answer outright? This choice is applicationspecific, but our approach can help system designers make these decisions explicit.
Around 10% to 15% of errors happens due to insufficient context. Such errors could be potentially fixed in future work by learning a questionconditioned decontextualizer which aims to gather all information related to the question. 2019) used NLI models to detect factual errors in abstractive summaries. For question answering, Harabagiu and Hickl (2006) showed that textual entailment can be used to enhance the accuracy of the opendomain QA systems; Trivedi et al. (2019) used a pretrained NLI model to select relevant sentences for multi-hop question answering; Yin et al. (2020) tested whether NLI models generalize to QA setting in a few-shot learning scenario.
Our work is most relevant to Mishra et al. (2021); they also learn an NLI model using examples generated from QA datasets. Our work differs from theirs in a few chief ways. First, we improve the conversion pipeline significantly with decontextualization and a better question converter. Second, we use framework to improve QA performance by using NLI as a verifier, which is only possible because the decontextualization allows us to focus on a single sentence. We also study whether the converted dataset is compatible with other off-the-shelf NLI datasets. By contrast, Mishra et al. (2021) use their converted NLI dataset to aid other tasks such as fact-checking. Finally, the contrast we establish here allows us to conduct a thorough human analysis over the converted NLI data and show how the task specifications of NLI and QA are different (Section 6.2).
Robust Question Answering Modern QA systems often give incorrect answers in challenging settings that require generalization (Rajpurkar et al., 2018;Chen and Durrett, 2019;Wallace et al., 2019;Gardner et al., 2020;Kaushik et al., 2019). Models focusing on robustness and generalizability have been proposed in recent years: Wang and Bansal (2018); Khashabi et al. (2020); Liu et al. (2020) use perturbation based methods and adversarial training; Lewis and Fan (2018) propose generative QA to prevent the model from overfitting to simple patterns; Yeh and Chen ( 2019 Another line of work to make models more robust is by introducing answer verification (Hu et al., 2019;Kamath et al., 2020;Wang et al., 2020;Zhang et al., 2021) as a final step for question answering models. Our work is in the same vein, but has certain advantages from using an NLI model.
First, the answer verification process is more ex-Entailment Error (NLI Prediction: Not Entail) Question: What were the results of the development of Florida's railroads? Predicted / Gold Answer: towns grew and farmland was cultivated / towns grew and farmland was cultivated Hypothesis: The results of the development of Florida's railroads were that towns grew and farmland was cultivated. Premise: Henry Flagler built a railroad along the east coast of Florida and eventually to Key West; towns grew and farmland was cultivated along the rail line. plicit so that one is able to spot where the error emerges. Second, we can incorporate NLI datasets from other domains into the training of our verifier, reducing reliance on in-domain labeled QA data.
This work presents a strong pipeline for converting QA examples into NLI examples, with the intent of verifying the answer with NLI predictions. The answer to the question posed in the title is yes (NLI models can validate these examples), with two caveats. First, it is helpful to create QA-specific data for the NLI model. Second, the information that is sufficient for a question to be fully answered may not align with annotations in the QA dataset. We encourage further explorations of the interplay between these tasks and careful analysis of the predictions of QA models.
The statistics of the datasets after processing through our pipeline is shown in Table 2. Both the premise length and the hypothesis length are quite similar except for the premise length of Triv-iaQA, despite their original context length differs greatly (Fisch et al., 2019).
We train our Bert-joint on the full NQ training set for 1 epoch. We initialize the model with bert-large-uncased-wwm. 4 The batch size is set to 8, window size is set to 512, and the optimizer we use is Adam (Kingma and Ba, 2015) with initial learning rate setting to 3e-5.
Each instance of the input is constructed as
, where [CLS] and [S] are the classification and separator tokens of the T5 model respectively. The output is the target sentence d.
The model is trained using the seq2seq framework of Huggingface (Wolf et al., 2020). The max source sequence length is set to 256 and the target sequence length is set to 512. Batch size is set to 12 and we use Deepspeed for memory optimization (Rasley et al., 2020). We train the model with 86k question-answer pairs for 1 epoch with Adam optimizer and an initial learning rate set to 3e-5. 95% of question answer pairs come from SQuAD and the remaining 5% come from four other question answering datasets (Demszky et al., 2018 Here, "Prem len" and "Hyp len" denote the average number of words with stop words removed in the premise and hypothesis respectively; "Word Overlap" denotes the Jaccard similarity between the premise and the hypothesis.
Each instance of the input is constructed as follows:
[CLS]T[S]x 1 , ..., x t-1 [S]x t [S]x t+1 , ..., x n [S] where [CLS] and [S] are the classification and separator tokens of the T5 model respectively. T denotes the context title which could be empty. x i denotes the ith sentence in the context and x t is the target sentence to decontextualize.
The model is trained using the seq2seq framework of Huggingface (Wolf et al., 2020). The max sequence length for both source and target is set to 512. Batch size is set to 4 and we use Deepspeed for memory optimization (Rasley et al., 2020). We train the model with 11k questionanswer pairs (Choi et al., 2021) for 5 epoch with Adam optimizer and an initial learning rate set to 3e-5.
The generated NQ-NLI training and development set contain 191k and 4,855 (premise, hypothesis) pairs from NQ respectively. We initialize the model with roberta-large (Liu et al., 2019) and train the model for 5 epochs. Batch size is set to 16, with Adam as the optimizer and initial learning rate set to 2e-6.
Our approach could be adapted to multiple choice QA, in which case this step could be omitted.
After filtering, we have 191,022/4,855 examples for the training and development sets respectively. For comparison, the original NQ contains 307,373/7,842 examples for training and development.