More Like this

1. He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

   https://doi.org/10.1162/tacl_a_00414  

   Jiang, Nanjiang; de Marneffe, Marie-Catherine (January 2021, Transactions of the Association for Computational Linguistics)

   Abstract We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that correlate with certain factuality labels, and it fails on instances where pragmatic reasoning is necessary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction. 

   more » « less
2. Asking Clarifying Questions Based on Negative Feedback in Conversational Search

   https://doi.org/10.1145/3471158.3472232  

   Ai, Q.; Bi, K.; Croft, W. B. (July 2021, Proceedings of the 7th ACM International Conference on the Theory of Information Retrieval (ICTIR 2021))

   null (Ed.)

   Users often need to look through multiple search result pages or reformulate queries when they have complex information-seeking needs. Conversational search systems make it possible to improve user satisfaction by asking questions to clarify users’ search intents. This, however, can take significant effort to answer a series of questions starting with “what/why/how”. To quickly identify user intent and reduce effort during interactions, we propose an intent clarification task based on yes/no questions where the system needs to ask the correct question about intents within the fewest conversation turns. In this task, it is essential to use negative feedback about the previous questions in the conversation history. To this end, we propose a Maximum-Marginal-Relevance (MMR) based BERT model (MMR-BERT) to leverage negative feedback based on the MMR principle for the next clarifying question selection. Experiments on the Qulac dataset show that MMR-BERT outperforms state-of-the-art baselines significantly on the intent identification task and the selected questions also achieve significantly better performance in the associated document retrieval tasks. 

   more » « less
3. Understanding BERT rankers under distillation

   https://doi.org/10.1145/3409256.3409838  

   Gao, Luyu; Dai, Zhuyun; Callan, Jamie (September 2020, ACM SIGIR International Conference on the Theory of Information Retrieval)

   null (Ed.)

   Deep language models, such as BERT pre-trained on large corpora,have given a huge performance boost to state-of-the-art information retrieval ranking systems. Knowledge embedded in such models allows them to pick up complex matching signals between passages and queries. However, the high computation cost during inference limits their deployment in real-world search scenarios. In this paper, we study if and how the knowledge for search within BERT can be transferred to a smaller ranker through distillation.Our experiments demonstrate that it is crucial to use a proper distillation procedure, which produces up to nine times speed upwhile preserving the state-of-the-art performance. 

   more » « less
4. Table Search Using a Deep Contextualized Language Model

   https://doi.org/10.1145/3397271.3401044  

   Chen, Zhiyu; Trabelsi, Mohamed; Heflin, Jeff; Xu, Yinan; Davison, Brian D. (January 2020, 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval)

   Pretrained contextualized language models such as BERT have achieved impressive results on various natural language processing benchmarks. Benefiting from multiple pretraining tasks and large scale training corpora, pretrained models can capture complex syntactic word relations. In this paper, we use the deep contextualized language model BERT for the task of ad hoc table retrieval. We investigate how to encode table content considering the table structure and input length limit of BERT. We also propose an approach that incorporates features from prior literature on table retrieval and jointly trains them with BERT. In experiments on public datasets, we show that our best approach can outperform the previous state-of-the-art method and BERT baselines with a large margin under different evaluation metrics. 

   more » « less
5. Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words

   Klafka, Josef; Ettinger, Allyson (July 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics)

   Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings encode about the context words that they are understood to reflect. To address this question, we introduce a suite of probing tasks that enable fine-grained testing of contextual embeddings for encoding of information about surrounding words. We apply these tasks to examine the popular BERT, ELMo and GPT contextual encoders, and find that each of our tested information types is indeed encoded as contextual information across tokens, often with near-perfect recoverability -- but the encoders vary in which features they distribute to which tokens, how nuanced their distributions are, and how robust the encoding of each feature is to distance. We discuss implications of these results for how different types of models break down and prioritize word-level context information when constructing token embeddings. 

   more » « less