More Like this

1. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

   Yasunaga, Michihiro ; Ren, Hongyu ; Bosselut, Antoine ; Liang, Percy ; Leskovec, Jure ( January 2021 , North American Chapter of the Association for Computational Linguistics (NAACL))

   null (Ed.)

   The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions. 

   more » « less
2. GreaseLM: Graph REASoning Enhanced Language Models for Question Answering

   https://doi.org/10.48550/arXiv.2201.08860  

   Zhang, X ; Bosselut, A ; Yasunaga, M ; Ren, H ; Liang, P ; Manning, C ; Leskovec, J ( January 2022 , International Conference on Representation Learning (ICLR))

   Answering complex questions about textual narratives requires reasoning over both stated context and the world knowledge that underlies it. However, pretrained language models (LM), the foundation of most modern QA systems, do not robustly represent latent relationships between concepts, which is necessary for reasoning. While knowledge graphs (KG) are often used to augment LMs with structured representations of world knowledge, it remains an open question how to effectively fuse and reason over the KG representations and the language context, which provides situational constraints and nuances. In this work, we propose GreaseLM, a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. Information from both modalities propagates to the other, allowing language context representations to be grounded by structured world knowledge, and allowing linguistic nuances (e.g., negation, hedging) in the context to inform the graph representations of knowledge. Our results on three benchmarks in the commonsense reasoning (i.e., CommonsenseQA, OpenbookQA) and medical question answering (i.e., MedQA-USMLE) domains demonstrate that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger. 

   more » « less
3. Deep Bidirectional Language-Knowledge Graph Pretraining

   Yasunaga, Michihiro ; Bosselut, Antoine ; Ren, Hongyu ; Zhang, Xikun ; Manning, Christopher D. ; Liang, Percy ; Leskovec, Jure ( January 2022 , Advances in neural information processing systems)

   Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon. 

   more » « less
4. SpartQA: A Textual Question Answering Benchmark for Spatial Reasoning

   Roshanak Mirzaee, Hossein Rajaby ( April 2021 , The 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-2021))

   null (Ed.)

   This paper proposes a question-answering (QA) benchmark for spatial reasoning on nat- ural language text which contains more real- istic spatial phenomena not covered by prior work and is challenging for state-of-the-art language models (LM). We propose a distant supervision method to improve on this task. Specifically, we design grammar and reason- ing rules to automatically generate a spatial de- scription of visual scenes and corresponding QA pairs. Experiments show that further pre- training LMs on these automatically generated data significantly improves LMs’ capability on spatial understanding, which in turn helps to better solve two external datasets, bAbI, and boolQ. We hope that this work can foster inves- tigations into more sophisticated models for spatial reasoning over text. 

   more » « less
5. How Can We Know What Language Models Know?

   https://doi.org/10.1162/tacl_a_00324  

   Jiang, Zhengbao ; Xu, Frank F. ; Araki, Jun ; Neubig, Graham ( July 2020 , Transactions of the Association for Computational Linguistics)

   Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA . 

   more » « less