More Like this

1. How Can We Know What Language Models Know?

   https://doi.org/10.1162/tacl_a_00324  

   Jiang, Zhengbao; Xu, Frank F.; Araki, Jun; Neubig, Graham (July 2020, Transactions of the Association for Computational Linguistics)

   Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA . 

   more » « less
2. What do tokens know about their characters and how do they know it?

   https://doi.org/10.18653/v1/2022.naacl-main.179  

   Kaushal, Ayush; Mahowald, Kyle (January 2022, Proceedings of NAACL)
3. Craniofacial skeletal response to encephalization: How do we know what we think we know?

   https://doi.org/10.1002/ajpa.23766  

   Lesciotto, Kate M.; Richtsmeier, Joan T. (January 2019, American Journal of Physical Anthropology)

   Abstract Dramatic changes in cranial capacity have characterized human evolution. Important evolutionary hypotheses, such as the spatial packing hypothesis, assert that increases in relative brain size (encephalization) have caused alterations to the modern human skull, resulting in a suite of traits unique among extant primates, including a domed cranial vault, highly flexed cranial base, and retracted facial skeleton. Most prior studies have used fossil or comparative primate data to establish correlations between brain size and cranial form, but the mechanistic basis for how changes in brain size impact the overall shape of the skull resulting in these cranial traits remains obscure and has only rarely been investigated critically. We argue that understanding how changes in human skull morphology could have resulted from increased encephalization requires the direct testing of hypotheses relating to interaction of embryonic development of the bones of the skull and the brain. Fossil and comparative primate data have thoroughly described the patterns of association between brain size and skull morphology. Here we suggest complementing such existing datasets with experiments focused on mechanisms responsible for producing the observed patterns to more thoroughly understand the role of encephalization in shaping the modern human skull. 

   more » « less
4. Taxation and Innovation: What Do We Know?

   https://doi.org/10.3386/w27109  

   Akcigit, Ufuk; Stantcheva, Stefanie (November 2021, Innovation and Public Policy)

   Goolsbee, Austan; Jones, Benjamin (Ed.)

   Tax policies are a wide array of tools, commonly used by governments to influence the economy. In this paper, we review the many margins through which tax policies can affect innovation, the main driver of economic growth in the long-run. These margins include the impact of tax policy on i) the quantity and quality of innovation; ii) the geographic mobility of innovation and inventors across U.S. states and countries; iii) the declining business dynamism in the U.S., firm entry, and productivity; iv) the quality composition of firms, inventors, and teams; and v) the direction of research effort, e.g., toward applied versus basic research, or toward dirty versus clean technologies. We give ideas drawn from research on how the design of policy can allow policy makers to foster the most productive firms without wasting public funds on less productive ones. 

   more » « less
5. Optimal Design of Large-scale Bayesian Linear Inverse Problems Under Reducible Model Uncertainty: Good to Know What You Don't Know

   https://doi.org/10.1137/20M1347292  

   Alexanderian, Alen; Petra, Noemi; Stadler, Georg; Sunseri, Isaac (January 2021, SIAM/ASA Journal on Uncertainty Quantification)

   null (Ed.)