More Like this

1. How Can We Know What Language Models Know?

   https://doi.org/10.1162/tacl_a_00324  

   Jiang, Zhengbao; Xu, Frank F.; Araki, Jun; Neubig, Graham (July 2020, Transactions of the Association for Computational Linguistics)

   Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA . 

   more » « less
2. Taxation and Innovation: What Do We Know?

   https://doi.org/10.3386/w27109  

   Akcigit, Ufuk; Stantcheva, Stefanie (November 2021, Innovation and Public Policy)

   Goolsbee, Austan; Jones, Benjamin (Ed.)

   Tax policies are a wide array of tools, commonly used by governments to influence the economy. In this paper, we review the many margins through which tax policies can affect innovation, the main driver of economic growth in the long-run. These margins include the impact of tax policy on i) the quantity and quality of innovation; ii) the geographic mobility of innovation and inventors across U.S. states and countries; iii) the declining business dynamism in the U.S., firm entry, and productivity; iv) the quality composition of firms, inventors, and teams; and v) the direction of research effort, e.g., toward applied versus basic research, or toward dirty versus clean technologies. We give ideas drawn from research on how the design of policy can allow policy makers to foster the most productive firms without wasting public funds on less productive ones. 

   more » « less
3. Considering What We Know and What We Don’t Know: Expectations and Confidence Guide Value Integration in Value-Based Decision-Making

   https://doi.org/10.1162/opmi.a.3  

   Frömer, Romy; Callaway, Frederick; Griffiths, Thomas L; Shenhav, Amitai (January 2025, Open Mind)

   Abstract When making decisions, we often have more information about some options than others. Previous work has shown that people are more likely to choose options that they look at more and those that they are more confident in. But should one always prefer options one knows more about? Intuition suggests not. Rather, how additional information impacts our preferences should depend critically on how valuable we expect the options to be. Here, we formalize this intuition in a Bayesian sequential sampling model where attention and confidence influence the precision of momentary evidence. Our model makes a key prediction: attention and confidence both increase choice probability for better-than-average options, and both decrease choice probability for worse-than-average options. We confirm this prediction in two experiments in which we independently manipulate value and attention. Our results offer a novel perspective on prior work on the role of attention and confidence in decision-making, showing that people rely on contextual knowledge and uncertainty estimates to adaptively learn about their options and make better decisions. 

   more » « less
4. What we (don't) know about global plant diversity

   https://doi.org/10.1111/ecog.04481  

   Cornwell, William K.; Pearse, William D.; Dalrymple, Rhiannon L.; Zanne, Amy E. (September 2019, Ecography)
5. Soil Carbon Saturation: What Do We Really Know?

   https://doi.org/10.1111/gcb.70197  

   Georgiou, Katerina; Angers, Denis; Champiny, Ryan E; Cotrufo, M Francesca; Craig, Matthew E; Doetterl, Sebastian; Grandy, A Stuart; Lavallee, Jocelyn M; Lin, Yang; Lugato, Emanuele; et al (May 2025, Global Change Biology)

   ABSTRACT Managing soils to increase organic carbon storage presents a potential opportunity to mitigate and adapt to global change challenges, while providing numerous co‐benefits and ecosystem services. However, soils differ widely in their potential for carbon sequestration, and knowledge of biophysical limits to carbon accumulation may aid in informing priority regions. Consequently, there is great interest in assessing whether soils exhibit a maximum capacity for storing organic carbon, particularly within organo–mineral associations given the finite nature of reactive minerals in a soil. While the concept of soil carbon saturation has existed for over 25 years, recent studies have argued for and against its importance. Here, we summarize the conceptual understanding of soil carbon saturation at both micro‐ and macro‐scales, define key terminology, and address common concerns and misconceptions. We review methods used to quantify soil carbon saturation, highlighting the theory and potential caveats of each approach. Critically, we explore the utility of the principles of soil carbon saturation for informing carbon accumulation, vulnerability to loss, and representations in process‐based models. We highlight key knowledge gaps and propose next steps for furthering our mechanistic understanding of soil carbon saturation and its implications for soil management. 

   more » « less