In this work, we carry out a data archaeology to infer books that are known to ChatGPT and GPT-4 using a name cloze membership inference query. We find that OpenAI models have memorized a wide collection of copyrighted materials, and that the degree of memorization is tied to the frequency with which passages of those books appear on the web. The ability of these models to memorize an unknown set of books complicates assessments of measurement validity for cultural analytics by contaminating test data; we show that models perform much better on memorized books than on non-memorized books for downstream tasks. We argue that this supports a case for open models whose training data is known.
more »
« less
QuALITY: Question Answering with Long Input Texts, Yes!
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
more »
« less
- Award ID(s):
- 1922658
- NSF-PAR ID:
- 10350913
- Date Published:
- Journal Name:
- NAACL 2022
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Ganguly, Debasis ; Gangopadhyay, Surupendu ; Mitra, Mandar ; Majumder, Prasenjit (Ed.)For most queries, the set of relevant documents spans multiple subtopics. Inspired by the neural ranking models and query-specific neural clustering models, we develop Topic-Mono-BERT which performs both tasks jointly. Based on text embeddings of BERT, our model learns a shared embedding that is optimized for both tasks. The clustering hypothesis would suggest that embeddings which place topically similar text in close proximity will also perform better on ranking tasks. Our model is trained with the Wikimarks approach to obtain training signals for relevance and subtopics on the same queries. Our task is to identify overview passages that can be used to construct a succinct answer to the query. Our empirical evaluation on two publicly available passage retrieval datasets suggests that including the clustering supervision in the ranking model leads to about 16% improvement in identifying text passages that summarize different subtopics within a query.more » « less
-
null (Ed.)The tectonic and paleoceanographic setting of the Great Australian Bight (GAB) and the Mentelle Basin (MB; adjacent to Naturaliste Plateau) offered an outstanding opportunity to investigate Cretaceous and Cenozoic climate change and ocean dynamics during the last phase of breakup among remnant Gondwana continents. Sediment recovered from sites in both regions during International Ocean Discovery Program Expedition 369 will provide a new perspective on Earth’s temperature variation at sub-polar latitudes (60°–62°S) across the extremes of the mid-Cretaceous hot greenhouse climate and the cooling that followed. The primary goals of the expedition were to • Investigate the timing and causes for the rise and collapse of the Cretaceous hot greenhouse climate and how this climate mode affected the climate-ocean system and oceanic biota; • Determine the relative roles of productivity, ocean temperature, and ocean circulation at high southern latitudes during Cretaceous oceanic anoxic events (OAEs); • Identify the main source regions for deep-water and intermediate-water masses in the southeast Indian Ocean and how these changed during Gondwana breakup; • Characterize how oceanographic conditions at the MB changed during the Cenozoic opening of the Tasman Passage and restriction of the Indonesian Gateway; • Resolve questions on the volcanic and sedimentary origins of the Australo-Antarctic Gulf and Mentelle Basin and provide stratigraphic control on the age and nature of the prebreakup successions. Hole U1512A in the GAB recovered a 691 m thick sequence of black claystone ranging from the early Turonian to the early Campanian. Age control is primarily based on calcareous nannofossils, but the presence of other microfossil groups provided consistent but low-resolution control. Despite the lithologic uniformity, long- and short-term variations in natural gamma ray and magnetic susceptibility intensities show cyclic alternations that suggest an orbital control of sediment deposition that will be useful for developing an astrochronology for the sequence. Sites U1513–U1516 were drilled between 850 and 3900 m water depth in the MB and penetrated 774, 517, 517, and 542 meters below seafloor (mbsf), respectively. Under a thin layer of Pleistocene–upper Miocene sediment, Site U1513 cored a succession of Cretaceous units from the Campanian to the Valanginian. Site U1514 sampled an expanded Pleistocene–Eocene sequence and terminated in the upper Albian. The Cenomanian–Turonian interval at Site U1514 recovered deformed sedimentary rocks that probably represent a detachment zone. Site U1515 is located on the west Australian margin at 850 m water depth and was the most challenging site to core because much of the upper 350 m was either chert or poorly consolidated sand. However, the prebreakup Jurassic(?) sediments interpreted from the seismic profiles were successfully recovered. Site U1516 cored an expanded Pleistocene, Neogene, and Paleogene section and recovered a complete Cenomanian/Turonian boundary interval containing five layers with high total organic carbon content. Recovery of well-preserved calcareous microfossil assemblages from different paleodepths will enable generation of paleotemperature and biotic records that span the rise and collapse of the Cretaceous hot greenhouse (including OAEs 1d and 2), providing insight to resultant changes in deep-water and surface water circulation that can be used to test predictions from earth system models. Paleotemperature proxies and other data will reveal the timing, magnitude, and duration of peak hothouse temperatures and any cold snaps that could have allowed growth of a polar ice sheet. The sites will also record the mid-Eocene–early Oligocene opening of the Tasman Gateway and the Miocene–Pliocene restriction of the Indonesian Gateway; both passages have important effects on global oceanography and climate. Understanding the paleoceanographic changes in a regional context provides a global test on models of Cenomanian–Turonian oceanographic and climatic evolution related both to extreme Turonian warmth and the evolution of OAE 2. The Early Cretaceous volcanic rocks and underlying Jurassic(?) sediments cored in different parts of the MB provide information on the timing of different stages of the Gondwana breakup. The recovered cores provide sufficient new age constraints to underpin a reevaluation of the basin-wide seismic stratigraphy and tectonic models for the region.more » « less
-
null (Ed.)The tectonic and paleoceanographic setting of the Great Australian Bight (GAB) and the Mentelle Basin (adjacent to Naturaliste Plateau) offered an opportunity to investigate Cretaceous and Cenozoic climate change and ocean dynamics during the last phase of breakup among remnant Gondwana continents. Sediment recovered from sites in both regions during International Ocean Discovery Program Expedition 369 will provide a new perspective on Earth’s temperature variation at subpolar latitudes (60°–62°S) across the extremes of the mid-Cretaceous hot greenhouse climate and the cooling that followed. Basalts and prebreakup sediments were also recovered and will provide constraints regarding the type and age of the Mentelle Basin basement and processes operating during the break up of Gondwana. The primary goals of the expedition were to 1. Investigate the timing and causes for the rise and collapse of the Cretaceous hot greenhouse climate and how this climate mode affected the climate–ocean system and oceanic biota; 2. Determine the relative roles of productivity, ocean temperature, and ocean circulation at high southern latitudes during Cretaceous oceanic anoxic events (OAEs); 3. Investigate potential source regions for deep-water and intermediate-water masses in the southeast Indian Ocean and how these changed during Gondwana breakup; 4. Characterize how oceanographic conditions at the Mentelle Basin changed during the Cenozoic opening of the Tasman Gateway and restriction of the Indonesian Gateway; and 5. Resolve questions on the volcanic and sedimentary origins of the Australo-Antarctic Gulf and Mentelle Basin and provide stratigraphic control on the age and nature of the prebreakup successions. Hole U1512A in the GAB recovered a 691 m thick sequence of black claystone ranging from the lower Turonian to the lower Campanian. Age control is primarily based on calcareous nannofossils, but the presence of other microfossil groups provided consistent low-resolution control. Despite the lithologic uniformity, long- and short-term variations in natural gamma radiation and magnetic susceptibility show cyclic alternations that suggest an orbital control of sediment deposition, which will be useful for developing an astrochronology for the sequence. Sites U1513, U1514, U1515, and U1516 were drilled in water depths between 850 and 3900 m in the Mentelle Basin and penetrated 774, 517, 517, and 542 meters below seafloor, respectively. Under a thin layer of Pleistocene to upper Miocene sediment, Site U1513 cored a succession of Cretaceous units from the Campanian to the Valanginian, as well as a succession of basalts. Site U1514 sampled an expanded Pleistocene to Eocene sequence and terminated in the upper Albian. The Cenomanian to Turonian interval at Site U1514 is represented by deformed sedimentary rocks that probably represent a detachment zone. Site U1515 is located on the west Australian margin at 850 m water depth and was the most challenging site to core because much of the upper 350 m was either chert or poorly consolidated sand. However, the prebreakup Jurassic(?) sediments interpreted from the seismic profiles were successfully recovered. Site U1516 cored an expanded Pleistocene, Neogene, and Paleogene section and recovered a complete Cenomanian/Turonian boundary interval containing five layers with high organic carbon content. Study of the well-preserved calcareous microfossil assemblages from different paleodepths will enable generation of paleotemperature and biotic records that span the rise and collapse of the Cretaceous hot greenhouse (including OAEs 1d and 2), providing insight to resultant changes in deep-water and surface water circulation that can be used to test predictions from earth system models. Measurements of paleotemperature proxies and other data will reveal the timing, magnitude, and duration of peak hothouse conditions and any cold snaps that could have allowed growth of a polar ice sheet. The sites contain a record of the mid-Eocene to early Oligocene opening of the Tasman Gateway and the Miocene to Pliocene restriction of the Indonesian Gateway; both passages have important effects on global oceanography and climate. Advancing understanding of the paleoceanographic changes in a regional context will provide a global test on models of Cenomanian to Turonian oceanographic and climatic evolution related both to extreme Turonian warmth and the evolution of OAE 2. The Early Cretaceous volcanic rocks and underlying Jurassic(?) sediments cored in different parts of the Mentelle Basin provide information on the timing of different stages of the Gondwana breakup. The recovered cores provide sufficient new age constraints to underpin a reevaluation of the basin-wide seismic stratigraphy and tectonic models for the region.more » « less
-
Synopsis The field of comparative biomechanics strives to understand the diversity of the biological world through the lens of physics. To accomplish this, researchers apply a variety of modeling approaches to explore the evolution of form and function ranging from basic lever models to intricate computer simulations. While advances in technology have allowed for increasing model complexity, insight can still be gained through the use of low-parameter “simple” models. All models, regardless of complexity, are simplifications of reality and must make assumptions; “simple” models just make more assumptions than complex ones. However, “simple” models have several advantages. They allow individual parameters to be isolated and tested systematically, can be made applicable to a wide range of organisms and make good starting points for comparative studies, allowing for complexity to be added as needed. To illustrate these ideas, we perform a case study on body form and center of mass stability in ants. Ants show a wide diversity of body forms, particularly in terms of the relative size of the head, petiole(s), and gaster (the latter two make-up the segments of the abdomen not fused to thorax in hymenopterans). We use a “simple” model to explore whether balance issues pertaining to the center of mass influence patterns of segment expansion across major ant clades. Results from phylogenetic comparative methods imply that the location of the center of mass in an ant’s body is under stabilizing selection, constraining the center of mass to the middle segment (thorax) over the legs. This is potentially maintained by correlated rates of evolution between the head and gaster on either end. While these patterns arise from a model that makes several assumptions/simplifications relating to shape and materials, they still offer intriguing insights into the body plan of ants across ∼68% of their diversity. The results from our case study illustrate how “simple,” low-parameter models both highlight fundamental biomechanical trends and aid in crystalizing specific questions and hypotheses for more complex models to address.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
