More Like this

1. Validity and Validation of Computer Simulations—A Methodological Inquiry with Application to Integrated Assessment Models

   https://doi.org/10.3390/knowledge3020018  

   Randall, Alan ; Ogland-Hand, Jonathan ( June 2023 , Knowledge)

   Our purpose is to advance a reasoned perspective on the scientific validity of computer simulation, using an example—integrated assessment modeling of climate change and its projected impacts—that is itself of great and urgent interest to policy in the real world. The spirited and continuing debate on the scientific status of integrated assessment models (IAMs) of global climate change has been conducted mostly among climate change modelers and users seeking guidance for climate policy. However, it raises a number and variety of issues that have been addressed, with various degrees of success, in other literature. The literature on methodology of simulation was mostly skeptical at the outset but has become more nuanced, casting light on some key issues relating to the validity and evidentiary standing of climate change IAMs (CC-IAMs). We argue that the goal of validation is credence, i.e., confidence or justified belief in model projections, and that validation is a matter of degree: (perfect) validity is best viewed as aspirational and, other things equal, it makes sense to seek more rather than less validation. We offer several conclusions. The literature on computer simulation has become less skeptical and more inclined to recognize that simulations are capable of providing evidence, albeit a different kind of evidence than, say, observation and experiments. CC-IAMs model an enormously complex system of systems and must respond to several challenges that include building more transparent models and addressing deep uncertainty credibly. Drawing on the contributions of philosophers of science and introspective practitioners, we offer guidance for enhancing the credibility of CC-IAMs and computer simulation more generally. 

   more » « less
2. Accelerating the Broad Implementation of Verification and Validation in Computational Models of the Mechanics of Materials and Structures

   The Minerals Metals & Materials Society, Inc. ( October 2020 , Accelerating the Broad Implementation of Verification and Validation in Computational Models of the Mechanics of Materials and Structures)

   This report is intended to provide value to scientists, engineers, software developers, designers, analysts, regulators, students, and other stakeholders associated with (or intending to work with) computational models related to the mechanics of materials and structures (MOMS). This includes both modelers and experimentalists within the materials science and engineering, mechanical engineering, solid mechanics, structural dynamics, and related communities, spanning academic, industrial, and government affiliation sectors. This report was written with two types of people in mind: novices who have little or no prior experience in robust verification and validation (V&V) and associated/inseparable uncertainty quantification (UQ) practices, and those who have some V&V/UQ experience, but want to establish more rigorous practices. More specifically, researchers, developers, and students associated with materials (both structural and soft materials) and solid mechanics modeling, who utilize advanced computation, materials data, and/or experimental validation tools, should find the information in this report especially useful. It is critical that the community widely adopts robust V&V/UQ practices in order to improve trust, reduce risk, and improve the reliability of MOMS computational models. Beyond practitioners in this field, other stakeholders who can influence the future of advanced computational modeling associated with MOMS should find this report useful, as well. This includes individuals who support financial and/ or time investments in science and technologies surrounding computational modeling, such as funding officers and other decision-makers at federal agencies, and leaders/managers in industry. Educators teaching undergraduate and graduate courses related to MOMS, as well as department heads and/or deans within the relevant disciplines, also could use the information in this report to advance associated curricula and enhance research products. 

   more » « less
3. Defining and Assessing Systems Thinking in Diverse Engineering Populations

   Mosyjowski, E.A. ; Daly, S.R. ; Lattuca, L. ( January 2019 , Proceedings of the American Society for Engineering Education Conference)

   Engineers are called to play an important role in addressing the complex problems of our global society, such as climate change and global health care. In order to adequately address these complex problems, engineers must be able to identify and incorporate into their decision making relevant aspects of systems in which their work is contextualized, a skill often referred to as systems thinking. However, within engineering, research on systems thinking tends to emphasize the ability to recognize potentially relevant constituent elements and parts of an engineering problem, rather than how these constituent elements and parts are embedded in broader economic, sociocultural, and temporal contexts and how all of these must inform decision making about problems and solutions. Additionally, some elements of systems thinking, such as an awareness of a particular sociocultural context or the coordination of work among members of a cross-disciplinary team, are not always recognized as core engineering skills, which alienates those whose strengths and passions are related to, for example, engineering systems that consider and impact social change. Studies show that women and minorities, groups underrepresented within engineering, are drawn to engineering in part for its potential to address important social issues. Emphasizing the importance of systems thinking and developing a more comprehensive definition of systems thinking that includes both constituent parts and contextual elements of a system will help students recognize the relevance and value of these other elements of engineering work and support full participation in engineering by a diverse group of students. We provide an overview of our study, in which we are examining systems thinking across a range of expertise to develop a scenario-based assessment tool that educators and researchers can use to evaluate engineering students’ systems thinking competence. Consistent with the aforementioned need to define and study systems thinking in a comprehensive, inclusive manner, we begin with a definition of systems thinking as a holistic approach to problem solving in which linkages and interactions of the immediate work with constituent parts, the larger sociocultural context, and potential impacts over time are identified and incorporated into decision making. In our study, we seek to address two key questions: 1) How do engineers of different levels of education and experience approach problems that require systems thinking? and 2) How do different types of life, educational, and work experiences relate to individuals’ demonstrated level of expertise in solving systems thinking problems? Our study is comprised of three phases. The first two phases include a semi-structured interview with engineering students and professionals about their experiences solving a problem requiring systems thinking and a think-aloud interview in which participants are asked to talk through how they would approach a given engineering scenario and later reflect on the experiences that inform their thinking. Data from these two phases will be used to develop a written assessment tool, which we will test by administering the written instrument to undergraduate and graduate engineering students in our third study phase. Our paper describes our study design and framing and includes preliminary findings from the first phase of our study. 

   more » « less
4. Two kinds of polar knowledge

   https://doi.org/10.1080/10899995.2020.1838849  

   Hamilton, Lawrence C. ( November 2020 , Journal of Geoscience Education)

   null (Ed.)

   Outreach and communication with the public have substantial value in polar research, in which studies often find changes of global importance that are happening far out of sight from the majority of people living at lower latitudes. Seeking evidence on the effectiveness of outreach programs, the U.S. National Science Foundation sponsored large-scale survey assessments before and after the International Polar Year in 2007/2008. Polar-knowledge questions have subsequently been tested and refined through other nationwide and regional surveys. More than a decade of such work has established that basic but fairly specific knowledge questions, with all answer choices sounding plausible but one being uniquely correct, can yield highly replicable results. Those results, however, paint a mixed picture of knowledge. Some factual questions seem to be interpreted by many respondents as if they had been asked for their personal beliefs about climate change, so their responses reflect sociopolitical identity rather than physical-world knowledge. Other factual questions, by design, do not link in obvious ways to climate-change beliefs—so responses have simpler interpretations in terms of knowledge gaps, and education needs. 

   more » « less
5. An interdisciplinary framework for evaluating 19th century landscape paintings for ecological research

   https://doi.org/10.1002/ecs2.4649  

   Warren, Dana R. ; Loeb, Harper M. ; Betjemann, Peter ; Munck, Isabel A. ; Keeton, William S. ; Shaw, David C. ; Harvey, Eleanor J. ( September 2023 , Ecosphere)

   Abstract As we contemplate the future of forest landscapes under changing climate conditions and land‐use demands, there is increasing value in studying historic forest conditions and how these landscapes have changed following past disturbances. Historic landscape paintings are a potential source of data on preindustrial forests with highly detailed, full‐color depictions of overstory and understory environments. They display key details about forest community composition, microhabitat features, and structural complexity from a time well before the advent of color photography. Despite these paintings' potential, their scientific applications have been impeded by questions of validity. How truly accurate are the images portrayed in these paintings? How much of an image is an artist's manipulation of a scene to best illustrate an allegory or romanticized view of nature? Following an established assessment model from historical ecology for evaluating resource validity, we demonstrate how scholarship on art history can be integrated with ecological understanding of forest landscapes to follow this model and address these questions of image veracity in 19th century American art. Further, to illustrate the potential use of these historic images in ecological studies, we present in a case study assessing microhabitat features of 10 different paintings. While this paper explores 19th century landscape art broadly, we focus our art historical review in particular on Asher Durand, a prolific and influential artist associated with the so‐called “Hudson River School” in the mid‐1800s. Durand left clear records about his perspectives on accurately depicting nature, and from a review of images and writings of Durand, we find support for the potential use of many of his paintings and sketches in historic forest ecology research. However, we also identify important caveats regarding potential ecological interpretations from these images. More broadly, because 19th century landscape paintings are not always directly transcriptive, and because regional art cultures differed in the 1800s, we cannot within this paper speak about landscape image veracity across all 19th century landscape art. However, in following established methods in historical ecology and integrating tools from art history research, we show that one can identify accurate historic landscape paintings for application in scientific studies. 

   more » « less