Search for: All records

Abstract As an increasing number of machine learning (ML) products enter the research-to-operations (R2O) pipeline, researchers have anecdotally noted a perceived hesitancy by operational forecasters to adopt this relatively new technology. One explanation often cited in the literature is that this perceived hesitancy derives from the complex and opaque nature of ML methods. Because modern ML models are trained to solve tasks by optimizing a potentially complex combination of mathematical weights, thresholds, and nonlinear cost functions, it can be difficult to determine how these models reach a solution from their given input. However, it remains unclear to what degree a model’s transparency may influence a forecaster’s decision to use that model or if that impact differs between ML and more traditional (i.e., non-ML) methods. To address this question, a survey was offered to forecaster and researcher participants attending the 2021 NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) with questions about how participants subjectively perceive and compare machine learning products to more traditionally derived products. Results from this study revealed few differences in how participants evaluated machine learning products compared to other types of guidance. However, comparing the responses between operational forecasters, researchers, and academics exposed notable differences in what factors the three groups considered to be most important for determining the operational success of a new forecast product. These results support the need for increased collaboration between the operational and research communities. Significance StatementParticipants of the 2021 Hazardous Weather Testbed Spring Forecasting Experiment were surveyed to assess how machine learning products are perceived and evaluated in operational settings. The results revealed little difference in how machine learning products are evaluated compared to more traditional methods but emphasized the need for explainable product behavior and comprehensive end-user training. 

Abstract Artificial intelligence and machine learning (AI/ML) have attracted a great deal of attention from the atmospheric science community. The explosion of attention on AI/ML development carries implications for the operational community, prompting questions about how novel AI/ML advancements will translate from research into operations. However, the field lacks empirical evidence on how National Weather Service (NWS) forecasters, as key intended users, perceive AI/ML and its use in operational forecasting. This study addresses this crucial gap through structured interviews conducted with 29 NWS forecasters from October 2021 through July 2023 in which we explored their perceptions of AI/ML in forecasting. We found that forecasters generally prefer the term “machine learning” over “artificial intelligence” and that labeling a product as being AI/ML did not hurt perceptions of the products and made some forecasters more excited about the product. Forecasters also had a wide range of familiarity with AI/ML, and overall, they were (tentatively) open to the use of AI/ML in forecasting. We also provide examples of specific areas related to AI/ML that forecasters are excited or hopeful about and that they are concerned or worried about. One concern that was raised in several ways was that AI/ML could replace forecasters or remove them from the forecasting process. However, forecasters expressed a widespread and deep commitment to the best possible forecasts and services to uphold the agency mission using whatever tools or products that are available to assist them. Last, we note how forecasters’ perceptions evolved over the course of the study. 

Abstract Artificial intelligence (AI) and machine learning (ML) pose a challenge for achieving science that is both reproducible and replicable. The challenge is compounded in supervised models that depend on manually labeled training data, as they introduce additional decision‐making and processes that require thorough documentation and reporting. We address these limitations by providing an approach to hand labeling training data for supervised ML that integrates quantitative content analysis (QCA)—a method from social science research. The QCA approach provides a rigorous and well‐documented hand labeling procedure to improve the replicability and reproducibility of supervised ML applications in Earth systems science (ESS), as well as the ability to evaluate them. Specifically, the approach requires (a) the articulation and documentation of the exact decision‐making process used for assigning hand labels in a “codebook” and (b) an empirical evaluation of the reliability” of the hand labelers. In this paper, we outline the contributions of QCA to the field, along with an overview of the general approach. We then provide a case study to further demonstrate how this framework has and can be applied when developing supervised ML models for applications in ESS. With this approach, we provide an actionable path forward for addressing ethical considerations and goals outlined by recent AGU work on ML ethics in ESS. 

Abstract Demands to manage the risks of artificial intelligence (AI) are growing. These demands and the government standards arising from them both call for trustworthy AI. In response, we adopt a convergent approach to review, evaluate, and synthesize research on the trust and trustworthiness of AI in the environmental sciences and propose a research agenda. Evidential and conceptual histories of research on trust and trustworthiness reveal persisting ambiguities and measurement shortcomings related to inconsistent attention to the contextual and social dependencies and dynamics of trust. Potentially underappreciated in the development of trustworthy AI for environmental sciences is the importance of engaging AI users and other stakeholders, which human–AI teaming perspectives on AI development similarly underscore. Co‐development strategies may also help reconcile efforts to develop performance‐based trustworthiness standards with dynamic and contextual notions of trust. We illustrate the importance of these themes with applied examples and show how insights from research on trust and the communication of risk and uncertainty can help advance the understanding of trust and trustworthiness of AI in the environmental sciences. 

Abstract We introduce the National Science Foundation (NSF) AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography (AI2ES). This AI institute was funded in 2020 as part of a new initiative from the NSF to advance foundational AI research across a wide variety of domains. To date AI2ES is the only NSF AI institute focusing on environmental science applications. Our institute focuses on developing trustworthy AI methods for weather, climate, and coastal hazards. The AI methods will revolutionize our understanding and prediction of high-impact atmospheric and ocean science phenomena and will be utilized by diverse, professional user groups to reduce risks to society. In addition, we are creating novel educational paths, including a new degree program at a community college serving underrepresented minorities, to improve workforce diversity for both AI and environmental science. 

Abstract The NSF AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography (AI2ES) focuses on creating trustworthy AI for a variety of environmental and Earth science phenomena. AI2ES includes leading experts from AI, atmospheric and ocean science, risk communication, and education, who work synergistically to develop and test trustworthy AI methods that transform our understanding and prediction of the environment. Trust is a social phenomenon, and our integration of risk communication research across AI2ES activities provides an empirical foundation for developing user‐informed, trustworthy AI. AI2ES also features activities to broaden participation and for workforce development that are fully integrated with AI2ES research on trustworthy AI, environmental science, and risk communication. 

Abstract Eastern Colorado is one of the most active hail regions in the United States, and individual hailstorms routinely surpass millions of dollars in crop loss and physical damage. Fifteen semistructured interviews with eastern Colorado farmers and ranchers were conducted in the summer of 2019 to gauge perceptions of the severity and vulnerability associated with hailstorms, as well as to understand how forecasts and warnings for severe hail are received and acted upon by the agricultural community. Results reveal a correspondence between perceived and observed frequency of hailstorms in eastern Colorado and highlight financial losses from crop destruction as the greatest threat from hailstorms. In contrast to the National Weather Service defining severe hail as at least 1.0 in. (25.4 mm) in diameter, the agricultural community conceptualizes hail severity according to impacts and damage. Small hail in large volumes or driven by a strong wind are the most worrisome scenarios for farmers, because small hail can most easily strip crop heads and stalks. Larger hailstones are perceived to pose less of a threat to crops but can produce significant damage to physical equipment and injure livestock. Eastern Colorado farmers and ranchers are avid weather watchers and associate environmental cues with hailstorms in addition to receiving warning messages, primarily via alerts on mobile telephones. Hailstorms elicit feelings of dejection and anxiety in some respondents, whereas others accept hailstorms as part of the job. Increasing awareness of the agricultural perceptions of hailstorms can help the meteorological community direct hail prediction research efforts and improve risk communication to the agricultural sector. 

Abstract During the last few decades, scientific capabilities for understanding and predicting weather and climate risks have advanced rapidly. At the same time, technological advances, such as the Internet, mobile devices, and social media, are transforming how people exchange and interact with information. In this modern information environment, risk communication, interpretation, and decision-making are rapidly evolving processes that intersect across space, time, and society. Instead of a linear or iterative process in which individual members of the public assess and respond to distinct pieces of weather forecast or warning information, this article conceives of weather prediction, communication, and decision-making as an interconnected dynamic system. In this expanded framework, information and uncertainty evolve in conjunction with people’s risk perceptions, vulnerabilities, and decisions as a hazardous weather threat approaches; these processes are intertwined with evolving social interactions in the physical and digital worlds. Along with the framework, the article presents two interdisciplinary research approaches for advancing the understanding of this complex system and the processes within it: analysis of social media streams and computational natural–human system modeling. Examples from ongoing research are used to demonstrate these approaches and illustrate the types of new insights they can reveal. This expanded perspective together with research approaches, such as those introduced, can help researchers and practitioners understand and improve the creation and communication of information in atmospheric science and other fields.