An official website of the United States government
This project developed a pre-interview survey, interview protocols, and materials for conducting interviews with expert users to better understand how they assess and make use decisions about new AI/ML guidance. Weather forecasters access and synthesize myriad sources of information when forecasting for high-impact, severe weather events. In recent years, artificial intelligence (AI) techniques have increasingly been used to produce new guidance tools with the goal of aiding weather forecasting, including for severe weather. For this study, we leveraged these advances to explore how National Weather Service (NWS) forecasters perceive the use of new AI guidance for forecasting severe hail and storm mode. We also specifically examine which guidance features are important for how forecasters assess the trustworthiness of new AI guidance. To this aim, we conducted online, structured interviews with NWS forecasters from across the Eastern, Central, and Southern Regions. The interviews covered the forecasters’ approaches and challenges for forecasting severe weather, perceptions of AI and its use in forecasting, and reactions to one of two experimental (i.e., non-operational) AI severe weather guidance: probability of severe hail or probability of storm mode. During the interview, the forecasters went through a self-guided review of different sets of information about the development (spin-up information, AI model technique, training of AI model, input information) and performance (verification metrics, interactive output, output comparison to operational guidance) of the presented guidance. The forecasters then assessed how the information influenced their perception of how trustworthy the guidance was and whether or not they would consider using it for forecasting. This project includes the pre-interview survey, survey data, interview protocols, and accompanying information boards used for the interviews. There is one set of interview materials in which AI/ML are mentioned throughout and another set where AI/ML were only mentioned at the end of the interviews. We did this to better understand how the label “AI/ML” did or did not affect how interviewees responded to interview questions and reviewed the information board. We also leverage think aloud methods with the information board, the instructions for which are included in the interview protocols.
more »
« less
The Experimental Warning Program of NOAA’s Hazardous Weather Testbed
Abstract NOAA’s Hazardous Weather Testbed (HWT) is a physical space and research framework to foster collaboration and evaluate emerging tools, technology, and products for NWS operations. The HWT’s Experimental Warning Program (EWP) focuses on research, technology, and communication that may improve severe and hazardous weather warnings and societal response. The EWP was established with three fundamental hypotheses: 1) collaboration with operational meteorologists increases the speed of the transition process and rate of adoption of beneficial applications and technology, 2) the transition of knowledge between research and operations benefits both the research and operational communities, and 3) including end users in experiments generates outcomes that are more reliable and useful for society. The EWP is designed to mimic the operations of any NWS Forecast Office, providing the opportunity for experiments to leverage live and archived severe weather activity anywhere in the United States. During the first decade of activity in the EWP, 15 experiments covered topics including new radar and satellite applications, storm-scale numerical models and data assimilation, total lightning use in severe weather forecasting, and multiple social science and end-user topics. The experiments range from exploratory and conceptual research to more controlled experimental design to establish statistical patterns and causal relationships. The EWP brought more than 400 NWS forecasters, 60 emergency managers, and 30 broadcast meteorologists to the HWT to participate in live demonstrations, archive events, and data-denial experiments influencing today’s operational warning environment and shaping the future of warning research, technology, and communication for years to come.
more »
« less
- Award ID(s):
- 1901712
- PAR ID:
- 10319166
- Date Published:
- Journal Name:
- Bulletin of the American Meteorological Society
- Volume:
- 102
- Issue:
- 12
- ISSN:
- 0003-0007
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This project developed a pre-interview survey, interview protocols, and materials for conducting interviews with expert users to better understand how they assess and make use decisions about new AI/ML guidance. Weather forecasters access and synthesize myriad sources of information when forecasting for high-impact, severe weather events. In recent years, artificial intelligence (AI) techniques have increasingly been used to produce new guidance tools with the goal of aiding weather forecasting, including for severe weather. For this study, we leveraged these advances to explore how National Weather Service (NWS) forecasters perceive the use of new AI guidance for forecasting severe hail and storm mode. We also specifically examine which guidance features are important for how forecasters assess the trustworthiness of new AI guidance. To this aim, we conducted online, structured interviews with NWS forecasters from across the Eastern, Central, and Southern Regions. The interviews covered the forecasters’ approaches and challenges for forecasting severe weather, perceptions of AI and its use in forecasting, and reactions to one of two experimental (i.e., non-operational) AI severe weather guidance: probability of severe hail or probability of storm mode. During the interview, the forecasters went through a self-guided review of different sets of information about the development (spin-up information, AI model technique, training of AI model, input information) and performance (verification metrics, interactive output, output comparison to operational guidance) of the presented guidance. The forecasters then assessed how the information influenced their perception of how trustworthy the guidance was and whether or not they would consider using it for forecasting. This project includes the pre-interview survey, survey data, interview protocols, and accompanying information boards used for the interviews. There is one set of interview materials in which AI/ML are mentioned throughout and another set where AI/ML were only mentioned at the end of the interviews. We did this to better understand how the label “AI/ML” did or did not affect how interviewees responded to interview questions and reviewed the information board. We also leverage think aloud methods with the information board, the instructions for which are included in the interview protocols.more » « less
-
This project developed a pre-interview survey, interview protocols, and materials for conducting interviews with expert users to better understand how they assess and make use decisions about new AI/ML guidance. Weather forecasters access and synthesize myriad sources of information when forecasting for high-impact, severe weather events. In recent years, artificial intelligence (AI) techniques have increasingly been used to produce new guidance tools with the goal of aiding weather forecasting, including for severe weather. For this study, we leveraged these advances to explore how National Weather Service (NWS) forecasters perceive the use of new AI guidance for forecasting severe hail and storm mode. We also specifically examine which guidance features are important for how forecasters assess the trustworthiness of new AI guidance. To this aim, we conducted online, structured interviews with NWS forecasters from across the Eastern, Central, and Southern Regions. The interviews covered the forecasters’ approaches and challenges for forecasting severe weather, perceptions of AI and its use in forecasting, and reactions to one of two experimental (i.e., non-operational) AI severe weather guidance: probability of severe hail or probability of storm mode. During the interview, the forecasters went through a self-guided review of different sets of information about the development (spin-up information, AI model technique, training of AI model, input information) and performance (verification metrics, interactive output, output comparison to operational guidance) of the presented guidance. The forecasters then assessed how the information influenced their perception of how trustworthy the guidance was and whether or not they would consider using it for forecasting. This project includes the pre-interview survey, survey data, interview protocols, and accompanying information boards used for the interviews. There is one set of interview materials in which AI/ML are mentioned throughout and another set where AI/ML were only mentioned at the end of the interviews. We did this to better understand how the label “AI/ML” did or did not affect how interviewees responded to interview questions and reviewed the information board. We also leverage think aloud methods with the information board, the instructions for which are included in the interview protocols.more » « less
-
Abstract Artificial Intelligence applications are rapidly expanding across weather, climate, and natural hazards. AI can be used to assist with forecasting weather and climate risks, including forecasting both the chance that a hazard will occur and the negative impacts from it, which means AI can help protect lives, property, and livelihoods on a global scale in our changing climate. To ensure that we are achieving this goal, the AI must be developed to be trustworthy, which is a complex and multifaceted undertaking. We present our work from the NSF AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography (AI2ES), where we are taking a convergence research approach. Our work deeply integrates across AI, environmental, and risk communication sciences. This involves collaboration with professional end-users to investigate how they assess the trustworthiness and usefulness of AI methods for forecasting natural hazards. In turn, we use this knowledge to develop AI that is more trustworthy. We discuss how and why end-users may trust or distrust AI methods for multiple natural hazards, including winter weather, tropical cyclones, severe storms, and coastal oceanography.more » « less
-
Abstract Severe storms produce hazardous weather phenomena, such as large hail, damaging winds, and tornadoes. However, relationships between convective parameters and confirmed severe weather occurrences are poorly quantified in south-central Brazil. This study explores severe weather reports and measurements from newly available datasets. Hail, damaging wind, and tornado reports are sourced from the PREVOTS project from June 2018 to December 2021, while measurements of convectively induced wind gusts from 1996 to 2019 are obtained from METAR reports and from Brazil’s operational network of automated weather stations. Proximal convective parameters were computed from ERA5 reanalysis for these reports and used to perform a discriminant analysis using mixed-layer CAPE and deep-layer shear (DLS). Compared to other regions, thermodynamic parameters associated with severe weather episodes exhibit lower magnitudes in south-central Brazil. DLS displays better performance in distinguishing different types of hazardous weather, but does not discriminate well between distinct severity levels. To address the sensitivity of the discriminant analysis to distinct environmental regimes and hazard types, five different discriminants are assessed. These include discriminants for any severe storm, severe hail only, severe wind gust only, and all environments but broken into “high” and “low” CAPE regimes. The best performance of the discriminant analysis is found for the “high” CAPE regime, followed by the severe wind regime. All discriminants demonstrate that DLS plays a more important role in conditioning Brazilian severe storm environments than other regions, confirming the need to ensure that parameters and discriminants are tuned to local severe weather conditions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1175/BAMS-D-21-0017.1
