An official website of the United States government
Abstract Pure artificial intelligence (AI)-based weather prediction (AIWP) models have made waves within the scientific community and the media, claiming superior performance to numerical weather prediction (NWP) models. However, these models often lack impactful output variables such as precipitation. One exception is Google DeepMind’s GraphCast model, which became the first mainstream AIWP model to predict precipitation, but performed only limited verification. We present an analysis of the ECMWF’s Integrated Forecasting System (IFS)-initialized (GRAPIFS) and the NCEP’s Global Forecast System (GFS)-initialized (GRAPGFS) GraphCast precipitation forecasts over the contiguous United States and compare to results from the GFS and IFS models using 1) grid-based, 2) neighborhood, and 3) object-oriented metrics verified against the fifth major global reanalysis produced by ECMWF (ERA5) and the NCEP/Environmental Modeling Center (EMC) stage IV precipitation analysis datasets. We affirmed that GRAPGFSand GRAPIFSperform better than the GFS and IFS in terms of root-mean-square error and stable equitable errors in probability space, but the GFS and IFS precipitation distributions more closely align with the ERA5 and stage IV distributions. Equitable threat score also generally favored GraphCast, particularly for lower accumulation thresholds. Fractions skill score for increasing neighborhood sizes shows greater gains for the GFS and IFS than GraphCast, suggesting the NWP models may have a better handle on intensity but struggle with the location. Object-based verification for GraphCast found positive area biases at low accumulation thresholds and large negative biases at high accumulation thresholds. GRAPGFSsaw similar performance gains to GRAPIFSwhen compared to their NWP counterparts, but initializing with the less familiar GFS conditions appeared to lead to an increase in light precipitation. Significance StatementPure artificial intelligence (AI)-based weather prediction (AIWP) has exploded in popularity with promises of better performance and faster run times than numerical weather prediction (NWP) models. However, less attention has been paid to their capability to predict impactful, sensible weather like precipitation, precipitation type, or specific meteorological features. We seek to address this gap by comparing precipitation forecast performance by an AI model called GraphCast to the Global Forecast System (GFS) and the Integrated Forecasting System (IFS) NWP models. While GraphCast does perform better on many verification metrics, it has some limitations for intense precipitation forecasts. In particular, it less frequently predicts intense precipitation events than the GFS or IFS. Overall, this article emphasizes the promise of AIWP while at the same time stresses the need for robust verification by domain experts.
more »
« less
Predictability Limit of the 2021 Pacific Northwest Heatwave From Deep‐Learning Sensitivity Analysis
Abstract The traditional method for estimating weather forecast sensitivity to initial conditions uses adjoint models, which are limited to short lead times due to linearization around a control forecast. The advent of deep‐learning frameworks enables a new approach using backpropagation and gradient descent to iteratively optimize initial conditions, minimizing forecast errors. We apply this approach to the June 2021 Pacific Northwest heatwave using the GraphCast model, yielding over 90% reduction in 10‐day forecast errors over the Pacific Northwest. Similar improvements are found for Pangu‐Weather model forecasts initialized with the GraphCast‐derived optimal, suggesting that model error is an unimportant part of the perturbations. Eliminating small scales from the perturbations also yields similar forecast improvements. Extending the length of the optimization window, we find forecast improvement to about 23 days, suggesting atmospheric predictability at the upper end of recent estimates.
more »
« less
- Award ID(s):
- 2202526
- PAR ID:
- 10556580
- Publisher / Repository:
- DOI PREFIX: 10.1029
- Date Published:
- Journal Name:
- Geophysical Research Letters
- Volume:
- 51
- Issue:
- 19
- ISSN:
- 0094-8276
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Weather forecasts made with imperfect models contain state‐dependent errors. Data assimilation (DA) partially corrects these errors with new information from observations. As such, the corrections, or “analysis increments,” produced by the DA process embed information about model errors. An attempt is made here to extract that information to improve numerical weather prediction. Neural networks (NNs) are trained to predict corrections to the systematic error in the National Oceanic and Atmospheric Administration's FV3‐GFS model based on a large set of analysis increments. A simple NN focusing on an atmospheric column significantly improves the estimated model error correction relative to a linear baseline. Leveraging large‐scale horizontal flow conditions using a convolutional NN, when compared to the simple column‐oriented NN, does not improve skill in correcting model error. The sensitivity of model error correction to forecast inputs is highly localized by vertical level and by meteorological variable, and the error characteristics vary across vertical levels. Once trained, the NNs are used to apply an online correction to the forecast during model integration. Improvements are evaluated both within a cycled DA system and across a collection of 10‐day forecasts. It is found that applying state‐dependent NN‐predicted corrections to the model forecast improves the overall quality of DA and improves the 10‐day forecast skill at all lead times.more » « less
-
Abstract Global deep learning weather prediction models have recently been shown to produce forecasts that rival those from physics-based models run at operational centers. It is unclear whether these models have encoded atmospheric dynamics or simply pattern matching that produces the smallest forecast error. Answering this question is crucial to establishing the utility of these models as tools for basic science. Here, we subject one such model, Pangu-Weather, to a set of four classical dynamical experiments that do not resemble the model training data. Localized perturbations to the model output and the initial conditions are added to steady time-averaged conditions, to assess the propagation speed and structural evolution of signals away from the local source. Perturbing the model physics by adding a steady tropical heat source results in a classical Matsuno–Gill response near the heating and planetary waves that radiate into the extratropics. A localized disturbance on the winter-averaged North Pacific jet stream produces realistic extratropical cyclones and fronts, including the spontaneous emergence of polar lows. Perturbing the 500-hPa height field alone yields adjustment from a state of rest to one of wind–pressure balance over ∼6 h. Localized subtropical low pressure systems produce Atlantic hurricanes, provided the initial amplitude exceeds about 4 hPa, and setting the initial humidity to zero eliminates hurricane development. We conclude that the model encodes realistic physics in all experiments and suggest that it can be used as a tool for rapidly testing a wide range of hypotheses.more » « less
-
Abstract The mean-state bias and the associated forecast errors of the El Niño–Southern Oscillation (ENSO) are investigated in a suite of 2-yr-lead retrospective forecasts conducted with the Community Earth System Model, version 1, for 1954–2015. The equatorial Pacific cold tongue in the forecasts is too strong and extends excessively westward due to a combination of the model’s inherent climatological bias, initialization imbalance, and errors in initial ocean data. The forecasts show a stronger cold tongue bias in the first year than that inherent to the model due to the imbalance between initial subsurface oceanic states and model dynamics. The cold tongue bias affects not only the pattern and amplitude but also the duration of ENSO in the forecasts by altering ocean–atmosphere feedbacks. The predicted sea surface temperature anomalies related to ENSO extend to the far western equatorial Pacific during boreal summer when the cold tongue bias is strong, and the predicted ENSO anomalies are too weak in the central-eastern equatorial Pacific. The forecast errors of pattern and amplitude subsequently lead to errors in ENSO phase transition by affecting the amplitude of the negative thermocline feedback in the equatorial Pacific and tropical interbasin adjustments during the mature phase of ENSO. These ENSO forecast errors further degrade the predictions of wintertime atmospheric teleconnections, land surface air temperature, and rainfall anomalies over the Northern Hemisphere. These mean-state and ENSO forecast biases are more pronounced in forecasts initialized in boreal spring–summer than other seasons due to the seasonal intensification of the Bjerknes feedback.more » « less
-
Abstract Numerous artificial intelligence-based weather prediction (AIWP) models have emerged over the past 2 years, mostly in the private sector. There is an urgent need to evaluate these models from a meteorological perspective, but access to the output of these models is limited. We detail two new resources to facilitate access to AIWP model output data in the hope of accelerating the investigation of AIWP models by the meteorological community. First, a 3-yr (and growing) reforecast archive beginning in October 2020 containing twice daily 10-day forecasts forFourCastNet v2-small,Pangu-Weather, andGraphCast Operationalis now available via an Amazon Simple Storage Service (S3) bucket through NOAA’s Open Data Dissemination (NODD) program (https://noaa-oar-mlwp-data.s3.amazonaws.com/index.html). This reforecast archive was initialized with both the NOAA’s Global Forecast System (GFS) and ECMWF’s Integrated Forecasting System (IFS) initial conditions in the hope that users can begin to perform the feature-based verification of impactful meteorological phenomena. Second, real-time output for these three models is visualized on our web page (https://aiweather.cira.colostate.edu) along with output from the GFS and the IFS. This allows users to easily compare output between each AIWP model and traditional, physics-based models with the goal of familiarizing users with the characteristics of AIWP models and determine whether the output aligns with expectations, is physically consistent and reasonable, and/or is trustworthy. We view these two efforts as a first step toward evaluating whether these new AIWP tools have a place in forecast operations.more » « less
