An official website of the United States government
Abstract Over the past decade, there has been a rapid increase in the development of predictive models at the intersection of molecular ecology, genomics, and global change. The common goal of these ‘genomic forecasting’ models is to integrate genomic data with environmental and ecological data in a model to make quantitative predictions about the vulnerability of populations to climate change.Despite rapid methodological development and the growing number of systems in which genomic forecasts are made, the forecasts themselves are rarely evaluated in a rigorous manner with ground‐truth experiments. This study reviews the evaluation experiments that have been done, introduces important terminology regarding the evaluation of genomic forecasting models, and discusses important elements in the design and reporting of ground‐truth experiments.To date, experimental evaluations of genomic forecasts have found high variation in the accuracy of forecasts, but it is difficult to compare studies on a common ground due to different approaches and experimental designs. Additionally, some evaluations may be biased toward higher performance because training data and testing data are not independent. In addition to independence between training data and testing data, important elements in the design of an evaluation experiment include the construction and parameterization of the forecasting model, the choice of fitness proxies to measure for test data, the construction of the evaluation model, the choice of evaluation metric(s), the degree of extrapolation to novel environments or genotypes, and the sensitivity, uncertainty and reproducbility of forecasts.Although genomic forecasting methods are becoming more accessible, evaluating their limitations in a particular study system requires careful planning and experimentation. Meticulously designed evaluation experiments can clarify the robustness of the forecasts for application in management. Clear reporting of basic elements of experimental design will improve the rigour of evaluations, and in turn our understanding of why models work in some cases and not others.
more »
« less
The power of forecasts to advance ecological theory
Abstract Ecological forecasting provides a powerful set of methods for predicting short‐ and long‐term change in living systems. Forecasts are now widely produced, enabling proactive management for many applied ecological problems. However, despite numerous calls for an increased emphasis on prediction in ecology, the potential for forecasting to accelerate ecological theory development remains underrealized.Here, we provide a conceptual framework describing how ecological forecasts can energize and advance ecological theory. We emphasize the many opportunities for future progress in this area through increased forecast development, comparison and synthesis.Our framework describes how a forecasting approach can shed new light on existing ecological theories while also allowing researchers to address novel questions. Through rigorous and repeated testing of hypotheses, forecasting can help to refine theories and understand their generality across systems. Meanwhile, synthesizing across forecasts allows for the development of novel theory about the relative predictability of ecological variables across forecast horizons and scales.We envision a future where forecasting is integrated as part of the toolset used in fundamental ecology. By outlining the relevance of forecasting methods to ecological theory, we aim to decrease barriers to entry and broaden the community of researchers using forecasting for fundamental ecological insight.
more »
« less
- PAR ID:
- 10369439
- Publisher / Repository:
- Wiley-Blackwell
- Date Published:
- Journal Name:
- Methods in Ecology and Evolution
- Volume:
- 14
- Issue:
- 3
- ISSN:
- 2041-210X
- Page Range / eLocation ID:
- p. 746-756
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Communicating and interpreting uncertainty in ecological model predictions is notoriously challenging, motivating the need for new educational tools, which introduce ecology students to core concepts in uncertainty communication. Ecological forecasting, an emerging approach to estimate future states of ecological systems with uncertainty, provides a relevant and engaging framework for introducing uncertainty communication to undergraduate students, as forecasts can be used as decision support tools for addressing real‐world ecological problems and are inherently uncertain. To provide critical training on uncertainty communication and introduce undergraduate students to the use of ecological forecasts for guiding decision‐making, we developed a hands‐on teaching module within the Macrosystems Environmental Data‐Driven Inquiry and Exploration (EDDIE;MacrosystemsEDDIE.org) educational program. Our module used an active learning approach by embedding forecasting activities in an R Shiny application to engage ecology students in introductory data science, ecological modeling, and forecasting concepts without needing advanced computational or programming skills. Pre‐ and post‐module assessment data from more than 250 undergraduate students enrolled in ecology, freshwater ecology, and zoology courses indicate that the module significantly increased students' ability to interpret forecast visualizations with uncertainty, identify different ways to communicate forecast uncertainty for diverse users, and correctly define ecological forecasting terms. Specifically, students were more likely to describe visual, numeric, and probabilistic methods of uncertainty communication following module completion. Students were also able to identify more benefits of ecological forecasting following module completion, with the key benefits of using forecasts for prediction and decision‐making most commonly described. These results show promise for introducing ecological model uncertainty, data visualizations, and forecasting into undergraduate ecology curricula via software‐based learning, which can increase students' ability to engage and understand complex ecological concepts.more » « less
-
Abstract Near‐term iterative forecasting is a powerful tool for ecological decision support and has the potential to transform our understanding of ecological predictability. However, to this point, there has been no cross‐ecosystem analysis of near‐term ecological forecasts, making it difficult to synthesize diverse research efforts and prioritize future developments for this emerging field. In this study, we analyzed 178 near‐term (≤10‐yr forecast horizon) ecological forecasting papers to understand the development and current state of near‐term ecological forecasting literature and to compare forecast accuracy across scales and variables. Our results indicated that near‐term ecological forecasting is widespread and growing: forecasts have been produced for sites on all seven continents and the rate of forecast publication is increasing over time. As forecast production has accelerated, some best practices have been proposed and application of these best practices is increasing. In particular, data publication, forecast archiving, and workflow automation have all increased significantly over time. However, adoption of proposed best practices remains low overall: for example, despite the fact that uncertainty is often cited as an essential component of an ecological forecast, only 45% of papers included uncertainty in their forecast outputs. As the use of these proposed best practices increases, near‐term ecological forecasting has the potential to make significant contributions to our understanding of forecastability across scales and variables. In this study, we found that forecastability (defined here as realized forecast accuracy) decreased in predictable patterns over 1–7 d forecast horizons. Variables that were closely related (i.e., chlorophyll and phytoplankton) displayed very similar trends in forecastability, while more distantly related variables (i.e., pollen and evapotranspiration) exhibited significantly different patterns. Increasing use of proposed best practices in ecological forecasting will allow us to examine the forecastability of additional variables and timescales in the future, providing a robust analysis of the fundamental predictability of ecological variables.more » « less
-
Abstract Ecosystem models offer a rigorous way to formalize scientific theories and are critical to evaluating complex interactions among ecological and biogeochemical processes. In addition to simulation and prediction, ecosystem models are a valuable tool for testing hypotheses about mechanisms and empirical findings because they reveal critical internal processes that are difficult to observe directly.However, many ecosystem models are difficult to manage and apply by scientists who lack advanced computing skills due to complex model structures, lack of consistent documentation, and low-level programming implementation, which facilitates computing but reduces accessibility.Here, we present the ‘pnetr’ R package, which is designed to provide an easy-to-manage ecosystem modeling framework and detailed documentation in both model structure and programming. The framework implements a family of widely used PnET (net photosynthesis, evapotranspiration) ecosystem models, which are relatively parsimonious but capture essential biogeochemical cycles of water, carbon, and nutrients. We chose the R programming language since it is familiar to many ecologists and has abundant statistical modeling resources. We showcase examples of model simulations and test the effects of phenology on carbon assimilation and wood production using data measured by the Environmental Measurement Station (EMS) eddy-covariance flux tower at Harvard Forest, MA.We hope ‘pnetr’ can facilitate further development of ecological theory and increase the accessibility of ecosystem modeling and ecological forecasting.more » « less
-
Abstract Ecosystems around the globe are experiencing changes in both the magnitude and fluctuations of environmental conditions due to land use and climate change. In response, ecologists are increasingly using near‐term, iterative ecological forecasts to predict how ecosystems will change in the future. To date, many near‐term, iterative forecasting systems have been developed using high temporal frequency (minute to hourly resolution) data streams for assimilation. However, this approach may be cost‐prohibitive or impossible for forecasting ecological variables that lack high‐frequency sensors or have high data latency (i.e., a delay before data are available for modeling after collection). To explore the effects of data assimilation frequency on forecast skill, we developed water temperature forecasts for a eutrophic drinking water reservoir and conducted data assimilation experiments by selectively withholding observations to examine the effect of data availability on forecast accuracy. We used in situ sensors, manually collected data, and a calibrated water quality ecosystem model driven by forecasted weather data to generate future water temperature forecasts using Forecasting Lake and Reservoir Ecosystems (FLARE), an open source water quality forecasting system. We tested the effect of daily, weekly, fortnightly, and monthly data assimilation on the skill of 1‐ to 35‐day‐ahead water temperature forecasts. We found that forecast skill varied depending on the season, forecast horizon, depth, and data assimilation frequency, but overall forecast performance was high, with a mean 1‐day‐ahead forecast root mean square error (RMSE) of 0.81°C, mean 7‐day RMSE of 1.15°C, and mean 35‐day RMSE of 1.94°C. Aggregated across the year, daily data assimilation yielded the most skillful forecasts at 1‐ to 7‐day‐ahead horizons, but weekly data assimilation resulted in the most skillful forecasts at 8‐ to 35‐day‐ahead horizons. Within a year, forecasts with weekly data assimilation consistently outperformed forecasts with daily data assimilation after the 8‐day forecast horizon during mixed spring/autumn periods and 5‐ to 14‐day‐ahead horizons during the summer‐stratified period, depending on depth. Our results suggest that lower frequency data (i.e., weekly) may be adequate for developing accurate forecasts in some applications, further enabling the development of forecasts broadly across ecosystems and ecological variables without high‐frequency sensor data.more » « less
