skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Principles in experimental design for evaluating genomic forecasts
Abstract Over the past decade, there has been a rapid increase in the development of predictive models at the intersection of molecular ecology, genomics, and global change. The common goal of these ‘genomic forecasting’ models is to integrate genomic data with environmental and ecological data in a model to make quantitative predictions about the vulnerability of populations to climate change.Despite rapid methodological development and the growing number of systems in which genomic forecasts are made, the forecasts themselves are rarely evaluated in a rigorous manner with ground‐truth experiments. This study reviews the evaluation experiments that have been done, introduces important terminology regarding the evaluation of genomic forecasting models, and discusses important elements in the design and reporting of ground‐truth experiments.To date, experimental evaluations of genomic forecasts have found high variation in the accuracy of forecasts, but it is difficult to compare studies on a common ground due to different approaches and experimental designs. Additionally, some evaluations may be biased toward higher performance because training data and testing data are not independent. In addition to independence between training data and testing data, important elements in the design of an evaluation experiment include the construction and parameterization of the forecasting model, the choice of fitness proxies to measure for test data, the construction of the evaluation model, the choice of evaluation metric(s), the degree of extrapolation to novel environments or genotypes, and the sensitivity, uncertainty and reproducbility of forecasts.Although genomic forecasting methods are becoming more accessible, evaluating their limitations in a particular study system requires careful planning and experimentation. Meticulously designed evaluation experiments can clarify the robustness of the forecasts for application in management. Clear reporting of basic elements of experimental design will improve the rigour of evaluations, and in turn our understanding of why models work in some cases and not others.  more » « less
Award ID(s):
2043905
PAR ID:
10582472
Author(s) / Creator(s):
Publisher / Repository:
Methods in Ecology and Evolution
Date Published:
Journal Name:
Methods in Ecology and Evolution
Volume:
15
Issue:
9
ISSN:
2041-210X
Page Range / eLocation ID:
1466 to 1482
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; ; ; ; et al (, Methods in Ecology and Evolution)
    Abstract Ecological forecasting provides a powerful set of methods for predicting short‐ and long‐term change in living systems. Forecasts are now widely produced, enabling proactive management for many applied ecological problems. However, despite numerous calls for an increased emphasis on prediction in ecology, the potential for forecasting to accelerate ecological theory development remains underrealized.Here, we provide a conceptual framework describing how ecological forecasts can energize and advance ecological theory. We emphasize the many opportunities for future progress in this area through increased forecast development, comparison and synthesis.Our framework describes how a forecasting approach can shed new light on existing ecological theories while also allowing researchers to address novel questions. Through rigorous and repeated testing of hypotheses, forecasting can help to refine theories and understand their generality across systems. Meanwhile, synthesizing across forecasts allows for the development of novel theory about the relative predictability of ecological variables across forecast horizons and scales.We envision a future where forecasting is integrated as part of the toolset used in fundamental ecology. By outlining the relevance of forecasting methods to ecological theory, we aim to decrease barriers to entry and broaden the community of researchers using forecasting for fundamental ecological insight. 
    more » « less
  2. ; ; ; (, Developmental Science)
    Abstract The ability to engage in counterfactual thinking (reason about what elsecouldhave happened) is critical to learning, agency, and social evaluation. However, not much is known about how individual differences in counterfactual reasoning may play a role in children's social evaluations. In the current study, we investigate how prompting children to engage in counterfactual thinking about positive moral actions impacts children's social evaluations. Eighty‐seven 4‐8‐year‐olds were introduced to a character who engaged in a positive moral action (shared a sticker with a friend) and asked about whatelsethe character could have done with the sticker (counterfactual simulation). Children were asked to generate either a high number of counterfactuals (five alternative actions) or a low number of counterfactuals (one alternative action). Children were then asked a series of social evaluation questions contrasting that character with one who did not have a choice and had no alternatives (was told to give away the sticker to his friend). Results show that children who generatedselfishcounterfactuals were more likely to positively evaluate the character with choice than children who did not generate selfish counterfactuals, suggesting that generating counterfactuals most distant from the chosen action (prosociality) leads children to view prosocial actions more positively. We also found age‐related changes: as children got older, regardless of the type of counterfactuals generated, they were more likely to evaluate the character with choice more positively. These results highlight the importance of counterfactual reasoning in the development of moral evaluations. Research HighlightsOlder children were more likely to endorse agents whochooseto share over those who do not have a choice.Children who were prompted to generate more counterfactuals were more likely to allocate resources to characters with choice.Children who generated selfish counterfactuals more positively evaluated agents with choice.Comparable to theories suggesting children punish willful transgressors more than accidental transgressors, we propose children also consider free will when making positive moral evaluations. 
    more » « less
  3. ; ; ; (, New Phytologist)
    Summary Spatiotemporal patterns ofSpartina alterniflorabelowground biomass (BGB) are important for evaluating salt marsh resiliency. To solve this, we created the BERM (Belowground Ecosystem Resiliency Model), which estimates monthly BGB (30‐m spatial resolution) from freely available data such as Landsat‐8 and Daymet climate summaries.Our modeling framework relied on extreme gradient boosting, and used field observations from four Georgia salt marshes as ground‐truth data. Model predictors included estimated tidal inundation, elevation, leaf area index, foliar nitrogen, chlorophyll, surface temperature, phenology, and climate data. The final model included 33 variables, and the most important variables were elevation, vapor pressure from the previous four months, Normalized Difference Vegetation Index (NDVI) from the previous five months, and inundation.Root mean squared error for BGB from testing data was 313 g m−2(11% of the field data range), explained variance (R2) was 0.62–0.77. Testing data results were unbiased across BGB values and were positively correlated with ground‐truth data across all sites and years (r = 0.56–0.82 and 0.45–0.95, respectively).BERM can estimate BGB withinSpartina alterniflorasalt marshes where environmental parameters are within the training data range, and can be readily extended through a reproducible workflow. This provides a powerful approach for evaluating spatiotemporal BGB and associated ecosystem function. 
    more » « less
  4. ; ; (, Methods in Ecology and Evolution)
    Abstract Drones have emerged as a cost‐effective solution to detect and map plant invasions, offering researchers and land managers flexibility in flight design, sensors and data collection schedules. A systematic review of trends in drone‐based image collection, data processing and analytical approaches is needed to advance the science of invasive species monitoring and management and improve scalability and replicability.We systematically reviewed studies using drones for plant invasion research to identify knowledge gaps, best practices and a path toward advancing the science of invasive plant monitoring and management. We devised a database of 33 standardized reporting parameters, coded each study to those parameters, calculated descriptive statistics and synthesized how these technologies are being implemented and used.Trends show a general increase in studies since 2009 with a bias toward temperate regions in North America and Europe. Most studies have focused on testing the validity of a machine learning or deep learning image classification technique with fewer studies focused on monitoring or modelling spread. Very few studies used drones for assessing ecosystem dynamics and impacts such as determining environmental drivers or tracking re‐emergence after disturbance. Overall, we noted a lack of standardized reporting on field survey design, flight design, drone systems, image processing and analyses, which hinders replicability and scalability of approaches. Based on these findings, we develop a standard framework for drone applications in invasive species monitoring to foster cross‐study comparability and reproducibility.We suggest several areas for advancing the use of drones in invasive plant studies including (1) utilizing standardized reporting frameworks to facilitate scientific research practices, (2) integrating drone data with satellite imagery to scale up relationships over larger areas, (3) using drones as an alternative to in‐person ground surveys and (4) leveraging drones to assess community trait shifts tied to plant fitness and reproduction. 
    more » « less
  5. ; ; ; ; ; ; ; (, Methods in Ecology and Evolution)
    Abstract Quantitative evaluations to optimize complexity have become standard for avoiding overfitting of ecological niche models (ENMs) that estimate species’ potential geographic distributions.ENMevalwas the first R package to make such evaluations (often termed model tuning) widely accessible for the Maxent algorithm. It also provided multiple methods for partitioning occurrence data and reported various performance metrics.Requests by users, recent developments in the field, and needs for software compatibility led to a major redesign and expansion. We additionally conducted a literature review to investigate trends inENMevaluse (2015–2019).ENMeval2.0 has a new object‐oriented structure for adding other algorithms, enables customizing algorithmic settings and performance metrics, generates extensive metadata, implements a null‐model approach to quantify significance and effect sizes, and includes features to increase the breadth of analyses and visualizations. In our literature review, we found insufficient reporting of model performance and parameterization, heavy reliance on model selection with AICc and low utilization of spatial cross‐validation; we explain howENMeval2.0 can help address these issues.This redesigned and expanded version can promote progress in the field and improve the information available for decision‐making. ​ 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email