skip to main content


Title: Simulation Tests of Methods in Evolution, Ecology, and Systematics: Pitfalls, Progress, and Principles
Complex statistical methods are continuously developed across the fields of ecology, evolution, and systematics (EES). These fields, however, lack standardized principles for evaluating methods, which has led to high variability in the rigor with which methods are tested, a lack of clarity regarding their limitations, and the potential for misapplication. In this review, we illustrate the common pitfalls of method evaluations in EES, the advantages of testing methods with simulated data, and best practices for method evaluations. We highlight the difference between method evaluation and validation and review how simulations, when appropriately designed, can refine the domain in which a method can be reliably applied. We also discuss the strengths and limitations of different evaluation metrics. The potential for misapplication of methods would be greatly reduced if funding agencies, reviewers, and journals required principled method evaluation. Expected final online publication date for the Annual Review of Ecology, Evolution, and Systematics, Volume 53 is November 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.  more » « less
Award ID(s):
1655701 2043905 1655344
NSF-PAR ID:
10348276
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Annual Review of Ecology, Evolution, and Systematics
Volume:
53
Issue:
1
ISSN:
1543-592X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Artificial intelligence (AI) is an emerging technology that has great potential in reducing energy consumption, environmental burdens, and operational risks of chemical production. However, large-scale applications of AI are still limited. One barrier is the lack of quantitative understandings of the potential benefits and risks of different AI applications. This study reviewed relevant AI literature and categorized those case studies by application types, impact categories, and application modes. Most studies assessed the energy, economic, and safety implications of AI applications, while few of them have evaluated the environmental impacts of AI, given the large data gaps and difficulties in choosing appropriate assessment methods. Based on the reviewed case studies in the chemical industry, we proposed a conceptual framework that encompasses approaches from industrial ecology, economics, and engineering to guide the selection of performance indicators and evaluation methods for a holistic assessment of AI's impacts. This framework could be a valuable tool to support the decision-making related to AI in the fundamental research and practical production of chemicals. Although this study focuses on the chemical industry, the insights of the literature review and the proposed framework could be applied to AI applications in other industries and broad industrial ecology fields. In the end, this study highlights future research directions for addressing the data challenges in assessing AI's impacts and developing AI-enhanced tools to support the sustainable development of the chemical industry. 
    more » « less
  2. Research on floral volatiles has grown substantially in the last 20 years, which has generated insights into their diversity and prevalence. These studies have paved the way for new research that explores the evolutionary origins and ecological consequences of different types of variation in floral scent, including community-level, functional, and environmentally induced variation. However, to address these types of questions, novel approaches are needed that can handle large sample sizes, provide quality control measures, and make volatile research more transparent and accessible, particularly for scientists without prior experience in this field. Drawing upon a literature review and our own experiences, we present a set of best practices for next-generation research in floral scent. We outline methods for data collection (experimental designs, methods for conducting field collections, analytical chemistry, compound identification) and data analysis (statistical analysis, database integration) that will facilitate the generation and interpretation of quality data. For the intermediate step of data processing, we created the R package bouquet , which provides a data analysis pipeline. The package contains functions that enable users to convert chromatographic peak integrations to a filtered data table that can be used in subsequent statistical analyses. This package includes default settings for filtering out non-floral compounds, including background contamination, based on our best-practice guidelines, but functions and workflows can be easily customized as necessary. Next-generation research into the ecology and evolution of floral scent has the potential to generate broadly relevant insights into how complex traits evolve, their genomic architecture, and their consequences for ecological interactions. In order to fulfill this potential, the methodology of floral scent studies needs to become more transparent and reproducible. By outlining best practices throughout the lifecycle of a project, from experimental design to statistical analysis, and providing an R package that standardizes the data processing pipeline, we provide a resource for new and seasoned researchers in this field and in adjacent fields, where high-throughput and multi-dimensional datasets are common. 
    more » « less
  3. Signals of local adaptation have been found in many plants and animals, highlighting the heterogeneity in the distribution of adaptive genetic variation throughout species ranges. In the coming decades, global climate change is expected to induce shifts in the selective pressures that shape this adaptive variation. These changes in selective pressures will likely result in varying degrees of local climate maladaptation and spatial reshuffling of the underlying distributions of adaptive alleles. There is a growing interest in using population genomic data to help predict future disruptions to locally adaptive gene-environment associations. One motivation behind such work is to better understand how the effects of changing climate on populations’ short-term fitness could vary spatially across species ranges. Here we review the current use of genomic data to predict the disruption of local adaptation across current and future climates. After assessing goals and motivations underlying the approach, we review the main steps and associated statistical methods currently in use and explore our current understanding of the limits and future potential of using genomics to predict climate change (mal)adaptation. Expected final online publication date for the Annual Review of Ecology, Evolution, and Systematics, Volume 51 is November 2, 2020. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates. 
    more » « less
  4. Abstract

    Museum specimens play a crucial role in addressing key questions in systematics, evolution, ecology, and conservation. With the advent of high‐throughput sequencing technologies, specimens that have long been the foundation of important biological discoveries can inform new perspectives as sources of genomic data. Despite the many possibilities associated with analyzing DNA from historical specimens, several challenges persist. Using avian systems as a model, we review DNA extraction protocols, sequencing technologies, and capture methods that are helping researchers overcome some of these difficulties. We highlight empirical examples in which researchers have used these technologies to address fundamental questions related to avian conservation and evolution. Increasing accessibility to new sequencing technologies will provide researchers with tools to tap into the wealth of information contained within our valuable natural history collections.

     
    more » « less
  5. null (Ed.)
    Analyses of morphological disparity have been used to characterize and investigate the evolution of variation in the anatomy, function and ecology of organisms since the 1980s. While a diversity of methods have been employed, it is unclear whether they provide equivalent insights. Here, we review the most commonly used approaches for characterizing and analysing morphological disparity, all of which have associated limitations that, if ignored, can lead to misinterpretation. We propose best practice guidelines for disparity analyses, while noting that there can be no ‘one-size-fits-all’ approach. The available tools should always be used in the context of a specific biological question that will determine data and method selection at every stage of the analysis. 
    more » « less