Phenology––the timing of life-history events––is a key trait for understanding responses of organisms to climate. The digitization and online mobilization of herbarium specimens is rapidly advancing our understanding of plant phenological response to climate and climatic change. The current common practice of manually harvesting data from individual specimens greatly restricts our ability to scale data collection to entire collections. Recent investigations have demonstrated that machine-learning models can facilitate data collection from herbarium specimens. However, present attempts have focused largely on simplistic binary coding of reproductive phenology (e.g., flowering or not). Here, we use crowd-sourced phenological data of numbers of buds, flowers, and fruits of more than 3000 specimens of six common wildflower species of the eastern United States (Anemone canadensis, A. hepatica, A. quinquefolia, Trillium erectum, T. grandiflorum, and T. undulatum} to train a model using Mask R-CNN to segment and count phenological features. A single global model was able to automate the binary coding of reproductive stage with greater than 90% accuracy. Segmenting and counting features were also successful, but accuracy varied with phenological stage and taxon. Counting buds was significantly more accurate than flowers or fruits. Moreover, botanical experts provided more reliable data than either crowd-sourcers or our Mask R-CNN model, highlighting the importance of high-quality human training data. Finally, we also demonstrated the transferability of our model to automated phenophase detection and counting of the three Trillium species, which have large and conspicuously-shaped reproductive organs. These results highlight the promise of our two-phase crowd-sourcing and machine-learning pipeline to segment and count reproductive features of herbarium specimens, providing high-quality data with which to study responses of plants to ongoing climatic change.
more »
« less
A new fine‐grained method for automated visual analysis of herbarium specimens: A case study for phenological data extraction
Herbarium specimens represent an outstanding source of material with which to study plant phenological changes in response to climate change. The fine‐scale phenological annotation of such specimens is nevertheless highly time consuming and requires substantial human investment and expertise, which are difficult to rapidly mobilize.
We trained and evaluated new deep learning models to automate the detection, segmentation, and classification of four reproductive structures of
The main outcome of our study is that the performance of detection and segmentation can vary significantly with: (i) the type of annotations used for training, (ii) the type of reproductive structures, and (iii) the size of the reproductive structures. In the case of
This method has great potential for automating the analysis of reproductive structures in high‐resolution images of herbarium sheets. Deeper investigations regarding the taxonomic scalability of this approach and its potential improvement will be conducted in future work.
- Award ID(s):
- 1802181
- NSF-PAR ID:
- 10456862
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Applications in Plant Sciences
- Volume:
- 8
- Issue:
- 6
- ISSN:
- 2168-0450
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Premise Herbarium specimens have been used to detect climate‐induced shifts in flowering time by using the day of year of collection (
DOY ) as a proxy for first or peak flowering date. Variation among herbarium sheets in their phenological status, however, undermines the assumption thatDOY accurately represents any particular phenophase. Ignoring this variation can reduce the explanatory power of pheno‐climatic models (PCM s) designed to predict the effects of climate on flowering date.Methods Here we present a protocol for the phenological scoring of imaged herbarium specimens using an ImageJ plugin, and we introduce a quantitative metric of a specimen's phenological status, the phenological index (
PI ), which we use inPCM s to control for phenological variation among specimens ofStreptanthus tortuosus (Brassicaceeae) when testing for the effects of climate onDOY . We demonstrate that includingPI as an independent variable improves model fit.Results Including
PI inPCM s increased the modelR 2relative toPCM s that excludedPI ; regression coefficients for climatic parameters, however, remained constant.Discussion Our protocol provides a simple, quantitative phenological metric for any observed plant. Including
PI inPCM s increasesR 2and enables predictions of theDOY of any phenophase under any specified climatic conditions. -
Machine learning (ML) can accelerate the extraction of phenological data from herbarium specimens; however, no studies have assessed whether ML-derived phenological data can be used reliably to evaluate ecological patterns. In this study, 709 herbarium specimens representing a widespread annual herb, Streptanthus tortuosus, were scored both manually by human observers and by a mask R-CNN object detection model to (1) evaluate the concordance between ML and manually-derived phenological data and (2) determine whether ML-derived data can be used to reliably assess phenological patterns. The ML model generally underestimated the number of reproductive structures present on each specimen; however, when these counts were used to provide a quantitative estimate of the phenological stage of plants on a given sheet (i.e., the phenological index or PI), the ML and manually-derived PI’s were highly concordant. Moreover, herbarium specimen age had no effect on the estimated PI of a given sheet. Finally, including ML-derived PIs as predictor variables in phenological models produced estimates of the phenological sensitivity of this species to climate, temporal shifts in flowering time, and the rate of phenological progression that are indistinguishable from those produced by models based on data provided by human observers. This study demonstrates that phenological data extracted using machine learning can be used reliably to estimate the phenological stage of herbarium specimens and to detect phenological patterns.more » « less
-
more » « less
Premise of the Study Phenological annotation models computed on large‐scale herbarium data sets were developed and tested in this study.
Methods Herbarium specimens represent a significant resource with which to study plant phenology. Nevertheless, phenological annotation of herbarium specimens is time‐consuming, requires substantial human investment, and is difficult to mobilize at large taxonomic scales. We created and evaluated new methods based on deep learning techniques to automate annotation of phenological stages and tested these methods on four herbarium data sets representing temperate, tropical, and equatorial American floras.
Results Deep learning allowed correct detection of fertile material with an accuracy of 96.3%. Accuracy was slightly decreased for finer‐scale information (84.3% for flower and 80.5% for fruit detection).
Discussion The method described has the potential to allow fine‐grained phenological annotation of herbarium specimens at large ecological scales. Deeper investigation regarding the taxonomic scalability of this approach is needed.
-
more » « less
Premise of the Study A novel method of estimating phenology of herbarium specimens was developed to facilitate more precise determination of plant phenological responses to explanatory variables (e.g., climate).
Methods and Results Simulated specimen data sets were used to compare the precision of phenological models using the new method and two common, alternative methods (flower presence/absence and ≥50% flowers present). The new “estimated phenophase” method was more precise and extracted a greater number of significant species‐level relationships; however, this method only slightly outperformed the simple “binary” (e.g., flowers present/absent) method.
Conclusions The new method enables estimation of phenological trends with greater precision. However, when time and resources are limited, a presence/absence method may offer comparable results at lower cost. Using a more restrictive approach, such as only including specimens in a certain phenophase, is not advised given the detrimental effect of decreased sample size on resulting models.
