skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Enhancing Reproducibility and Replicability in Remote Sensing Deep Learning Research and Practice
Many issues can reduce the reproducibility and replicability of deep learning (DL) research and application in remote sensing, including the complexity and customizability of architectures, variable model training and assessment processes and practice, inability to fully control random components of the modeling workflow, data leakage, computational demands, and the inherent nature of the process, which is complex, difficult to perform systematically, and challenging to fully document. This communication discusses key issues associated with convolutional neural network (CNN)-based DL in remote sensing for undertaking semantic segmentation, object detection, and instance segmentation tasks and offers suggestions for best practices for enhancing reproducibility and replicability and the subsequent utility of research results, proposed workflows, and generated data. We also highlight lingering issues and challenges facing researchers as they attempt to improve the reproducibility and replicability of their experiments.  more » « less
Award ID(s):
2046059
PAR ID:
10418214
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Remote Sensing
Volume:
14
Issue:
22
ISSN:
2072-4292
Page Range / eLocation ID:
5760
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, Remote Sensing)
    The field of remote sensing has undergone a remarkable shift where vast amounts of imagery are now readily available to researchers. New technologies, such as uncrewed aircraft systems, make it possible for anyone with a moderate budget to gather their own remotely sensed data, and methodological innovations have added flexibility for processing and analyzing data. These changes create both the opportunity and need to reproduce, replicate, and compare remote sensing methods and results across spatial contexts, measurement systems, and computational infrastructures. Reproducing and replicating research is key to understanding the credibility of studies and extending recent advances into new discoveries. However, reproducibility and replicability (R&R) remain issues in remote sensing because many studies cannot be independently recreated and validated. Enhancing the R&R of remote sensing research will require significant time and effort by the research community. However, making remote sensing research reproducible and replicable does not need to be a burden. In this paper, we discuss R&R in the context of remote sensing and link the recent changes in the field to key barriers hindering R&R while discussing how researchers can overcome those barriers. We argue for the development of two research streams in the field: (1) the coordinated execution of organized sequences of forward-looking replications, and (2) the introduction of benchmark datasets that can be used to test the replicability of results and methods. 
    more » « less
  2. (, Publications listing National Academy of Sciences National Academy of Engineering Institute of Medicine National Research Council)
    null (Ed.)
    One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science. 
    more » « less
  3. ; ; ; ; ; ; ; (, Hydrology and Earth System Sciences)
    Heavy rains and tropical storms often result in floods, which are expected to increase in frequency and intensity. Flood prediction models and inundation mapping tools provide decision-makers and emergency responders with crucial information to better prepare for these events. However, the performance of models relies on the accuracy and timeliness of data received from in situ gaging stations and remote sensing; each of these data sources has its limitations, especially when it comes to real-time monitoring of floods. This study presents a vision-based framework for measuring water levels and detecting floods using computer vision and deep learning (DL) techniques. The DL models use time-lapse images captured by surveillance cameras during storm events for the semantic segmentation of water extent in images. Three different DL-based approaches, namely PSPNet, TransUNet, and SegFormer, were applied and evaluated for semantic segmentation. The predicted masks are transformed into water level values by intersecting the extracted water edges, with the 2D representation of a point cloud generated by an Apple iPhone 13 Pro lidar sensor. The estimated water levels were compared to reference data collected by an ultrasonic sensor. The results showed that SegFormer outperformed other DL-based approaches by achieving 99.55 % and 99.81 % for intersection over union (IoU) and accuracy, respectively. Moreover, the highest correlations between reference data and the vision-based approach reached above 0.98 for both the coefficient of determination (R2) and Nash–Sutcliffe efficiency. This study demonstrates the potential of using surveillance cameras and artificial intelligence for hydrologic monitoring and their integration with existing surveillance infrastructure. 
    more » « less
  4. ; ; ; ; ; (, ECS Sensors Plus)
    The ability to repeat research is vital in confirming the validity of scientific discovery and is relevant to ubiquitous sensor research. Investigation of novel sensors and sensing mechanisms intersect several Federal and non-Federal agencies. Despite numerous studies on sensors at different stages of development, the absence of new field-ready or commercial sensors seems limited by reproducibility. Current research practices in sensors needs sustainable transformations. The scientific community seeks ways to incorporate reproducibility and repeatability to validate published results. A case study on the reproducibility of low-cost air quality sensors is presented. In this context, the article discusses (a) open source data management frameworks in alignment with findability, accessibility, interoperability, and reuse (FAIR) principles to facilitate sensor reproducibility; (b) suggestions for journals focused on sensors to incorporate a reproducibility editorial board and incentivization for data sharing; (c) practice of reproducibility by targeted focus issues; and (d) education of current and the next generation of diverse student and faculty community on FAIR principles. The existence of different types of sensors such as physical, chemical, biological, and magnetic (to name a few) and the fact that the sensing field spans multiple disciplines (electrical engineering, mechanical engineering, physics, chemistry, and electrochemistry) call for a generic model for reproducibility. Considering the available metrics, the authors propose eight FAIR metric standards to that transcend disciplines: citation standards, design and analysis transparency, data transparency, analytical methods transparency, research materials transparency, hardware transparency, preregistration of studies, and replication. 
    more » « less
  5. ; ; ; ; ; ; ; ; ; ; et al (, Nature Methods)
    Abstract Neuroscience is advancing standardization and tool development to support rigor and transparency. Consequently, data pipeline complexity has increased, hindering FAIR (findable, accessible, interoperable and reusable) access. brainlife.io was developed to democratize neuroimaging research. The platform provides data standardization, management, visualization and processing and automatically tracks the provenance history of thousands of data objects. Here, brainlife.io is described and evaluated for validity, reliability, reproducibility, replicability and scientific utility using four data modalities and 3,200 participants. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email