skip to main content

Explore Research Products in the NSF-PAR

Title: Differentiable Programming for Hyperspectral Unmixing using a Physics-based Dispersion Model
Hyperspectral unmixing is an important remote sensing task with applications including material identification and analysis. Characteristic spectral features make many pure materials identifiable from their visible-to-infrared spectra, but quantifying their presence within a mixture is a challenging task due to nonlinearities and factors of variation. In this paper, spectral variation is considered from a physics-based approach and incorporated into an end-to-end spectral unmixing algorithm via differentiable programming. The dispersion model is introduced to simulate realistic spectral variation, and an efficient method to fit the parameters is presented. Then, this dispersion model is utilized as a generative model within an analysis-by-synthesis spectral unmixing algorithm. Further, a technique for inverse rendering using a convolutional neural network to predict parameters of the generative model is introduced to enhance performance and speed when training data is available. Results achieve state-of-the-art on both infrared and visible-to-near-infrared (VNIR) datasets, and show promise for the synergy between physicsbased models and deep learning in hyperspectral unmixing in the future.  more » « less
Award ID(s):
1909192
NSF-PAR ID:
10193714
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
European Conference on Computer Vision (ECCV)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ( , Remote Sensing)
    null (Ed.)
    Alaska has witnessed a significant increase in wildfire events in recent decades that have been linked to drier and warmer summers. Forest fuel maps play a vital role in wildfire management and risk assessment. Freely available multispectral datasets are widely used for land use and land cover mapping, but they have limited utility for fuel mapping due to their coarse spectral resolution. Hyperspectral datasets have a high spectral resolution, ideal for detailed fuel mapping, but they are limited and expensive to acquire. This study simulates hyperspectral data from Sentinel-2 multispectral data using the spectral response function of the Airborne Visible/Infrared Imaging Spectrometer-Next Generation (AVIRIS-NG) sensor, and normalized ground spectra of gravel, birch, and spruce. We used the Uniform Pattern Decomposition Method (UPDM) for spectral unmixing, which is a sensor-independent method, where each pixel is expressed as the linear sum of standard reference spectra. The simulated hyperspectral data have spectral characteristics of AVIRIS-NG and the reflectance properties of Sentinel-2 data. We validated the simulated spectra by visually and statistically comparing it with real AVIRIS-NG data. We observed a high correlation between the spectra of tree classes collected from AVIRIS-NG and simulated hyperspectral data. Upon performing species level classification, we achieved a classification accuracy of 89% for the simulated hyperspectral data, which is better than the accuracy of Sentinel-2 data (77.8%). We generated a fuel map from the simulated hyperspectral image using the Random Forest classifier. Our study demonstrated that low-cost and high-quality hyperspectral data can be generated from Sentinel-2 data using UPDM for improved land cover and vegetation mapping in the boreal forest. 
    more » « less
  2. ; ; ; ; ( , Remote Sensing)
    null (Ed.)
    Accurate, precise, and timely estimation of crop yield is key to a grower’s ability to proactively manage crop growth and predict harvest logistics. Such yield predictions typically are based on multi-parametric models and in-situ sampling. Here we investigate the extension of a greenhouse study, to low-altitude unmanned aerial systems (UAS). Our principal objective was to investigate snap bean crop (Phaseolus vulgaris) yield using imaging spectroscopy (hyperspectral imaging) in the visible to near-infrared (VNIR; 400–1000 nm) region via UAS. We aimed to solve the problem of crop yield modelling by identifying spectral features explaining yield and evaluating the best time period for accurate yield prediction, early in time. We introduced a Python library, named Jostar, for spectral feature selection. Embedded in Jostar, we proposed a new ranking method for selected features that reaches an agreement between multiple optimization models. Moreover, we implemented a well-known denoising algorithm for the spectral data used in this study. This study benefited from two years of remotely sensed data, captured at multiple instances over the summers of 2019 and 2020, with 24 plots and 18 plots, respectively. Two harvest stage models, early and late harvest, were assessed at two different locations in upstate New York, USA. Six varieties of snap bean were quantified using two components of yield, pod weight and seed length. We used two different vegetation detection algorithms. the Red-Edge Normalized Difference Vegetation Index (RENDVI) and Spectral Angle Mapper (SAM), to subset the fields into vegetation vs. non-vegetation pixels. Partial least squares regression (PLSR) was used as the regression model. Among nine different optimization models embedded in Jostar, we selected the Genetic Algorithm (GA), Ant Colony Optimization (ACO), Simulated Annealing (SA), and Particle Swarm Optimization (PSO) and their resulting joint ranking. The findings show that pod weight can be explained with a high coefficient of determination (R2 = 0.78–0.93) and low root-mean-square error (RMSE = 940–1369 kg/ha) for two years of data. Seed length yield assessment resulted in higher accuracies (R2 = 0.83–0.98) and lower errors (RMSE = 4.245–6.018 mm). Among optimization models used, ACO and SA outperformed others and the SAM vegetation detection approach showed improved results when compared to the RENDVI approach when dense canopies were being examined. Wavelengths at 450, 500, 520, 650, 700, and 760 nm, were identified in almost all data sets and harvest stage models used. The period between 44–55 days after planting (DAP) the optimal time period for yield assessment. Future work should involve transferring the learned concepts to a multispectral system, for eventual operational use; further attention should also be paid to seed length as a ground truth data collection technique, since this yield indicator is far more rapid and straightforward. 
    more » « less
  3. ; ; ; ; ; ; ; ; ( , Proc. SPIE 10497, Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues XVI)
    Autofluorescence has historically been considered a nuisance in medical imaging. Many endogenous fluorophores, specifically, collagen, elastin, NADH, and FAD, are found throughout the human body. Diagnostically, these signals can be prohibitive since they can outcompete signals introduced for diagnostic purposes. Recent advances in hyperspectral imaging have allowed the acquisition of significantly more data in a shorter time period by scanning the excitation spectra of fluorophores. The reduced acquisition time and increased signal-to-noise ratio allow for separation of significantly more fluorophores than previously possible. Here, we propose to utilize excitation-scanning of autofluorescence to examine tissues and diagnose pathologies. Spectra of autofluorescent molecules were obtained using a custom inverted microscope (TE-2000, Nikon Instruments) with a Xe arc lamp and thin film tunable filter array (VersaChrome, Semrock, Inc.) Scans utilized excitation wavelengths from 360 nm to 550 nm in 5 nm increments. The resultant spectra were used to examine hyperspectral image stacks from various collaborative studies, including an atherosclerotic rat model and a colon cancer study. Hyperspectral images were analyzed with ENVI and custom Matlab scripts including linear spectral unmixing (LSU) and principal component analysis (PCA). Initial results suggest the ability to separate the signals of endogenous fluorophores and measure the relative concentrations of fluorophores among healthy and diseased states of similar tissues. These results suggest pathology-specific changes to endogenous fluorophores can be detected using excitationscanning hyperspectral imaging. Future work will expand the library of pure molecules and will examine more defined disease states. 
    more » « less
  4. ; ; ; ; ; ; ; ( , Plant Methods)
    Abstract Background Remote sensing instruments enable high-throughput phenotyping of plant traits and stress resilience across scale. Spatial (handheld devices, towers, drones, airborne, and satellites) and temporal (continuous or intermittent) tradeoffs can enable or constrain plant science applications. Here, we describe the technical details of TSWIFT (Tower Spectrometer on Wheels for Investigating Frequent Timeseries), a mobile tower-based hyperspectral remote sensing system for continuous monitoring of spectral reflectance across visible-near infrared regions with the capacity to resolve solar-induced fluorescence (SIF). Results We demonstrate potential applications for monitoring short-term (diurnal) and long-term (seasonal) variation of vegetation for high-throughput phenotyping applications. We deployed TSWIFT in a field experiment of 300 common bean genotypes in two treatments: control (irrigated) and drought (terminal drought). We evaluated the normalized difference vegetation index (NDVI), photochemical reflectance index (PRI), and SIF, as well as the coefficient of variation (CV) across the visible-near infrared spectral range (400 to 900 nm). NDVI tracked structural variation early in the growing season, following initial plant growth and development. PRI and SIF were more dynamic, exhibiting variation diurnally and seasonally, enabling quantification of genotypic variation in physiological response to drought conditions. Beyond vegetation indices, CV of hyperspectral reflectance showed the most variability across genotypes, treatment, and time in the visible and red-edge spectral regions. Conclusions TSWIFT enables continuous and automated monitoring of hyperspectral reflectance for assessing variation in plant structure and function at high spatial and temporal resolutions for high-throughput phenotyping. Mobile, tower-based systems like this can provide short- and long-term datasets to assess genotypic and/or management responses to the environment, and ultimately enable the spectral prediction of resource-use efficiency, stress resilience, productivity and yield. 
    more » « less
  5. ; ( , Agricultural and Forest Meteorology)
    Salt marshes are highly productive ecosystems relevant for Blue Carbon assessments, but information for estimating gross primary productivity (GPP) from proximal remote sensing (PRS) is limited. Temperate salt marshes have seasonal canopy structure and metabolism changes, defining different canopy phenological phases, GPP rates, and spectral reflectance. We combined multi-annual PRS data (i.e., PhenoCam, discrete hyperspectral measurements, and automated spectral reflectance sensors) with GPP derived from eddy covariance. We tested the performance of empirical models to predict GPP from 12 common vegetation indices (VIs; e.g., NDVI, EVI, PSRI, GCC), Sun-Induced Fluorescence (SIF), and reflectance from different areas of the electromagnetic spectrum (i.e., VIS-IR, RedEdge, IR, and SIF) across the annual cycle and canopy phenological phases (i.e., Greenup, Maturity, Senescence, and Dormancy). Plant Senescence Reflectance Index (PSRI) from hyperspectral data and the Greenness Index (GCC) from PhenoCam, showed the strongest relationship with daily GPP across the annual cycle and within phenological phases (r2=0.30–0.92). Information from the visible-infrared electromagnetic region (VIS-IR) coupled with a partial least square approach (PLSR) showed the highest data-model agreement with GPP, mainly because of its relevance to respond to physiological and structural changes in the canopy, compared with indices (e.g., GCC) that particularly react to changes in the greenness of the canopy. The most relevant electromagnetic regions to model GPP were ∼550 nm and ∼710 nm. Canopy phenological phases impose challenges for modeling GPP with VIs and the PLSR approach, particularly during Maturity, Senescence, and Dormancy. As more eddy covariance sites are established in salt marshes, the application of PRS can be widely tested. Our results highlight the potential to use canopy reflectance from the visible spectrum region for modeling annual GPP in salt marshes as an example of advances within the AmeriFlux network. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email