Search for: All records

DeepTensor is a computationally efficient framework for low-rank decomposition of matrices and tensors using deep generative networks. We decompose a tensor as the product of low-rank tensor factors where each low-rank tensor is generated by a deep network (DN) that is trained in a self-supervised manner to minimize the mean-square approximation error. Our key observation is that the implicit regularization inherent in DNs enables them to capture nonlinear signal structures that are out of the reach of classical linear methods like the singular value decomposition (SVD) and principal components analysis (PCA). We demonstrate that the performance of DeepTensor is robust to a wide range of distributions and a computationally efficient drop-in replacement for the SVD, PCA, nonnegative matrix factorization (NMF), and similar decompositions by exploring a range of real-world applications, including hyperspectral image denoising, 3D MRI tomography, and image classification. 

Abstract Mesoscopic calcium imaging enables studies of cell-type specific neural activity over large areas. A growing body of literature suggests that neural activity can be different when animals are free to move compared to when they are restrained. Unfortunately, existing systems for imaging calcium dynamics over large areas in non-human primates (NHPs) are table-top devices that require restraint of the animal’s head. Here, we demonstrate an imaging device capable of imaging mesoscale calcium activity in a head-unrestrained male non-human primate. We successfully miniaturize our system by replacing lenses with an optical mask and computational algorithms. The resulting lensless microscope can fit comfortably on an NHP, allowing its head to move freely while imaging. We are able to measure orientation columns maps over a 20 mm²field-of-view in a head-unrestrained macaque. Our work establishes mesoscopic imaging using a lensless microscope as a powerful approach for studying neural activity under more naturalistic conditions. 

Sub-wavelength diffractive meta-optics have emerged as a versatile platform to manipulate light fields at will, due to their ultra-small form factor and flexible multifunctionalities. However, miniaturization and multimodality are typically compromised by a reduction in imaging performance; thus, meta-optics often yield lower resolution and stronger aberration compared to traditional refractive optics. Concurrently, computational approaches have become popular to improve the image quality of traditional cameras and exceed limitations posed by refractive lenses. This in turn often comes at the expense of higher power and latency, and such systems are typically limited by the availability of certain refractive optics. Limitations in both fields have thus sparked cross-disciplinary efforts to not only overcome these roadblocks but also to go beyond and provide synergistic meta-optical–digital solutions that surpass the potential of the individual components. For instance, an application-specific meta-optical frontend can preprocess the light field of a scene and focus it onto the sensor with a desired encoding, which can either ease the computational load on the digital backend or can intentionally alleviate certain meta-optical aberrations. In this review, we introduce the fundamentals, summarize the development of meta-optical computational imaging, focus on latest advancements that redefine the current state of the art, and give a perspective on research directions that leverage the full potential of sub-wavelength photonic platforms in imaging and sensing applications. The current advancement of meta-optics and recent investments by foundries and technology partners have the potential to provide synergistic future solutions for highly efficient, compact, and low-power imaging systems. 

Neural radiance fields (NeRFs) show potential for transforming images captured worldwide into immersive 3D visual experiences. However, most of this captured visual data remains siloed in our camera rolls as these images contain personal details. Even if made public, the problem of learning 3D representations of billions of scenes captured daily in a centralized manner is computationally intractable. Our approach, DecentNeRF, is the first attempt at decentralized, crowd-sourced NeRFs that require less server computing for a scene than a centralized approach. Instead of sending the raw data, our approach requires users to send a 3D representation, distributing the high computation cost of training centralized NeRFs between the users. It learns photorealistic scene representations by decomposing users’ 3D views into personal and global NeRFs and a novel optimally weighted aggregation of only the latter. We validate the advantage of our approach to learn NeRFs with photorealism and minimal server computation cost on structured synthetic and real-world photo tourism datasets. We further analyze how secure aggregation of global NeRFs in DecentNeRF minimizes the undesired reconstruction of personal content by the server. 

Significance: Accurate identification between pathologic (e.g., tumors) and healthy brain tissue is a critical need in neurosurgery. However, conventional surgical adjuncts have significant limitations toward achieving this goal (e.g., image guidance based on pre-operative imaging becomes inaccurate up to 3 cm as surgery proceeds). Hyperspectral imaging (HSI) has emerged as a potential powerful surgical adjunct to enable surgeons to accurately distinguish pathologic from normal tissues. Aim: We review HSI techniques in neurosurgery; categorize, explain, and summarize their technical and clinical details; and present some promising directions for future work. Approach: We performed a literature search on HSI methods in neurosurgery focusing on their hardware and implementation details; classification, estimation, and band selection methods; publicly available labeled and unlabeled data; image processing and augmented reality visualization systems; and clinical study conclusions. Results: We present a detailed review of HSI results in neurosurgery with a discussion of over 25 imaging systems, 45 clinical studies, and 60 computational methods. We first provide a short overview of HSI and the main branches of neurosurgery. Then, we describe in detail the imaging systems, computational methods, and clinical results for HSI using reflectance or fluorescence. Clinical implementations of HSI yield promising results in estimating perfusion and mapping brain function, classifying tumors and healthy tissues (e.g., in fluorescence-guided tumor surgery, detecting infiltrating margins not visible with conventional systems), and detecting epileptogenic regions. Finally, we discuss the advantages and disadvantages of HSI approaches and interesting research directions as a means to encourage future development. Conclusions: We describe a number of HSI applications across every major branch of neurosurgery. We believe these results demonstrate the potential of HSI as a powerful neurosurgical adjunct as more work continues to enable rapid acquisition with smaller footprints, greater spectral and spatial resolutions, and improved detection. 

Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferrable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for ≈+10dB gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER's features. 

Timely and accurate detection of hurricane debris is critical for effective disaster response and community resilience. While post‐disaster aerial imagery is readily available, robust debris segmentation solutions applicable across multiple disaster regions remain limited. Since environmental and imaging conditions vary by region and alter debris's visual signatures, developing a debris segmentation model suitable for different hazard scenarios is challenging, especially given the scarcity of training data. This study addresses these challenges by relying on vast pretrained knowledge from the foundation vision model Contrastive Language‐Image Pre‐Training (CLIP) and fine‐tuning a lightweight decoder on labeled debris imagery, thereby achieving robust performance in segmenting non‐vegetative debris within complex urban backgrounds with a relatively small, high‐quality dataset. To enable fine‐tuning, this work introduces an open‐source dataset comprising approximately 1200 manually annotated aerial RGB images of structural debris from Hurricanes Ian, Ida, and Ike. To mitigate human biases and enhance data quality, labels from multiple annotators are strategically aggregated into consensus annotations. The resulting fine‐tuned model, named CLIPSeg‐debris, achieves a Dice score of 0.86 on data from Hurricane Ida—a disaster event entirely excluded during training—outperforming baseline models. This approach thereby enables broader adoption across multiple hazard scenarios and effectively adapts to various backgrounds, sensor resolutions, and debris conditions.