More Like this

1. An end-to-end recurrent compressed sensing method to denoise, detect and demix calcium imaging data

   https://doi.org/10.1038/s42256-024-00892-w  

   Zhang, Kangning; Tang, Sean; Zhu, Vivian; Barchini, Majd; Yang, Weijian (September 2024, Nature Machine Intelligence)

   Two-photon calcium imaging provides large-scale recordings of neuronal activities at cellular resolution. A robust, automated and high-speed pipeline to simultaneously segment the spatial footprints of neurons and extract their temporal activity traces while decontaminating them from background, noise and overlapping neurons is highly desirable to analyse calcium imaging data. Here we demonstrate DeepCaImX, an end-to-end deep learning method based on an iterative shrinkage-thresholding algorithm and a long short-term memory neural network to achieve the above goals altogether at a very high speed and without any manually tuned hyperparameter. DeepCaImX is a multi-task, multi-class and multi-label segmentation method composed of a compressed sensing-inspired neural network with a recurrent layer and fully connected layers. The neural network can simultaneously generate accurate neuronal footprints and extract clean neuronal activity traces from calcium imaging data. We trained the neural network with simulated datasets and benchmarked it against existing state-of-the-art methods with in vivo experimental data. DeepCaImX outperforms existing methods in the quality of segmentation and temporal trace extraction as well as processing speed. DeepCaImX is highly scalable and will benefit the analysis of mesoscale calcium imaging. 

   more » « less
2. CAT: Closed-loop Adversarial Training for Safe End-to-End Driving

   Zhang, Linrui; Peng, Zhenghao; Li, Quanyi; Zhou, Bolei. (November 2023, Conference on Robot Learning (CoRL) 2023)

   Driving safety is a top priority for autonomous vehicles. Orthogonal to prior work handling accident-prone traffic events by algorithm designs at the policy level, we investigate a Closed-loop Adversarial Training (CAT) framework for safe end-to-end driving in this paper through the lens of environment augmentation. CAT aims to continuously improve the safety of driving agents by training the agent on safety-critical scenarios that are dynamically generated over time. A novel resampling technique is developed to turn log-replay real-world driving scenarios into safety-critical ones via probabilistic factorization, where the adversarial traffic generation is modeled as the multiplication of standard motion prediction sub-problems. Consequently, CAT can launch more efficient physical attacks compared to existing safety-critical scenario generation methods and yields a significantly less computational cost in the iterative learning pipeline. We incorporate CAT into the MetaDrive simulator and validate our approach on hundreds of driving scenarios imported from real-world driving datasets. Experimental results demonstrate that CAT can effectively generate adversarial scenarios countering the agent being trained. After training, the agent can achieve superior driving safety in both log-replay and safety-critical traffic scenarios on the held- out test set. Code and data are available at https://metadriverse.github.io/cat. 

   more » « less
3. Learnable-pattern scanning based deep compressed imaging

   https://doi.org/10.1117/12.2577998  

   Zhang, Kangning; Hu, Junjie; Yang, Weijian (March 2021, High-Speed Biomedical Imaging and Spectroscopy VI)

   Goda, Keisuke; Tsia, Kevin K. (Ed.)

   We present a new deep compressed imaging modality by scanning a learned illumination pattern on the sample and detecting the signal with a single-pixel detector. This new imaging modality allows a compressed sampling of the object, and thus a high imaging speed. The object is reconstructed through a deep neural network inspired by compressed sensing algorithm. We optimize the illumination pattern and the image reconstruction network by training an end-to-end auto-encoder framework. Comparing with the conventional single-pixel camera and point-scanning imaging system, we accomplish a high-speed imaging with a reduced light dosage, while preserving a high imaging quality. 

   more » « less
4. Attacking Vision-based Perception in End-to-End Autonomous Driving Models

   Boloor, Adith; Garimella, Karthik; He, Xin; Gill, Christopher; Vorobeychik, Yevgeniy; Zhang, Xuan (January 2020, Journal of systems architecture)

   Recent advances in machine learning, especially techniques such as deep neural networks, are enabling a range of emerging applications. One such example is autonomous driving, which often relies on deep learning for perception. However, deep learning-based perception has been shown to be vulnerable to a host of subtle adversarial manipulations of images. Nevertheless, the vast majority of such demonstrations focus on perception that is disembodied from end-to-end control. We present novel end-to-end attacks on autonomous driving in simulation, using simple physically realizable attacks: the painting of black lines on the road. These attacks target deep neural network models for endto-end autonomous driving control. A systematic investigation shows that such attacks are easy to engineer, and we describe scenarios (e.g., right turns) in which they are highly effective. We define several objective functions that quantify the success of an attack and develop techniques based on Bayesian Optimization to efficiently traverse the search space of higher dimensional attacks. Additionally, we define a novel class of hijacking attacks, where painted lines on the road cause the driverless car to follow a target path. Through the use of network deconvolution, we provide insights into the successful attacks, which appear to work by mimicking activations of entirely different scenarios. Our code is available on https://github.com/xz-group/AdverseDrive 

   more » « less
5. Deep compressed imaging via optimized pattern scanning

   https://doi.org/10.1364/PRJ.410556  

   Zhang, Kangning; Hu, Junjie; Yang, Weijian (March 2021, Photonics Research)

   The need for high-speed imaging in applications such as biomedicine, surveillance, and consumer electronics has called for new developments of imaging systems. While the industrial effort continuously pushes the advance of silicon focal plane array image sensors, imaging through a single-pixel detector has gained significant interest thanks to the development of computational algorithms. Here, we present a new imaging modality, deep compressed imaging via optimized-pattern scanning, which can significantly increase the acquisition speed for a single-detector-based imaging system. We project and scan an illumination pattern across the object and collect the sampling signal with a single-pixel detector. We develop an innovative end-to-end optimized auto-encoder, using a deep neural network and compressed sensing algorithm, to optimize the illumination pattern, which allows us to reconstruct faithfully the image from a small number of measurements, with a high frame rate. Compared with the conventional switching-mask-based single-pixel camera and point-scanning imaging systems, our method achieves a much higher imaging speed, while retaining a similar imaging quality. We experimentally validated this imaging modality in the settings of both continuous-wave illumination and pulsed light illumination and showed high-quality image reconstructions with a high compressed sampling rate. This new compressed sensing modality could be widely applied in different imaging systems, enabling new applications that require high imaging speeds. 

   more » « less