More Like this

1. Learn an Index Operator by CNN for Solving Diffusive Optical Tomography: A Deep Direct Sampling Method

   https://doi.org/10.1007/s10915-023-02115-7  

   Guo, Ruchi; Jiang, Jiahua; Li, Yi (March 2023, Journal of Scientific Computing)

   Abstract In this work, we investigate the diffusive optical tomography (DOT) problem in the case that limited boundary measurements are available. Motivated by the direct sampling method (DSM) proposed in Chow et al. (SIAM J Sci Comput 37(4):A1658–A1684, 2015), we develop a deep direct sampling method (DDSM) to recover the inhomogeneous inclusions buried in a homogeneous background. In this method, we design a convolutional neural network to approximate the index functional that mimics the underling mathematical structure. The benefits of the proposed DDSM include fast and easy implementation, capability of incorporating multiple measurements to attain high-quality reconstruction, and advanced robustness against the noise. Numerical experiments show that the reconstruction accuracy is improved without degrading the efficiency, demonstrating its potential for solving the real-world DOT problems. 

   more » « less
2. Transformer Meets Boundary Value Inverse Problem

   Guo, Ruchi; Cao, Shuhao; Chen, Long (May 2023, International Conference on Learning Representations)

   A Transformer-based deep direct sampling method is proposed for electrical impedance tomography, a well-known severely ill-posed nonlinear boundary value inverse problem. A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images. An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks? Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels. Then, by introducing learnable non-local kernels, the direct sampling is recast to a modified attention mechanism. The new method achieves superior accuracy over its predecessors and contemporary operator learners and shows robustness to noises in benchmarks. This research shall strengthen the insights that, despite being invented for natural language processing tasks, the attention mechanism offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures. 

   more » « less
3. Robust Embedded Deep K-means Clustering

   https://doi.org/10.1145/3357384.3357985  

   Zhang, Rui; Tong, Hanghang; Xia, Yinglong; Zhu, Yada (November 2019, CIKM)

   Deep neural network clustering is superior to the conventional clustering methods due to deep feature extraction and nonlinear dimensionality reduction. Nevertheless, deep neural network leads to a rough representation regarding the inherent relationship of the data points. Therefore, it is still difficult for deep neural network to exploit the effective structure for direct clustering. To address this issue,we propose a robust embedded deep K-means clustering (REDKC) method. The proposed RED-KC approach utilizes the δ-norm metric to constrain the feature mapping process of the auto-encoder network, so that data are mapped to a latent feature space, which is more conducive to the robust clustering. Compared to the existing auto-encoder networks with the fixed prior, the proposed RED-KC is adaptive during the process of feature mapping. More importantly, the proposed RED-KC embeds the clustering process with the autoencoder network, such that deep feature extraction and clustering can be performed simultaneously. Accordingly, a direct and efficient clustering could be obtained within only one step to avoid the inconvenience of multiple separate stages, namely, losing pivotal information and correlation. Consequently, extensive experiments are provided to validate the effectiveness of the proposed approach. 

   more » « less
4. Perceptron Theory for Predicting the Accuracy of Neural Networks

   Kleyko, Denis; Rosato, Antonello; Frady, E. Paxon; Panella, Massimo; Sommer, Friedrich T. (February 2023, IEEE transactions on neural networks and learning systems)

   Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks with different architectures. A general theory of classification with perceptrons is developed by generalizing an existing theory for analyzing reservoir computing models and connectionist models for symbolic reasoning known as vector symbolic architectures. Our statistical theory offers three formulas leveraging the signal statistics with increasing detail. The formulas are analytically intractable, but can be evaluated numerically. The description level that captures maximum details requires stochastic sampling methods. Depending on the network model, the simpler formulas already yield high prediction accuracy. The quality of the theory predictions is assessed in three experimental settings, a memorization task for echo state networks (ESNs) from reservoir computing literature, a collection of classification datasets for shallow randomly connected networks, and the ImageNet dataset for deep convolutional neural networks. We find that the second description level of the perceptron theory can predict the performance of types of ESNs, which could not be described previously. Furthermore, the theory can predict deep multilayer neural networks by being applied to their output layer. While other methods for prediction of neural networks performance commonly require to train an estimator model, the proposed theory requires only the first two moments of the distribution of the postsynaptic sums in the output neurons. Moreover, the perceptron theory compares favorably to other methods that do not rely on training an estimator model. 

   more » « less
5. Adaptive Privacy Preserving Deep Learning Algorithms for Medical Data

   Zhang, Xinyue; Ding, Jiahao; Wu, Maoqiang; Wong, Stephen TC.; Nguyen, Hien V.; Pan, Miao (January 2021, IEEE Winter Conference on Applications of Computer Vision)

   null (Ed.)

   Deep learning holds a great promise of revolutionizing healthcare and medicine. Unfortunately, various inference attack models demonstrated that deep learning puts sensitive patient information at risk. The high capacity of deep neural networks is the main reason behind the privacy loss. In particular, patient information in the training data can be unintentionally memorized by a deep network. Adversarial parties can extract that information given the ability to access or query the network. In this paper, we propose a novel privacy-preserving mechanism for training deep neural networks. Our approach adds decaying Gaussian noise to the gradients at every training iteration. This is in contrast to the mainstream approach adopted by Google's TensorFlow Privacy, which employs the same noise scale in each step of the whole training process. Compared to existing methods, our proposed approach provides an explicit closed-form mathematical expression to approximately estimate the privacy loss. It is easy to compute and can be useful when the users would like to decide proper training time, noise scale, and sampling ratio during the planning phase. We provide extensive experimental results using one real-world medical dataset (chest radiographs from the CheXpert dataset) to validate the effectiveness of the proposed approach. The proposed differential privacy based deep learning model achieves significantly higher classification accuracy over the existing methods with the same privacy budget. 

   more » « less