More Like this

1. Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

   Fang, Ronglong; Xu, Yuesheng (December 2024, 38th Conference on Neural Information Processing Systems (NeurIPS 2024).)

   Deep neural networks (DNNs) have showcased their remarkable precision in approximating smooth functions. However, they suffer from the spectral bias, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to effectively capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN (with trainable parameters) composed with the SNNs (with fixed parameters) trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs. The code is available on GitHub: Addressing Spectral Bias via MGDL. 

   more » « less
2. Separating and reintegrating latent variables to improve classification of genomic data

   https://doi.org/10.1093/biostatistics/kxab046  

   Payne, Nora Yujia; Gagnon-Bartsch, Johann A (January 2022, Biostatistics)

   Summary Genomic data sets contain the effects of various unobserved biological variables in addition to the variable of primary interest. These latent variables often affect a large number of features (e.g., genes), giving rise to dense latent variation. This latent variation presents both challenges and opportunities for classification. While some of these latent variables may be partially correlated with the phenotype of interest and thus helpful, others may be uncorrelated and merely contribute additional noise. Moreover, whether potentially helpful or not, these latent variables may obscure weaker effects that impact only a small number of features but more directly capture the signal of primary interest. To address these challenges, we propose the cross-residualization classifier (CRC). Through an adjustment and ensemble procedure, the CRC estimates and residualizes out the latent variation, trains a classifier on the residuals, and then reintegrates the latent variation in a final ensemble classifier. Thus, the latent variables are accounted for without discarding any potentially predictive information. We apply the method to simulated data and a variety of genomic data sets from multiple platforms. In general, we find that the CRC performs well relative to existing classifiers and sometimes offers substantial gains. 

   more » « less
3. An Information Theoretic Interpretation to Deep Neural Networks

   https://doi.org/10.3390/e24010135  

   Xu, Xiangxiang; Huang, Shao-Lun; Zheng, Lizhong; Wornell, Gregory W. (January 2022, Entropy)

   With the unprecedented performance achieved by deep learning, it is commonly believed that deep neural networks (DNNs) attempt to extract informative features for learning tasks. To formalize this intuition, we apply the local information geometric analysis and establish an information-theoretic framework for feature selection, which demonstrates the information-theoretic optimality of DNN features. Moreover, we conduct a quantitative analysis to characterize the impact of network structure on the feature extraction process of DNNs. Our investigation naturally leads to a performance metric for evaluating the effectiveness of extracted features, called the H-score, which illustrates the connection between the practical training process of DNNs and the information-theoretic framework. Finally, we validate our theoretical results by experimental designs on synthesized data and the ImageNet dataset. 

   more » « less
4. Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

   https://doi.org/10.1175/AIES-D-22-0063.1  

   Dematties, Dario; Raut, Bhupendra A.; Park, Seongha; Jackson, Robert C.; Shahkarami, Sean; Kim, Yongho; Sankaran, Rajesh; Beckman, Pete; Collis, Scott M.; Ferrier, Nicola (March 2023, Artificial Intelligence for the Earth Systems)

   Abstract Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ∼ 85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Furthermore, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks,suggesting a new alternative for cloud characterization. 

   more » « less
5. Machine learning identifies abnormal Ca2+ transients in human induced pluripotent stem cell-derived cardiomyocytes

   https://doi.org/10.1038/s41598-020-73801-x  

   Hwang, Hyun; Liu, Rui; Maxwell, Joshua T.; Yang, Jingjing; Xu, Chunhui (October 2020, Scientific Reports)

   Abstract Human-induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs) provide an excellent platform for potential clinical and research applications. Identifying abnormal Ca²⁺transients is crucial for evaluating cardiomyocyte function that requires labor-intensive manual effort. Therefore, we develop an analytical pipeline for automatic assessment of Ca²⁺transient abnormality, by employing advanced machine learning methods together with an Analytical Algorithm. First, we adapt an existing Analytical Algorithm to identify Ca²⁺transient peaks and determine peak abnormality based on quantified peak characteristics. Second, we train a peak-level Support Vector Machine (SVM) classifier by using human-expert assessment of peak abnormality as outcome and profiled peak variables as predictive features. Third, we train another cell-level SVM classifier by using human-expert assessment of cell abnormality as outcome and quantified cell-level variables as predictive features. This cell-level SVM classifier can be used to assess additional Ca²⁺transient signals. By applying this pipeline to our Ca²⁺transient data, we trained a cell-level SVM classifier using 200 cells as training data, then tested its accuracy in an independent dataset of 54 cells. As a result, we obtained 88% training accuracy and 87% test accuracy. Further, we provide a free R package to implement our pipeline for high-throughput CM Ca²⁺analysis. 

   more » « less