More Like this

1. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

   Sheng Liu; Xiao Li; Yuexiang Zhai; Chong You; Zhihui Zhu; Carlos Fernandez-Granda; Qing Qu (January 2021, Neural Information Processing Systems (NeurIPS))
2. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

   Liu, Sheng; Li, Xiao; Zhai, Yuexiang; You, Chong; Zhu, Zhihui; Fernandez-Granda, Carlos; Qu, Qing (January 2021, Advances in neural information processing systems)
3. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

   Liu, Sheng; Li, Xiao; Zhai, Yuexiang; You, Chong; Zhu, Zhihui; Fernandez-Granda, Carlos; Qu, Qing (January 2021, Advances in neural information processing systems)

   Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets). In particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convolutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. In contrast, we introduce a simple and efficient Convolutional Normalization'' (ConvNorm) method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to be conveniently incorporated into any ConvNets. Our method is inspired by recent work on preconditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), we show that the ConvNorm improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via numerical experiments on CIFAR and ImageNet. Our implementation is available online at \url{https://github.com/shengliu66/ConvNorm}. 

   more » « less
4. Kriging Convolutional Networks

   https://doi.org/10.1609/aaai.v34i04.5716  

   Appleby, Gabriel; Liu, Linfeng; Liu, Li-Ping (June 2020, Proceedings of the AAAI Conference on Artificial Intelligence)

   null (Ed.)

   Spatial interpolation is a class of estimation problems where locations with known values are used to estimate values at other locations, with an emphasis on harnessing spatial locality and trends. Traditional kriging methods have strong Gaussian assumptions, and as a result, often fail to capture complexities within the data. Inspired by the recent progress of graph neural networks, we introduce Kriging Convolutional Networks (KCN), a method of combining advantages of Graph Neural Networks (GNN) and kriging. Compared to standard GNNs, KCNs make direct use of neighboring observations when generating predictions. KCNs also contain the kriging method as a specific configuration. Empirically, we show that this model outperforms GNNs and kriging in several applications. 

   more » « less
5. Evenly Cascaded Convolutional Networks

   https://doi.org/10.1109/BigData.2018.8622196  

   Ye, Chengxi; Devaraj, Chinmaya; Maynord, Michael; Fermuller, Cornelia; Aloimonos, Yiannis (December 2018, 2018 IEEE International Conference on Big Data)

   We introduce Evenly Cascaded convolutional Network (ECN), a neural network taking inspiration from the cascade algorithm of wavelet analysis. ECN employs two feature streams - a low-level and high-level steam. At each layer these streams interact, such that low-level features are modulated using advanced perspectives from the high-level stream. ECN is evenly structured through resizing feature map dimensions by a consistent ratio, which removes the burden of ad-hoc specification of feature map dimensions. ECN produces easily interpretable features maps, a result whose intuition can be understood in the context of scale-space theory. We demonstrate that ECN's design facilitates the training process through providing easily trainable shortcuts. We report new state-of-the-art results for small networks, without the need for additional treatment such as pruning or compression - a consequence of ECN's simple structure and direct training. A 6-layered ECN design with under 500k parameters achieves 95.24% and 78.99% accuracy on CIFAR-10 and CIFAR-100 datasets, respectively, outperforming the current state-of-the-art on small parameter networks, and a 3 million parameter ECN produces results competitive to the state-of-the-art. 

   more » « less