More Like this

1. Interpretation of comparative LCAs: external normalization and a method of mutual differences

   https://doi.org/10.1007/s11367-017-1281-3  

   Prado, Valentina; Wender, Ben A.; Seager, Thomas P. (December 2017, The International Journal of Life Cycle Assessment)
2. Attentive Normalization

   https://doi.org/10.1007/978-3-030-58520-4_5  

   Li, X.; Sun, W.; Wu, T. (August 2020, 16th European Conference on Computer Vision (2020))

   Vedaldi, A. (Ed.)

   In state-of-the-art deep neural networks, both feature normalization and feature attention have become ubiquitous. They are usually studied as separate modules, however. In this paper, we propose a light-weight integration between the two schema and present Attentive Normalization (AN). Instead of learning a single affine transformation, AN learns a mixture of affine transformations and utilizes their weighted-sum as the final affine transformation applied to re-calibrate features in an instance-specific way. The weights are learned by leveraging channel-wise feature attention. In experiments, we test the proposed AN using four representative neural architectures. In the ImageNet-1000 classification benchmark and the MS-COCO 2017 object detection and instance segmentation benchmark. AN obtains consistent performance improvement for different neural architectures in both benchmarks with absolute increase of top-1 accuracy in ImageNet-1000 between 0.5\% and 2.7\%, and absolute increase up to 1.8\% and 2.2\% for bounding box and mask AP in MS-COCO respectively. We observe that the proposed AN provides a strong alternative to the widely used Squeeze-and-Excitation (SE) module. The source codes are publicly available at \href{https://github.com/iVMCL/AOGNet-v2}{the ImageNet Classification Repo} and \href{https://github.com/iVMCL/AttentiveNorm\_Detection}{the MS-COCO Detection and Segmentation Repo}. 

   more » « less
3. On Separate Normalization in Self-supervised Transformers

   Chen, X; Wang, Y; Du, Y; Hassoun, S; Liu, L (December 2023, Curran Associates)

   Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the class token [CLS] and the tokens. We propose in this paper a new yet simple normalization method that separately normalizes embedding vectors respectively corresponding to normal tokens and the [CLS] token, in order to better capture their distinct characteristics and enhance downstream task performance. Our empirical study shows that the [CLS] embeddings learned with our separate normalization layer better encode the global contextual information and are distributed more uniformly in its anisotropic space. When the conventional normalization layer is replaced with a separate normalization layer, we observe an average 2.7% performance improvement in learning tasks from the image, natural language, and graph domains. 

   more » « less
4. On Separate Normalization in Self-supervised Transformers

   Chen, Xiaohui; Wang, Yinkai; Du, Yuanqi; Hassoun, Soha; Liu, Li-Ping (December 2023, Advances in Neural Information Processing Systems 36)

   Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains. 

   more » « less
5. The Normalization Barrier Revisited

   https://doi.org/10.1145/3694848.3694851  

   Ding, Shuo; Zhang, Qirun (October 2024, ACM)