More Like this

1. Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

   Yan, Shen; Zheng, Yu; Ao, Wei; Zeng, Xiao; Zhang, Mi (January 2020, Conference on Neural Information Processing Systems)

   Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels improves the downstream architecture search efficiency. To explain this finding, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies. 

   more » « less
2. Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

   Yan, Shen; Zheng, Yu; Ao, Wei; Zeng, Xiao; Zhang, Mi (January 2020, Conference on Neural Information Processing Systems (NeurIPS'20))

   null (Ed.)
3. Persistent Processor Architecture

   https://doi.org/10.1145/3613424.3623772  

   Zeng, Jianping; Jeong, Jungi; Jung, Changhee (October 2023, IEEE/ACM International Symposium on Microarchitecture)
4. AHomogeneous Transformer Architecture

   Gan, Yulu; Poggio, Tomaso (September 2023, Center for Brains, Minds and Machines (CBMM))

   While the Transformer architecture has made a substantial impact in the field of machine learning, it is unclear what purpose each component serves in the overall architecture. Heterogeneous nonlinear circuits such as multi-layer RELU networks are interleaved with layers of soft-max units. We introduce here a homogeneous architecture based on Hyper Radial Basis Function (HyperBF) units. Evalua- tions on CIFAR10, CIFAR100, and Tiny ImageNet demonstrate a performance comparable to standard vision transformers. 

   more » « less
5. The SuperCodelet architecture

   Jose M Monsalve Diaz, Kevin Harms (January 2022, The 1st International Workshop on Extreme Heterogeneity Solutions (ExHet’ 22, in conjunction with PPoPP’22)