More Like this

1. Dual Contradistinctive Generative Autoencoder

   Parmar, Gaurav; Li, Dacheng; Lee, Kwonjoon; Tu, Zhuowen (June 2021, IEEE Computer Society Conference on Computer Vision and Pattern Recognition)

   null (Ed.)
2. Kernel-elastic autoencoder for molecular design

   https://doi.org/10.1093/pnasnexus/pgae168  

   Li, Haote; Shee, Yu; Allen, Brandon; Maschietto, Federica; Morgunov, Anton; Batista, Victor (March 2024, PNAS Nexus)

   Amon, Cristina (Ed.)

   Abstract We introduce the kernel-elastic autoencoder (KAE), a self-supervised generative model based on the transformer architecture with enhanced performance for molecular design. KAE employs two innovative loss functions: modified maximum mean discrepancy (m-MMD) and weighted reconstruction (LWCEL). The m-MMD loss has significantly improved the generative performance of KAE when compared to using the traditional Kullback–Leibler loss of VAE, or standard maximum mean discrepancy. Including the weighted reconstruction loss LWCEL, KAE achieves valid generation and accurate reconstruction at the same time, allowing for generative behavior that is intermediate between VAE and autoencoder not available in existing generative approaches. Further advancements in KAE include its integration with conditional generation, setting a new state-of-the-art benchmark in constrained optimizations. Moreover, KAE has demonstrated its capability to generate molecules with favorable binding affinities in docking applications, as evidenced by AutoDock Vina and Glide scores, outperforming all existing candidates from the training dataset. Beyond molecular design, KAE holds promise to solve problems by generation across a broad spectrum of applications. 

   more » « less
3. KATE: K-Competitive Autoencoder for Text

   https://doi.org/10.1145/3097983.3098017  

   Chen, Yu; Zaki, Mohammed J. (August 2017, Proceedings of the ACM SIGKDD International Conference on Data Mining and Knowledge Discovery)
4. Guided Variational Autoencoder for Disentanglement Learning

   Ding, Zheng; Xu, Yifan; Xu, Weijian; Parmar, Gaurav; Yang, Yang Yang; Welling, Max; Tu, Zhuowen (June 2020, IEEE Computer Society Conference on Computer Vision and Pattern Recognition)

   We propose an algorithm, guided variational autoencoder (Guided-VAE), that is able to learn a controllable generative model by performing latent representation disentanglement learning. The learning objective is achieved by providing signals to the latent encoding/embedding in VAE without changing its main backbone architecture, hence retaining the desirable properties of the VAE. We design an unsupervised strategy and a supervised strategy in Guided-VAE and observe enhanced modeling and controlling capability over the vanilla VAE. In the unsupervised strategy, we guide the VAE learning by introducing a lightweight decoder that learns latent geometric transformation and principal components; in the supervised strategy, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement of the latent variables. Guided-VAE enjoys its transparency and simplicity for the general representation learning task, as well as disentanglement learning. On a number of experiments for representation learning, improved synthesis/sampling, better disentanglement for classification, and reduced classification errors in meta learning have been observed. 

   more » « less
5. Analyzing autoencoder-based acoustic word embeddings

   Matusevych, Yevgen; Kamper, Herman; Goldwater, Sharon (April 2020, ICLR Workshop on Bridging AI and Cognitive Science)

   Recent studies have introduced methods for learning acoustic word embeddings (AWEs)—fixed-size vector representations of words which encode their acoustic features. Despite the widespread use of AWEs in speech processing research, they have only been evaluated quantitatively in their ability to discriminate between whole word tokens. To better understand the applications of AWEs in various downstream tasks and in cognitive modeling, we need to analyze the representation spaces of AWEs. Here we analyze basic properties of AWE spaces learned by a sequence-to-sequence encoder-decoder model in six typologically diverse languages. We first show that these AWEs preserve some information about words’ absolute duration and speaker. At the same time, the representation space of these AWEs is organized such that the distance between words’ embeddings increases with those words’ phonetic dissimilarity. Finally, the AWEs exhibit a word onset bias, similar to patterns reported in various studies on human speech processing and lexical access. We argue this is a promising result and encourage further evaluation of AWEs as a potentially useful tool in cognitive science, which could provide a link between speech processing and lexical memory. 

   more » « less