Search for: All records

Abstract Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems. 

accepted 

Gas Metal Arc Welding (GMAW) is a critical industrial technique known for its high productivity, flexibility, and adaptability to automation. Despite the significant advancements in robotic welding, challenges remain in fully automating the arc welding process, particularly due to the complex dynamics of the weld pool associated with GMAW. A human-robot collaborative (HRC) system where humans operate robots may conveniently provide the needed adaptive control to the complex GMAW. While in conventional HRC systems humans receive process feedback to make adaptive adjustments, we propose provide humans with predictive future feedback to further ease the human decision and reduce the needed skills/trainings. To this end, this study explores the integration of deep learning models, specifically Generative Adversarial Networks (GANs) combined with Gated Recurrent Units (GRUs), to model and predict the dynamic behavior of the weld pool during GMAW. By leveraging time-series data of torch movement and corresponding weld pool images, the proposed GRU-GAN model generates high-fidelity weld pool images, capturing the intricate relationship between speed variations and weld pool morphology. Through extensive experimentation, including the design of an acceptable Encoder-Decoder structure for the GAN, we demonstrate that incorporating both temporal and speed sequence information significantly enhances the model's predictive capabilities. The findings validate the hypothesis that dynamic torch speed adjustments, akin to those performed by skilled human welders, can be effectively modeled to improve the quality of automated welding processes. Future work will be devoted to human-based model predictive control (MPC) in an HRC environment. 

Deep-learning models have allowed effective end-to-end SE systems in the Speech Enhancement (SE) field. Most of these methods are trained using a fixed reconstruction loss in a supervised setting. Often these losses do not perfectly represent the desired perceptual quality metrics, resulting in sub-optimal performance. Recently, there have been efforts to learn the behavior of those metrics directly via neural nets for training SE models. However, an accurate estimation of the true metric function introduces statistical complexity for training because it attempts to capture the exact value of the metric. We propose an adversarial training strategy based on statistical correlation that avoids the complexity of estimating the SE metric while learning to mimic its overall behavior. We call this framework CorrGAN and show its significant improvement over standard losses of the SOTA baselines and achieve SOTA performance on the VoiceBank+DEMAND dataset. 

In recent years, using orthogonal matrices has been shown to be a promising approach to improving recurrent neural networks (RNNs) with training, stability, and convergence, particularly to control gradients. While gated recurrent unit (GRU) and long short-term memory (LSTM) architectures address the vanishing gradient problem by using a variety of gates and memory cells, they are still prone to the exploding gradient problem. In this work, we analyze the gradients in GRU and propose the use of orthogonal matrices to prevent exploding gradient problems and enhance long-term memory. We study where to use orthogonal matrices and propose a Neumann series–based scaled Cayley transformation for training orthogonal matrices in GRU, which we call Neumann-Cayley orthogonal GRU (NC-GRU). We present detailed experiments of our model on several synthetic and real-world tasks, which show that NC-GRU significantly outperforms GRU and several other RNNs. 

Long-term time-series forecasting remains challenging due to the difficulty in capturing long-term dependencies, achieving linear scalability, and maintaining computational efficiency. We introduce TimeMachine, an innovative model that leverages Mamba, a state-space model, to capture long-term dependencies in multivariate time series data while maintaining linear scalability and small memory footprints. TimeMachine exploits the unique properties of time series data to produce salient contextual cues at multi-scales and leverage an innovative integrated quadruple-Mamba architecture to unify the handling of channel-mixing and channel-independence situations, thus enabling effective selection of contents for prediction against global and local contexts at different scales. Experimentally, TimeMachine achieves superior performance in prediction accuracy, scalability, and memory efficiency, as extensively validated using benchmark datasets. Code availability: https://github.com/Atik-Ahamed/TimeMachine