More Like this

1. Grounded Sequence to Sequence Transduction

   https://doi.org/10.1109/JSTSP.2020.2998415  

   Lucia Specia, Loıc Barrault (July 2020, IEEE journal of selected topics in signal processing)

   null (Ed.)

   Speech recognition and machine translation have made major progress over the past decades, providing practical systems to map one language sequence to another. Although multiple modalities such as sound and video are becoming increasingly available, the state-of-the-art systems are inherently unimodal, in the sense that they take a single modality --- either speech or text --- as input. Evidence from human learning suggests that additional modalities can provide disambiguating signals crucial for many language tasks. Here, we describe the dataset, a large, open-domain collection of videos with transcriptions and their translations. We then show how this single dataset can be used to develop systems for a variety of language tasks and present a number of models meant as starting points. Across tasks, we find that building multi-modal architectures that perform better than their unimodal counterpart remains a challenge. This leaves plenty of room for the exploration of more advanced solutions that fully exploit the multi-modal nature of the dataset, and the general direction of multimodal learning with other datasets as well. 

   more » « less
2. Sequence-to-sequence AMR Parsing with Ancestor Information

   https://doi.org/10.18653/v1/2022.acl-short.63  

   Yu, Chen; Gildea, Daniel (January 2022, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics)
3. Deep Reinforcement Learning for Sequence-to-Sequence Models

   https://doi.org/10.1109/TNNLS.2019.2929141  

   Keneshloo, Yaser; Shi, Tian; Ramakrishnan, Naren; Reddy, Chandan K. (July 2020, IEEE Transactions on Neural Networks and Learning Systems)

   In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide stateof-the-art performance in a wide variety of tasks, such as machine translation, headline generation, text summarization, speech-to-text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoder–decoder models produce competitive results, many researchers have proposed additional improvements over these seq2seq models, e.g., using an attention-based model over the input, pointer-generation models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with seq2seq models that enable remembering long-term memories. We present some of the most recent frameworks that combine the concepts from RL and deep neural networks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization and provide some targeted experiments for these RL models, both in terms of performance and training time. 

   more » « less
4. SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair

   https://doi.org/10.1109/TSE.2019.2940179  

   Chen, Zimin; Kommrusch, Steve James; Tufano, Michele; Pouchet, Louis-Noel; Poshyvanyk, Denys; Monperrus, Martin (September 2019, IEEE Transactions on Software Engineering)

   This paper presents a novel end-to-end approach to program repair based on sequence-to-sequence learning. We devise, implement, and evaluate a technique, called SEQUENCER, for fixing bugs based on sequence-to-sequence learning on source code. This approach uses the copy mechanism to overcome the unlimited vocabulary problem that occurs with big code. Our system is data-driven; we train it on 35,578 samples, carefully curated from commits to open-source repositories. We evaluate SEQUENCER on 4,711 independent real bug fixes, as well on the Defects4J benchmark used in program repair research. SEQUENCER is able to perfectly predict the fixed line for 950/4,711 testing samples, and find correct patches for 14 bugs in Defects4J benchmark. SEQUENCER captures a wide range of repair operators without any domain-specific top-down design. 

   more » « less
5. Sequence-to-sequence translation from mass spectra to peptides with a transformer model

   https://doi.org/10.1038/s41467-024-49731-x  

   Yilmaz, Melih; Fondrie, William_E; Bittremieux, Wout; Melendez, Carlo_F; Nelson, Rowan; Ananth, Varun; Oh, Sewoong; Noble, William_Stafford (July 2024, Nature Communications)

   Abstract A fundamental challenge in mass spectrometry-based proteomics is the identification of the peptide that generated each acquired tandem mass spectrum. Approaches that leverage known peptide sequence databases cannot detect unexpected peptides and can be impractical or impossible to apply in some settings. Thus, the ability to assign peptide sequences to tandem mass spectra without prior information—de novo peptide sequencing—is valuable for tasks including antibody sequencing, immunopeptidomics, and metaproteomics. Although many methods have been developed to address this problem, it remains an outstanding challenge in part due to the difficulty of modeling the irregular data structure of tandem mass spectra. Here, we describe Casanovo, a machine learning model that uses a transformer neural network architecture to translate the sequence of peaks in a tandem mass spectrum into the sequence of amino acids that comprise the generating peptide. We train a Casanovo model from 30 million labeled spectra and demonstrate that the model outperforms several state-of-the-art methods on a cross-species benchmark dataset. We also develop a version of Casanovo that is fine-tuned for non-enzymatic peptides. Finally, we demonstrate that Casanovo’s superior performance improves the analysis of immunopeptidomics and metaproteomics experiments and allows us to delve deeper into the dark proteome. 

   more » « less