More Like this

1. SciTokens SSH: Token-based Authentication for Remote Login to Scientific Computing Environments

   https://doi.org/10.1145/3311790.3399613  

   Gao, You Alex; Basney, Jim; Withers, Alex (January 2020, Practice and Experience in Advanced Research Computing)

   SciTokens SSH is a pluggable authentication module (PAM) that uses JSON Web Tokens (JWTs) for authentication to the Secure Shell (SSH) remote login service. SciTokens SSH supports multiple token issuers with local token verification, so scientific computing providers are not forced to rely on a single OAuth server for token issuance and verification. The decentralized design for SciTokens SSH was motivated by the distributed nature of scientific computing environments, where scientists use computational resources from multiple providers, with a variety of security policies, distributed across the globe. 

   more » « less
2. Simple Token-Level Confidence Improves Caption Correctness

   https://doi.org/10.1109/WACV57701.2024.00564  

   Petryk, Suzanne; Whitehead, Spencer; Gonzalez, Joseph E; Darrell, Trevor; Rohrbach, Anna; Rohrbach, Marcus (January 2024, IEEE)
3. Parameter-Efficient Tuning with Special Token Adaptation

   Yang, Xiaocong Yang; Huang, James Y.; Zhou, Wenxuan; Chen, Muhao (May 2023, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL))

   Vlachos, Andreas; Augenstein, Isabelle (Ed.)

   Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models. 

   more » « less
4. MambaByte: Token-free Selective State Space Model

   Wang, Junxiong; Gangavarapu, Tushaar; Yan, Jing Nathan; Rush, Alexander M (October 2024, COLM)

   Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a 2.6× inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling. 

   more » « less
5. Token Sparsification for Faster Medical Image Segmentation

   https://doi.org/10.1007/978-3-031-34048-2_57  

   Zhou, L; Liu, H.; Bae, J.; He, J.; Samaras, D.; Prasanna, P. (June 2023, Information Processing in Medical Imaging. IPMI 2023.)