More Like this

1. Simulating Hard Attention Using Soft Attention

   https://doi.org/10.1162/TACL.a.597  

   Yang, Andy; Strobl, Lena; Chiang, David; Angluin, Dana (January 2026, Transactions of the Association for Computational Linguistics)

   We study conditions under which transformers using soft attention can simulate hard attention, that is, effectively focus all attention on a subset of positions. First, we examine several subclasses of languages recognized by hard-attention transformers, which can be defined in variants of linear temporal logic. We demonstrate how soft-attention transformers can compute formulas of these logics using unbounded positional embeddings or temperature scaling. Second, we demonstrate how temperature scaling allows softmax transformers to simulate general hard-attention transformers, using a temperature that depends on the minimum gap between the maximum attention scores and other attention scores. 

   more » « less
2. Predicting Human Attention using Computational Attention

   Yang, Zhibo; Mondal, Sounak; Ahn, Seoyoung; Zelinsky, Gregory; Hoai, Minh; Samaras, Dimitris (January 2023, Computer Vision and Pattern Recognition)
3. Attention to Action: Leveraging Attention for Object Navigation

   Chen, Shi; Zhao, Qi (January 2021, The British Machine Vision Conference)
4. Boosted Attention: Leveraging Human Attention for Image Captioning

   https://doi.org/10.1007/978-3-030-01252-6_5  

   Chen, Shi; Zhao, Qi (September 2018, European Conference on Computer Vision)
5. Attention without Constraint: Alpha Lateralization in Uncued Willed Attention

   https://doi.org/10.1523/ENEURO.0258-22.2023  

   Nadra, John G; Bengson, Jesse J; Morales, Alexander B; Mangun, George R (June 2023, eneuro)

   Abstract Studies of voluntary visual spatial attention have used attention-directing cues, such as arrows, to induce or instruct observers to focus selective attention on relevant locations in visual space to detect or discriminate subsequent target stimuli. In everyday vision, however, voluntary attention is influenced by a host of factors, most of which are quite different from the laboratory paradigms that use attention-directing cues. These factors include priming, experience, reward, meaning, motivations, and high-level behavioral goals. Attention that is endogenously directed in the absence of external attention-directing cues has been referred to as “self-initiated attention” or, as in our prior work, as “willed attention” where volunteers decide where to attend in response to a prompt to do so. Here, we used a novel paradigm that eliminated external influences (i.e., attention-directing cues and prompts) about where and/or when spatial attention should be directed. Using machine learning decoding methods, we showed that the well known lateralization of EEG alpha power during spatial attention was also present during purely self-generated attention. By eliminating explicit cues or prompts that affect the allocation of voluntary attention, this work advances our understanding of the neural correlates of attentional control and provides steps toward the development of EEG-based brain–computer interfaces that tap into human intentions. 

   more » « less