More Like this

1. Visual Instruction Inversion: Image Editing via Visual Prompting

   Nguyen, T; Li, Y; Ojha, U; Lee, Yong Jae (December 2023, Neural Information Processing Systems (NeurIPS))
2. Visual Remapping

   https://doi.org/10.1146/annurev-vision-032321-100012  

   Golomb, Julie D.; Mazer, James A. (September 2021, Annual Review of Vision Science)

   null (Ed.)

   Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates. 

   more » « less
3. Visual Instruction Tuning

   Liu, H; Li, C; Wu, Q; Lee, Yong Jae (December 2023, Neural Information Processing Systems (NeurIPS))
4. Visual Acoustic Matching

   Changan Chen, Ruohan Gao (June 2022, arXivorg)
5. Visual Prompt Tuning

   https://doi.org/10.1007/978-3-031-19827-4_41  

   Jia, Menglin; Tang, Luming; Chen, Bor-Chun; Cardie, Claire; Belongie, Serge; Hariharan, Bharath; Lim, Ser-Nam (October 2022, European Conference on Computer Vision)

   The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost. 

   more » « less