More Like this

1. Visual Prompt Tuning

   https://doi.org/10.1007/978-3-031-19827-4_41  

   Jia, Menglin; Tang, Luming; Chen, Bor-Chun; Cardie, Claire; Belongie, Serge; Hariharan, Bharath; Lim, Ser-Nam (October 2022, European Conference on Computer Vision)

   The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost. 

   more » « less
2. Pareto Prompt Optimization

   Zhao, Guang; Yoon, Byung_Jun; Park, Gilchan; Jha, Shantenu; Yoo, Shinjae; Qian, Xiaoning (April 2025, 13th International Conference on Learning Representations (ICLR 2025))
3. Pareto Prompt Optimization

   Zhao, Guang; Yoon, Byung-Jun; Park, Gilchan; Jha, Shantenu; Yoo, Shinjae; Qian, Xiaoning (April 2025, The Monticello gazette)
4. Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation

   Jain, Abhinav; Chaudhuri, Swarat; Reps, Thomas W; Jermaine, Christopher M (December 2024, http://papers.nips.cc/paper_files/paper/2024/hash/548551c07a68c8f0a87d67c6167cedb1-Abstract-Conference.html)

   Parameter-Efficient Fine-Tuning (PEFT) has become the standard for customising Foundation Models (FMs) to user-specific downstream tasks. However, typical PEFT methods require storing multiple task-specific adapters, creating scalability issues as these adapters must be housed and run at the FM server. Traditional prompt tuning offers a potential solution by customising them through task-specific input prefixes, but it under-performs compared to other PEFT methods like LoRA. To address this gap, we propose Low-Rank Prompt Adaptation (LoPA), a prompttuning-based approach that performs on par with state-of-the-art PEFT methods and full fine-tuning while being more parameter-efficient and not requiring a server-based adapter. LoPA generates soft prompts by balancing between sharing task-specific information across instances and customization for each instance. It uses a low-rank decomposition of the soft-prompt component encoded for each instance to achieve parameter efficiency. We provide a comprehensive evaluation on multiple natural language understanding and code generation and understanding tasks across a wide range of foundation models with varying sizes. 

   more » « less
5. Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation

   Jain, Abhinav; Chaudhuri, Swarat; Reps, Thomas; Jermaine, Christopher (December 2024, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024)