More Like this

1. No Preference Left Behind: Group Distributional Preference Optimization

   Yao, B; Cai, Z; Chuang, Y S; Yang, S; Jiang, M; Yang, D; Hu, J (April 2025, The Thirteenth International Conference on Learning Representations.)

   Preferences within a group of people are not uniform but follow a distribution. While existing alignment methods like Direct Preference Optimization (DPO) attempt to steer models to reflect human preferences, they struggle to capture the distributional pluralistic preferences within a group. These methods often skew toward dominant preferences, overlooking the diversity of opinions, especially when conflicting preferences arise. To address this issue, we propose Group Distributional Preference Optimization (GDPO), a novel framework that aligns language models with the distribution of preferences within a group by incorporating the concept of beliefs that shape individual preferences. GDPO calibrates a language model using statistical estimation of the group's belief distribution and aligns the model with belief-conditioned preferences, offering a more inclusive alignment framework than traditional methods. In experiments using both synthetic controllable opinion generation and real-world movie review datasets, we show that DPO fails to align with the targeted belief distributions, while GDPO consistently reduces this alignment gap during training. Additionally, our evaluation metrics demonstrate that GDPO outperforms existing approaches in aligning with group distributional preferences, marking a significant advance in pluralistic alignment. 

   more » « less
2. Inverse Preference Learning: Preference-based RL without a Reward Function

   Hejna, Joey; Sadigh, Dorsa (December 2023, Conference on Neural Information Processing Systems (NeurIPS))
3. DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback

   Xiong, Guojun; Dinesha, Ujwal; Mukherjee, Debajoy; Li, Jian; Shakkottai, Srinivas (April 2025, The Thirteenth International Conference on Learning Representations (ICLR 2025))

   Restless multi-armed bandits (RMAB) has been widely used to model constrained sequential decision making problems, where the state of each restless arm evolves according to a Markov chain and each state transition generates a scalar reward. However, the success of RMAB crucially relies on the availability and quality of reward signals. Unfortunately, specifying an exact reward function in practice can be challenging and even infeasible. In this paper, we introduce Pref-RMAB, a new RMAB model in the presence of preference signals, where the decision maker only observes pairwise preference feedback rather than scalar reward from the activated arms at each decision epoch. Preference feedback, however, arguably contains less information than the scalar reward, which makes Pref-RMAB seemingly more difficult. To address this challenge, we present a direct online preference learning (DOPL) algorithm for Pref-RMAB to efficiently explore the unknown environments, adaptively collect preference data in an online manner, and directly leverage the preference feedback for decision-makings. We prove that DOPL yields a sublinear regret. To our best knowledge, this is the first algorithm to ensure $$\tilde{\mathcal{O}}(\sqrt{T\ln T})$$ regret for RMAB with preference feedback. Experimental results further demonstrate the effectiveness of DOPL. 

   more » « less
4. Preference-Based Privacy Markets

   https://doi.org/10.1109/ACCESS.2020.3014882  

   Pal, Ranjan; Crowcroft, Jon; Wang, Yixuan; Li, Yong; De, Swades; Tarkoma, Sasu; Liu, Mingyan; Nag, Bodhibrata; Kumar, Abhishek; Hui, Pan (January 2020, IEEE Access)

   null (Ed.)
5. Preference-Based Image Generation

   https://doi.org/10.1109/WACV45572.2020.9093406  

   Kazemi, Hadi; Taherkhani, Fariborz; Nasrabadi, Nasser M. (March 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV))

   null (Ed.)