More Like this

1. Selection via Proxy: Efficient Data Selection for Deep Learning

   Coleman, C.; Yeh, C.; Mussmann, S.; Mirzasoleiman, B.; Bailis, P.; Liang, P.; Leskovec, J.; Zaharia, M. (April 2020, International Conference on Learning Representations (ICLR))

   Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this “selection via proxy” (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10 faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1:6 end-to-end training time improvement. 

   more » « less
2. MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

   Wang, T; Gui, D; Hu, Y; Lin, S; Zhang, L (July 2025, Proceedings of Machine Learning Research)

   Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning large language models (LLMs). Yet its reliance on a singular reward model often overlooks the diversity of human preferences. Recent approaches address this limitation by leveraging multi-dimensional feedback to fine-tune corresponding reward models and train LLMs using reinforcement learning. However, the process is costly and unstable, especially given the competing and heterogeneous nature of human preferences. In this paper, we propose Mixing Preference Optimization (MPO), a post-processing framework for aggregating single-objective policies as an alternative to both multi-objective RLHF (MORLHF) and MaxMin-RLHF. MPO avoids alignment from scratch. Instead, it log-linearly combines existing policies into a unified one with the weight of each policy computed via a batch stochastic mirror descent. Empirical results demonstrate that MPO achieves balanced performance across diverse preferences, outperforming or matching existing models with significantly reduced computational costs. 

   more » « less
3. Optimizing Attention and Cognitive Control Costs Using Temporally Layered Architectures

   https://doi.org/10.1162/neco_a_01718  

   Patel, Devdhar; Sejnowski, Terrence; Siegelmann, Hava (November 2024, Neural Computation)

   Abstract The current reinforcement learning framework focuses exclusively on performance, often at the expense of efficiency. In contrast, biological control achieves remarkable performance while also optimizing computational energy expenditure and decision frequency. We propose a decision-bounded Markov decision process (DB-MDP) that constrains the number of decisions and computational energy available to agents in reinforcement learning environments. Our experiments demonstrate that existing reinforcement learning algorithms struggle within this framework, leading to either failure or suboptimal performance. To address this, we introduce a biologically inspired, temporally layered architecture (TLA), enabling agents to manage computational costs through two layers with distinct timescales and energy requirements. TLA achieves optimal performance in decision-bounded environments and in continuous control environments, matching state-of-the-art performance while using a fraction of the computing cost. Compared to current reinforcement learning algorithms that solely prioritize performance, our approach significantly lowers computational energy expenditure while maintaining performance. These findings establish a benchmark and pave the way for future research on energy and time-aware control. 

   more » « less
4. Reinforcement learning beyond the Bellman equation: Exploring critic objectives using evolution

   https://doi.org/10.1162/isal_a_00338  

   Leite, Abe; Candadai, Madhavun; Izquierdo, Eduardo J. (January 2020, ALIFE 2020: The 2020 Conference on Artificial Life)

   Living organisms learn on multiple time scales: evolutionary as well as individual-lifetime learning. These two learning modes are complementary: the innate phenotypes developed through evolution significantly influence lifetime learning. However, it is still unclear how these two learning methods interact and whether there is a benefit to part of the system being optimized on a different time scale using a population-based approach while the rest of it is trained on a different time-scale using an individualistic learning algorithm. In this work, we study the benefits of such a hybrid approach using an actor-critic framework where the critic part of an agent is optimized over evolutionary time based on its ability to train the actor part of an agent during its lifetime. Typically, critics are optimized on the same time-scale as the actor using the Bellman equation to represent long-term expected reward. We show that evolution can find a variety of different solutions that can still enable an actor to learn to perform a behavior during its lifetime. We also show that although the solutions found by evolution represent different functions, they all provide similar training signals during the lifetime. This suggests that learning on multiple time-scales can effectively simplify the overall optimization process in the actor-critic framework by finding one of many solutions that can still train an actor just as well. Furthermore, analysis of the evolved critics can yield additional possibilities for reinforcement learning beyond the Bellman equation. 

   more » « less
5. Adjoint-based machine learning for active flow control

   https://doi.org/10.1103/PhysRevFluids.9.013901  

   Liu, Xuemin; MacArt, Jonathan F (January 2024, Physical Review Fluids)

   We develop neural-network active flow controllers using a deep learning partial differential equation augmentation method (DPM). The end-to-end sensitivities for optimization are computed using adjoints of the governing equations without restriction on the terms that may appear in the objective function. In one-dimensional Burgers' examples with analytic (manufactured) control functions, DPM-based control is comparably effective to standard supervised learning for in-sample solutions and more effective for out-of-sample solutions, i.e., with different analytic control functions. The influence of the optimization time interval and neutral-network width is analyzed, the results of which influence algorithm design and hyperparameter choice, balancing control efficacy with computational cost. We subsequently develop adjoint-based controllers for two flow scenarios. First, we compare the drag-reduction performance and optimization cost of adjoint-based controllers and deep reinforcement learning (DRL)-based controllers for two-dimensional, incompressible, confined flow over a cylinder at Re=100, with control achieved by synthetic body forces along the cylinder boundary. The required model complexity for the DRL-based controller is 4229 times that required for the DPM-based controller. In these tests, the DPM-based controller is 4.85 times more effective and 63.2 times less computationally intensive to train than the DRL-based controller. Second, we test DPM-based control for compressible, unconfined flow over a cylinder and extrapolate the controller to out-of-sample Reynolds numbers. We also train a simplified, steady, offline controller based on the DPM control law. Both online (DPM) and offline (steady) controllers stabilize the vortex shedding with a 99% drag reduction, demonstrating the robustness of the learning approach. For out-of-sample flows (Re={50,200,300,400}), both the online and offline controllers successfully reduce drag and stabilize vortex shedding, indicating that the DPM-based approach results in a stable model. A key attractive feature is the flexibility of adjoint-based optimization, which permits optimization over arbitrarily defined control laws without the need to match a priori known functions. 

   more » « less