More Like this

1. Bayesian Reinforcement Learning in Factored POMDPs

   Katt, Sammie; Oliehoek, Frans A.; Amato, Christopher (January 2019, Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems)

   Model-based Bayesian Reinforcement Learning (BRL) provides a principled solution to dealing with the exploration-exploitation trade-off, but such methods typically assume a fully observable environments. The few Bayesian RL methods that are applicable in partially observable domains, such as the Bayes-Adaptive POMDP (BA-POMDP), scale poorly. To address this issue, we introduce the Factored BA-POMDP model (FBA-POMDP), a framework that is able to learn a compact model of the dynamics by exploiting the underlying structure of a POMDP. The FBA-POMDP framework casts the problem as a planning task, for which we adapt the Monte-Carlo Tree Search planning algorithm and develop a belief tracking method to approximate the joint posterior over the state and model variables. Our empirical results show that this method outperforms a number of BRL baselines and is able to learn efficiently when the factorization is known, as well as learn both the factorization and the model parameters simultaneously. 

   more » « less
2. Adapting World Models with Latent-State Dynamics Residuals

   Lanier, J; Kim, K; Karamzade, A; Liu, Y; Sinha, A; He, K; Corsi, D; Fox, R (July 2026, LMDC)

   Simulation-to-reality reinforcement learning (RL) faces the challenge of reconciling discrepancies between simulated and real-world dynamics, which can degrade agent performance. When real data is scarce, a promising approach involves learning corrections to simulator forward dynamics represented as a residual error function; however, this operation is impractical with high-dimensional states such as images. To overcome this, we propose ReDRAW, a latent-state autoregressive world model pretrained in simulation and calibrated to a target environment through residual corrections of latent-state dynamics rather than of explicit observed states. Using this adapted world model, ReDRAWenables RL agents to be optimized with imagined rollouts under corrected dynamics and then deployed in the real world. In multiple vision-based DeepMind Control Suite domains and a physical robot visual lane-following task, ReDRAW effectively models changes to dynamics and avoids overfitting in low data regimes where traditional transfer methods fail. Keywords: Reinforcement Learning, World Models, Sim-to-Real, Transfer Learning 

   more » « less
3. Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings

   Chen, Lili; Lee, Kimin; Srinivas, Aravind; Abbeel, Pieter (December 2021, Advances in neural information processing systems)

   null (Ed.)

   Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games. Finally, we show that SEER is useful for computation-efficient transfer learning in RL because lower layers of CNNs extract generalizable features, which can be used for different tasks and domains. 

   more » « less
4. Deep Computerized Adaptive Testing

   https://doi.org/10.1017/psy.2026.10106  

   Li, Jiguang; Gibbons, Robert; Ročková, Veronika (June 2026, Psychometrika)

   Abstract Computerized adaptive tests (CATs) play a crucial role in educational assessment and diagnostic screening in behavioral health. Unlike traditional linear tests that administer a fixed set of pre-assembled items, CATs adaptively tailor the test to an examinee’s latent trait level based on their previous responses. We introduce a novel CAT system that builds on recent advances in Bayesian multivariate IRT. Our approach leverages direct sampling from the latent factor posterior distributions, significantly accelerating existing information-theoretic item-selection methods by eliminating the need for computationally intensive Markov chain Monte Carlo simulations. To address the potential suboptimality of one-step-ahead item-selection rules, we also develop a double deep Q-learning algorithm that efficiently learns an optimal item-selection policy offline using a calibrated item bank. Through simulation and real-data studies, we demonstrate that our approach not only accelerates existing item-selection methods but also highlights the potential of reinforcement learning (RL) in CATs. Notably, our Q-learning-based strategy consistently achieves the fastest posterior variance reduction, leading to earlier test termination. These results demonstrate the promise of combining exact posterior sampling with RL to deliver scalable, high-precision CATs. 

   more » « less
5. Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo

   Haque, Ishfaq; Lan, Qingfeng; Xu, Pan; Mahmood, A Rupam; Precup, Doina; Anandkumar, Anima; Azizzadenesheli, Kamyar (January 2024, The Twelfth International Conference on Learning Representations)

   We present a scalable and effective exploration strategy based on Thompson sampling for reinforcement learning (RL). One of the key shortcomings of existing Thompson sampling algorithms is the need to perform a Gaussian approximation of the posterior distribution, which is not a good surrogate in most practical settings. We instead directly sample the Q function from its posterior distribution, by using Langevin Monte Carlo, an efficient type of Markov Chain Monte Carlo (MCMC) method. Our method only needs to perform noisy gradient descent updates to learn the exact posterior distribution of the Q function, which makes our approach easy to deploy in deep RL. We provide a rigorous theoretical analysis for the proposed method and demonstrate that, in the linear Markov decision process (linear MDP) setting, it has a regret bound of $$\tilde{O}(d^{3/2}H^{3/2}\sqrt{T})$$, where $$d$$ is the dimension of the feature mapping, $$H$$ is the planning horizon, and $$T$$ is the total number of steps. We apply this approach to deep RL, by using Adam optimizer to perform gradient updates. Our approach achieves better or similar results compared with state-of-the-art deep RL algorithms on several challenging exploration tasks from the Atari57 suite.\footnote{Our code is available at \url{https://github.com/hmishfaq/LMC-LSVI}} 

   more » « less