More Like this

1. Q-Functionals For Value-Based Continuous Control

   Lobel, S; Rammohan, S; He, B; Yu, S; Konidaris, GD (February 2023, Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence)

   We present Q-functionals, an alternative architecture for continuous control deep reinforcement learning. Instead of returning a single value for a state-action pair, our network transforms a state into a function that can be rapidly evaluated in parallel for many actions, allowing us to efficiently choose high-value actions through sampling. This contrasts with the typical architecture of off-policy continuous control, where a policy network is trained for the sole purpose of selecting actions from the Q-function. We represent our action-dependent Q-function as a weighted sum of basis functions (Fourier, Polynomial, etc) over the action space, where the weights are state-dependent and output by the Q-functional network. Fast sampling makes practical a variety of techniques that require Monte-Carlo integration over Q-functions, and enables action-selection strategies besides simple value-maximization. We characterize our framework, describe various implementations of Q-functionals, and demonstrate strong performance on a suite of continuous control tasks. 

   more » « less
2. ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation

   Zhou Xian, Nikolaos Gkanatsios (November 2023, Conference on Robot Learning/Proceedings of Machine Learning Research)

   We present ChainedDiffuser, a policy architecture that unifies action keypose prediction and trajectory diffusion generation for learning robot manipulation from demonstrations. Our main innovation is to use a global transformerbased action predictor to predict actions at keyframes, a task that requires multimodal semantic scene understanding, and to use a local trajectory diffuser to predict trajectory segments that connect predicted macro-actions. ChainedDiffuser sets a new record on established manipulation benchmarks, and outperforms both state-of-the-art keypose (macro-action) prediction models that use motion planners for trajectory prediction, and trajectory diffusion policies that do not predict keyframe macro-actions. We conduct experiments in both simulated and realworld environments and demonstrate ChainedDiffuser’s ability to solve a wide range of manipulation tasks involving interactions with diverse objects. 

   more » « less
3. Pareto-Based Narrative Planning: Making NPCs More Rational

   https://doi.org/10.1609/aiide.v21i1.36837  

   Siler, Molly; Fisher, Mira; Ware, Stephen G (November 2025, Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment)

   A narrative planner decides the actions for all characters in a story while justifying each action according to the acting characters' own individual goals. However, an action that contributes to a goal may still seem irrational when considered alongside other actions the character may take; for instance, it may sacrifice a more important goal, or expose the character to unnecessary risks. We redefine the narrative planning character model to use a multiobjective framework where character actions are chosen from a Pareto front of best and safest options. We discuss how this framework can be applied to generate a policy for how the non-player characters should behave in any given state of an interactive narrative, and how we applied such a policy in the Traffic Stop de-escalation training simulation. 

   more » « less
4. Decision Making in Non-Stationary Environments with Policy-Augmented Search

   Pettet, Ava; Zhang, Yunuo; Luo, Baiting; Wray, Kyle; Baier, Hendrik; Laszka, Aron; Dubey, Abhishek; Mukhopadhyay, Ayan (May 2024, International Conference on Autonomous Agents and Multiagent Systems)

   Sequential decision-making under uncertainty is present in many important problems. Two popular approaches for tackling such problems are reinforcement learning and online search (e.g., Monte Carlo tree search). While the former learns a policy by interacting with the environment (typically done before execution), the latter uses a generative model of the environment to sample promising action trajectories at decision time. Decision-making is particularly challenging in non-stationary environments, where the environment in which an agent operates can change over time. Both approaches have shortcomings in such settings -- on the one hand, policies learned before execution become stale when the environment changes and relearning takes both time and computational effort. Online search, on the other hand, can return sub-optimal actions when there are limitations on allowed runtime. In this paper, we introduce \textit{Policy-Augmented Monte Carlo tree search} (PA-MCTS), which combines action-value estimates from an out-of-date policy with an online search using an up-to-date model of the environment. We prove theoretical results showing conditions under which PA-MCTS selects the one-step optimal action and also bound the error accrued while following PA-MCTS as a policy. We compare and contrast our approach with AlphaZero, another hybrid planning approach, and Deep Q Learning on several OpenAI Gym environments. Through extensive experiments, we show that under non-stationary settings with limited time constraints, PA-MCTS outperforms these baselines. 

   more » « less
5. Hyperfiniteness for group actions on trees

   https://doi.org/10.1090/proc/16851  

   Kunnawalkam_Elayavalli, Srivatsav; Oyakawa, Koichi; Shinko, Forte; Spaas, Pieter (July 2024, Proceedings of the American Mathematical Society)

   We identify natural conditions for a countable group acting on a countable tree which imply that the orbit equivalence relation of the induced action on the Gromov boundary is Borel hyperfinite. Examples of this condition include acylindrical actions. We also identify a natural weakening of the aforementioned conditions that implies measure hyperfiniteness of the boundary action. We then document examples of group actions on trees whose boundary action is not hyperfinite. 

   more » « less