An official website of the United States government
In standard reinforcement learning settings, agents typically assume immediate feedback about the effects of their actions after taking them. However, in practice, this assumption may not hold true due to physical constraints and can significantly impact the performance of learning algorithms. In this paper, we address observation delays in partially observable environments. We propose leveraging world models, which have shown success in integrating past observations and learning dynamics, to handle observation delays. By reducing delayed POMDPs to delayed MDPs with world models, our methods can effectively handle partial observability, where existing approaches achieve sub-optimal performance or degrade quickly as observability decreases. Experiments suggest that one of our methods can outperform a naive model-based approach by up to 250%. Moreover, we evaluate our methods on visual delayed environments, for the first time showcasing delay-aware reinforcement learning continuous control with visual observations.
more »
« less
Cuttlefish exert self-control in a delay of gratification task
The ability to exert self-control varies within and across taxa. Some species can exert self-control for several seconds whereas others, such as large-brained vertebrates, can tolerate delays of up to several minutes. Advanced self-control has been linked to better performance in cognitive tasks and has been hypothesized to evolve in response to specific socio-ecological pressures. These pressures are difficult to uncouple because previously studied species face similar socio-ecological challenges. Here, we investigate self-control and learning performance in cuttlefish, an invertebrate that is thought to have evolved under partially different pressures to previously studied vertebrates. To test self-control, cuttlefish were presented with a delay maintenance task, which measures an individual's ability to forgo immediate gratification and sustain a delay for a better but delayed reward. Cuttlefish maintained delay durations for up to 50–130 s. To test learning performance, we used a reversal-learning task, whereby cuttlefish were required to learn to associate the reward with one of two stimuli and then subsequently learn to associate the reward with the alternative stimulus. Cuttlefish that delayed gratification for longer had better learning performance. Our results demonstrate that cuttlefish can tolerate delays to obtain food of higher quality comparable to that of some large-brained vertebrates.
more »
« less
- PAR ID:
- 10312588
- Date Published:
- Journal Name:
- Proceedings of the Royal Society B: Biological Sciences
- Volume:
- 288
- Issue:
- 1946
- ISSN:
- 0962-8452
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Several sources of delay in an epidemic network might negatively affect the stability and robustness of the entire network. In this paper, a multi-delayed Susceptible-Infectious-Susceptible (SIS) model is applied on a metapopulation network, where the epidemic delays are categorized into local and global delays. While local delays result from intra-population lags such as symptom development duration or recovery period, global delays stem from inter-population lags, e.g., transition duration between subpopulations. The theoretical results for a network of subpopulations with identical linear SIS dynamics and different types of time-delay show that depending on the type of time-delay in the network, different eigenvalues of the underlying graph should be evaluated to obtain the feasible regions of stability. The delay-dependent stability of such epidemic networks has been analytically derived, which eliminates potentially expensive computations required by current algorithms. The effect of time-delay on the H2 norm-based performance of a class of epidemic networks with additive noise inputs and multiple delays is studied and the closed form of their performance measure is derived using the solution of delayed Lyapunov equations. As a case study, the theoretical findings are implemented on a network of United States’ busiest airports.more » « less
-
Time-continuous dimensional descriptions of emotions (e.g., arousal, valence) allow researchers to characterize short-time changes and to capture long-term trends in emotion expression. However, continuous emotion labels are generally not synchronized with the input speech signal due to delays caused by reaction-time, which is inherent in human evaluations. To deal with this challenge, we introduce a new convolutional neural network (multi-delay sinc network) that is able to simultaneously align and predict labels in an end-to-end manner. The proposed network is a stack of convolutional layers followed by an aligner network that aligns the speech signal and emotion labels. This network is implemented using a new convolutional layer that we introduce, the delayed sinc layer. It is a time-shifted low-pass (sinc) filter that uses a gradient-based algorithm to learn a single delay. Multiple delayed sinc layers can be used to compensate for a non-stationary delay that is a function of the acoustic space. We test the efficacy of this system on two common emotion datasets, RECOLA and SEWA, and show that this approach obtains state-of-the-art speech-only results by learning time-varying delays while predicting dimensional descriptors of emotions.more » « less
-
Introduction: Impulsivity is a symptom of Attention-Deficit/Hyperactivity Disorder (ADHD) and variants in the Lphn3 (Adgrl3) gene [OMIM 616417] have been linked to ADHD. This project utilized a delay-discounting (DD) task to examine the impact of Lphn3 deletion in rats on impulsive choice. “Positive control” measures were also collected in Spontaneously Hypertensive Rats (SHRs), another animal model of ADHD. Methods: For Experiment I, rats were given the option to press one lever for a delayed reward of 3 food pellets or the other lever for an immediate reward of 1 pellet. Impulsive choice was measured as the tendency to discount the larger, delayed reward. We hypothesized that impulsive choice would be greater in the SHR and Lphn3 knockout (KO) rats relative to their control strains - Wistar-Kyoto (WKY) and Lphn3 wildtype (WT) rats, respectively. Results: The results did not completely support the hypothesis, as only the SHRs (but not the Lphn3 KO rats) demonstrated a decrease in the percent choice for the larger reward. Because subsequent trials did not begin until the end of the delay period regardless of which lever was selected, rats were required to wait for the next trial to start even if they picked the immediate lever. Experiment II examined whether the rate of reinforcement influenced impulsive choice by using a DD task that incorporated a 1 sec inter-trial interval (ITI) immediately after delivery of either the immediate (1 pellet) or delayed (3 pellet) reinforcer. The results of Experiment II found no difference in the percent choice for the larger reward between Lphn3 KO and WT rats, demonstrating reinforcement rate did not influence impulsive choice in Lphn3 KO rats. Discussion: Overall, there were impulsivity differences among the ADHD models, as SHRs exhibited deficits in impulsive choice, while the Lphn3 KO rats did not.more » « less
-
We formulate a predictor-based controller for a high-DOF manipulator to compensate a time-invariant input delay during a pick-and-place task. Robot manipulators are widely used in tele-manipulation systems on the account of their reliable, fast, and precise motions while they are subject to large delays. Using common control algorithms on such delay systems can cause not only poor control performance, but also catastrophic instability in engineering applications. Therefore, delays need to be compensated in designing robust control laws. As a case study, we focus on a 7-DOF Baxter manipulator subject to three different input delays. First, delay-free dynamic equations of the Baxter manipulator are derived using the Lagrangian method. Then, we formulate a predictor-based controller, in the presence of input delay, in order to track desired trajectories. Finally, the effects of input delays in the absence of a robust predictor are investigated, and then the performance of the predictor-based controller is experimentally evaluated to reveal robustness of the algorithm formulated. Simulation and experimental results demonstrate that the predictor-based controller effectively compensates input delays and achieves closed-loop stability.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1098/rspb.2020.3161
