Search for: All records

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings. 

Inside mammalian cells, single genes are known to be transcribed in stochastic bursts leading to the synthesis of nuclear RNAs that are subsequently exported to the cytoplasm to create mRNAs. We systematically characterize the role of export processes in shaping the extent of random fluctuations (i.e. noise) in the mRNA level of a given gene. Using the method of Partitioning of Poisson arrivals, we derive an exact analytical expression for the noise in mRNA level assuming that the nuclear retention time of each RNA is an independent and identically distributed random variable following an arbitrary distribution. These results confirm recent experimental/theoretical findings that decreasing the nuclear export rate buffers the noise in mRNA level, and counterintuitively, decreasing the noise in the nuclear retention time enhances the noise in the mRNA level. Next, we further generalize the model to consider a dynamic extrinsic disturbance that affects the nuclear-to-cytoplasm export. Our results show that noise in the mRNA level varies non-monotonically with the disturbance timescale. More specifically, high- and low-frequency external disturbances have little impact on the mRNA noise level, while noise is amplified at intermediate frequencies. In summary, our results systematically uncover how the coupling of bursty transcription with nuclear export can both attenuate or amplify noise in mRNA levels depending on the nuclear retention time distribution and the presence of extrinsic fluctuations.