This content will become publicly available on August 18, 2024
Title: Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks
A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.
The Drosophila mushroom body exhibits dopamine dependent synaptic plasticity that underlies the acquisition of associative memories. Recordings of dopamine neurons in this system have identified signals related to external reinforcement such as reward and punishment. However, other factors including locomotion, novelty, reward expectation, and internal state have also recently been shown to modulate dopamine neurons. This heterogeneity is at odds with typical modeling approaches in which these neurons are assumed to encode a global, scalar error signal. How is dopamine dependent plasticity coordinated in the presence of such heterogeneity? We develop a modeling approach that infers a pattern of dopamine activity sufficient to solve defined behavioral tasks, given architectural constraints informed by knowledge of mushroom body circuitry. Model dopamine neurons exhibit diverse tuning to task parameters while nonetheless producing coherent learned behaviors. Notably, reward prediction error emerges as a mode of population activity distributed across these neurons. Our results provide a mechanistic framework that accounts for the heterogeneity of dopamine activity during learning and behavior.
Decision making in natural settings requires efficient exploration to handle uncertainty.
Since associations between actions and outcomes are uncertain, animals need to balance the
explorations and exploitation to select the actions that lead to maximal rewards. The computa-
tional principles by which animal brains explore during decision-making are poorly understood.
Our challenge here was to build a biologically plausible neural network that efficiently explores
an environment and understands its effectiveness mathematically.
One of the most evolutionarily conserved and important systems in decision making is basal
ganglia (BG)1. In particular, the dopamine activities (DA) in BG is thought to represent
reward prediction error (RPE) to facilitate reinforcement learning2. Therefore, our starting
point is a cortico-BG loop motif3. This network adjusts exploration based on neuronal noises
and updates its value estimate through RPE. To account for the fact that animals adjust
exploration based on experience, we modified the network in two ways. First, it is recently
discovered that DA does not simply represent the scalar RPE value; rather it represents RPE in
a distribution4. We incorporated the distributional RPE framework and further the hypothesis,
allowing an RPE distribution to update the posterior of action values encoded by cortico-BG
connections. Second, it is known that the firing in the layer 2/3 of cortex fires is variable and
sparse5. Our network thus included a random sparsification of cortical activity as a mechanism
of sampling from this posterior for experience-based exploration. Combining these two features,
our network is able to take the uncertainty of our value estimates into account to accomplish
efficient exploration in a variety of environments.
Decision making in natural settings requires efficient exploration to handle uncertainty.
Since associations between actions and outcomes are uncertain, animals need to balance the
explorations and exploitation to select the actions that lead to maximal rewards. The computa-
tional principles by which animal brains explore during decision-making are poorly understood.
Our challenge here was to build a biologically plausible neural network that efficiently explores
an environment and understands its effectiveness mathematically.
One of the most evolutionarily conserved and important systems in decision making is basal
ganglia (BG)1. In particular, the dopamine activities (DA) in BG is thought to represent
reward prediction error (RPE) to facilitate reinforcement learning2. Therefore, our starting
point is a cortico-BG loop motif3. This network adjusts exploration based on neuronal noises
and updates its value estimate through RPE. To account for the fact that animals adjust
exploration based on experience, we modified the network in two ways. First, it is recently
discovered that DA does not simply represent the scalar RPE value; rather it represents RPE in
a distribution4. We incorporated the distributional RPE framework and further the hypothesis,
allowing an RPE distribution to update the posterior of action values encoded by cortico-BG
connections. Second, it is known that the firing in the layer 2/3 of cortex fires is variable and
sparse5. Our network thus included a random sparsification of cortical activity as a mechanism
of sampling from this posterior for experience-based exploration. Combining these two features,
our network is able to take the uncertainty of our value estimates into account to accomplish
efficient exploration in a variety of environments.
Abstract The learning of stimulus-outcome associations allows for predictions about the environment. Ventral striatum and dopaminergic midbrain neurons form a larger network for generating reward prediction signals from sensory cues. Yet, the network plasticity mechanisms to generate predictive signals in these distributed circuits have not been entirely clarified. Also, direct evidence of the underlying interregional assembly formation and information transfer is still missing. Here we show that phasic dopamine is sufficient to reinforce the distinctness of stimulus representations in the ventral striatum even in the absence of reward. Upon such reinforcement, striatal stimulus encoding gives rise to interregional assemblies that drive dopaminergic neurons during stimulus-outcome learning. These assemblies dynamically encode the predicted reward value of conditioned stimuli. Together, our data reveal that ventral striatal and midbrain reward networks form a reinforcing loop to generate reward prediction coding.
Wang, Mien Brabeeba; Lynch, Nancy; Halassa, Michael(
, Thalamocortical Interactions Gordon Research Conference 2024)
Animals flexibly select actions that maximize future rewards despite facing uncertainty in sen-
sory inputs, action-outcome associations or contexts. The computational and circuit mechanisms
underlying this ability are poorly understood.
A clue to such computations can be found in the neural systems involved in representing sensory
features, sensorimotor-outcome associations and contexts. Specifically, the basal ganglia (BG) have
been implicated in forming sensorimotor-outcome association [1] while the thalamocortical loop
between the prefrontal cortex (PFC) and mediodorsal thalamus (MD) has been shown to engage in
contextual representations [2, 3]. Interestingly, both human and non-human animal experiments
indicate that the MD represents different forms of uncertainty [3, 4]. However, finding evidence
for uncertainty representation gives little insight into how it is utilized to drive behavior.
Normative theories have excelled at providing such computational insights. For example, de-
ploying traditional machine learning algorithms to fit human decision-making behavior has clarified
how associative uncertainty alters exploratory behavior [5, 6]. However, despite their computa-
tional insight and ability to fit behaviors, normative models cannot be directly related to neural
mechanisms. Therefore, a critical gap exists between what we know about the neural representa-
tion of uncertainty on one end and the computational functions uncertainty serves in cognition.
This gap can be filled with mechanistic neural models that can approximate normative models as
well as generate experimentally observed neural representations.
In this work, we build a mechanistic cortico-thalamo-BG loop network model that directly
fills this gap. The model includes computationally-relevant mechanistic details of both BG and
thalamocortical circuits such as distributional activities of dopamine [7] and thalamocortical pro-
jection modulating cortical effective connectivity [3] and plasticity [8] via interneurons. We show
that our network can more efficiently and flexibly explore various environments compared to com-
monly used machine learning algorithms and we show that the mechanistic features we include
are crucial for handling different types of uncertainty in decision-making. Furthermore, through
derivation and mathematical proofs, we approximate our models to two novel normative theories.
We show mathematically the first has near-optimal performance on bandit tasks. The second is
a generalization on the well-known CUMSUM algorithm, which is known to be optimal on single
change point detection tasks [9]. Our normative model expands on this by detecting multiple
sequential contextual changes. To our knowledge, our work is the first to link computational in-
sights, normative models and neural realization together in decision-making under various forms
of uncertainty.
Blackwell, Kim T., and Doya, Kenji. Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks. Retrieved from https://par.nsf.gov/biblio/10472794. PLOS Computational Biology 19.8 Web. doi:10.1371/journal.pcbi.1011385.
Blackwell, Kim T., & Doya, Kenji. Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks. PLOS Computational Biology, 19 (8). Retrieved from https://par.nsf.gov/biblio/10472794. https://doi.org/10.1371/journal.pcbi.1011385
Blackwell, Kim T., and Doya, Kenji.
"Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks". PLOS Computational Biology 19 (8). Country unknown/Code not available: PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1011385.https://par.nsf.gov/biblio/10472794.
@article{osti_10472794,
place = {Country unknown/Code not available},
title = {Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks},
url = {https://par.nsf.gov/biblio/10472794},
DOI = {10.1371/journal.pcbi.1011385},
abstractNote = {A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.},
journal = {PLOS Computational Biology},
volume = {19},
number = {8},
publisher = {PLOS Computational Biology},
author = {Blackwell, Kim T. and Doya, Kenji},
editor = {Cai, Ming Bo}
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
