An official website of the United States government
Animal brains evolved to optimize behavior in dynamic environments, flexibly selecting actions that maximize future rewards in different contexts. A large body of experimental work indicates that such optimization changes the wiring of neural circuits, appropriately mapping environmental input onto behavioral outputs. A major unsolved scientific question is how optimal wiring adjustments, which must target the connections responsible for rewards, can be accomplished when the relation between sensory inputs, action taken, environmental context with rewards is ambiguous. The credit assignment problem can be categorized into context-independent structural credit assignment and context-dependent continual learning. In this perspective, we survey prior approaches to these two problems and advance the notion that the brain’s specialized neural architectures provide efficient solutions. Within this framework, the thalamus with its cortical and basal ganglia interactions serves as a systems-level solution to credit assignment. Specifically, we propose that thalamocortical interaction is the locus of meta-learning where the thalamus provides cortical control functions that parametrize the cortical activity association space. By selecting among these control functions, the basal ganglia hierarchically guide thalamocortical plasticity across two timescales to enable meta-learning. The faster timescale establishes contextual associations to enable behavioral flexibility while the slower one enables generalization to new contexts.
more »
« less
The functional role of the striatum as an action evaluation circuit: a network-level theory
Action selection is important for species survival. The basal ganglia, a subcortical structure, has long been thought to play a crucial role in action selection and movement initiation. Classical theories suggest that an important role of the striatum, the input region of the basal ganglia, is to select actions to be performed based on cortical projections carrying action information. However, thanks to recent progress in neural recording techniques, new experimental evidence suggests that the striatum does not perform action selection. Rather, the striatum plays an advisory role. Thus the classical theories of the basal ganglia need to be revisited and revised. As a rst step, in this work we hypothesize a new computational role for the striatum. We present a network-level theory in which the striatum transforms cortical action bids into action evaluations. Based on the region’s neural circuitry, we theorize that the role of the striatum is to transform bids to action values that are normalized, contrast-enhanced, orthogonalized, and encoded as continuous values through the use of two separate neuron populations with bipolar tuning and both feedforward and collateral inhibitory mechanisms. We simulate our network and investigate the role of the network components in its dynamics. Finally, we compare the behavior of our network to previous literature on decision-making behavior in rodents and primates.
more »
« less
- PAR ID:
- 10584420
- Publisher / Repository:
- 10th workshop on Biological Distributed Algorithms (BDA)
- Date Published:
- Format(s):
- Medium: X
- Location:
- Nantes, France
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
To effectively interact with their environment, humans must often select actions from multiple incompatible options. Existing theories propose that during motoric response-conflict, inappropriate motor activity is actively (and perhaps non-selectively) suppressed by an inhibitory fronto-basal ganglia mechanism. We here tested this theory across three experiments. First, using scalp-EEG, we found that both outright action-stopping and response-conflict during action-selection invoke low-frequency activity of a common fronto-central source, whose activity relates to trial-by-trial behavioral indices of inhibition in both tasks. Second, using simultaneous intracranial recordings from the basal ganglia and motor cortex, we found that response-conflict increases the influence of the subthalamic nucleus on M1-representations of incorrect response-tendencies. Finally, using transcranial magnetic stimulation, we found that during the same time period when conflict-related STN-to-M1 communication is increased, cortico-spinal excitability is broadly suppressed. Together, these findings demonstrate that fronto-basal ganglia networks buttress action-selection under response-conflict by rapidly and non-selectively net-inhibiting inappropriate motor tendencies.more » « less
-
Abstract The striatum plays an important role in learning, selecting, and executing actions. As a major input hub of the basal ganglia, it receives and processes a diverse array of signals related to sensory, motor, and cognitive information. Aberrant neural activity in this area is implicated in a wide variety of neurological and psychiatric disorders. It is therefore important to understand the hallmarks of disrupted striatal signal processing. This review surveys literature examining howin vivostriatal microcircuit dynamics are impacted in animal models of one of the most widely studied movement disorders, Parkinson's disease. The review identifies four major features of aberrant striatal dynamics: altered relative levels of direct and indirect pathway activity, impaired information processing by projection neurons, altered information processing by interneurons, and increased synchrony.more » « less
-
Decision making in natural settings requires efficient exploration to handle uncertainty. Since associations between actions and outcomes are uncertain, animals need to balance the explorations and exploitation to select the actions that lead to maximal rewards. The computa- tional principles by which animal brains explore during decision-making are poorly understood. Our challenge here was to build a biologically plausible neural network that efficiently explores an environment and understands its effectiveness mathematically. One of the most evolutionarily conserved and important systems in decision making is basal ganglia (BG)1. In particular, the dopamine activities (DA) in BG is thought to represent reward prediction error (RPE) to facilitate reinforcement learning2. Therefore, our starting point is a cortico-BG loop motif3. This network adjusts exploration based on neuronal noises and updates its value estimate through RPE. To account for the fact that animals adjust exploration based on experience, we modified the network in two ways. First, it is recently discovered that DA does not simply represent the scalar RPE value; rather it represents RPE in a distribution4. We incorporated the distributional RPE framework and further the hypothesis, allowing an RPE distribution to update the posterior of action values encoded by cortico-BG connections. Second, it is known that the firing in the layer 2/3 of cortex fires is variable and sparse5. Our network thus included a random sparsification of cortical activity as a mechanism of sampling from this posterior for experience-based exploration. Combining these two features, our network is able to take the uncertainty of our value estimates into account to accomplish efficient exploration in a variety of environments.more » « less
-
Decision making in natural settings requires efficient exploration to handle uncertainty. Since associations between actions and outcomes are uncertain, animals need to balance the explorations and exploitation to select the actions that lead to maximal rewards. The computa- tional principles by which animal brains explore during decision-making are poorly understood. Our challenge here was to build a biologically plausible neural network that efficiently explores an environment and understands its effectiveness mathematically. One of the most evolutionarily conserved and important systems in decision making is basal ganglia (BG)1. In particular, the dopamine activities (DA) in BG is thought to represent reward prediction error (RPE) to facilitate reinforcement learning2. Therefore, our starting point is a cortico-BG loop motif3. This network adjusts exploration based on neuronal noises and updates its value estimate through RPE. To account for the fact that animals adjust exploration based on experience, we modified the network in two ways. First, it is recently discovered that DA does not simply represent the scalar RPE value; rather it represents RPE in a distribution4. We incorporated the distributional RPE framework and further the hypothesis, allowing an RPE distribution to update the posterior of action values encoded by cortico-BG connections. Second, it is known that the firing in the layer 2/3 of cortex fires is variable and sparse5. Our network thus included a random sparsification of cortical activity as a mechanism of sampling from this posterior for experience-based exploration. Combining these two features, our network is able to take the uncertainty of our value estimates into account to accomplish efficient exploration in a variety of environments.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
