An official website of the United States government
Sensorimotor tasks that humans perform are often affected by different sources of uncertainty. Nevertheless, the central nervous system (CNS) can gracefully coordinate our movements. Most learning frameworks rely on the internal model principle, which requires a precise internal representation in the CNS to predict the outcomes of our motor commands. However, learning a perfect internal model in a complex environment over a short period of time is a nontrivial problem. Indeed, achieving proficient motor skills may require years of training for some difficult tasks. Internal models alone may not be adequate to explain the motor adaptation behavior during the early phase of learning. Recent studies investigating the active regulation of motor variability, the presence of suboptimal inference, and model-free learning have challenged some of the traditional viewpoints on the sensorimotor learning mechanism. As a result, it may be necessary to develop a computational framework that can account for these new phenomena. Here, we develop a novel theory of motor learning, based on model-free adaptive optimal control, which can bypass some of the difficulties in existing theories. This new theory is based on our recently developed adaptive dynamic programming (ADP) and robust ADP (RADP) methods and is especially useful for accounting for motor learning behavior when an internal model is inaccurate or unavailable. Our preliminary computational results are in line with experimental observations reported in the literature and can account for some phenomena that are inexplicable using existing models.
more »
« less
Human motor learning is robust to control-dependent noise
Noises are ubiquitous in sensorimotor interactions and contaminate the information provided to the central nervous system (CNS) for motor learning. An interesting question is how the CNS manages motor learning with imprecise information. Integrating ideas from reinforcement learning and adaptive optimal control, this paper develops a novel computational mechanism to explain the robustness of human motor learning to the imprecise information, caused by control-dependent noise that exists inherently in the sensorimotor systems. Starting from an initial admissible control policy, in each learning trial the mechanism collects and uses the noisy sensory data (caused by the control-dependent noise) to form an imprecise evaluation of the performance of the current policy and then constructs an updated policy based on the imprecise evaluation. As the number of learning trials increases, the generated policies mathematically provably converge to a (potentially small) neighborhood of the optimal policy under mild conditions, despite the imprecise information in the learning process. The mechanism directly synthesizes the policies from the sensory data, without identifying an internal forward model. Our preliminary computational results on two classic arm reaching tasks are in line with experimental observations reported in the literature. The model-free control principle proposed in the paper sheds more lights into the inherent robustness of human sensorimotor systems to the imprecise information, especially control-dependent noise, in the CNS.
more »
« less
- Award ID(s):
- 1903781
- PAR ID:
- 10479219
- Publisher / Repository:
- Springer
- Date Published:
- Journal Name:
- Biological Cybernetics
- Volume:
- 116
- Issue:
- 3
- ISSN:
- 1432-0770
- Page Range / eLocation ID:
- 307 to 325
- Subject(s) / Keyword(s):
- Robustness · Reinforcement learning · Policy iteration · Sensorimotor control · Arm reaching
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Risk sensitivity is a fundamental aspect of biological motor control that accounts for both the expectation and variability of movement cost in the face of uncertainty. However, most computational models of biological motor control rely on model-based risk-sensitive optimal control, which requires an accurate internal representation in the central neural system to predict the outcomes of motor commands. In reality, the dynamics of human-environment interaction is too complex to be accurately modeled, and noise further complicates system identification. To address this issue, this paper proposes a novel risk-sensitive computational mechanism for biological motor control based on reinforcement learning (RL) and adaptive dynamic programming (ADP). The proposed ADP-based mechanism suggests that humans can directly learn an approximation of the risk-sensitive optimal feedback controller from noisy sensory data without the need for system identification. Numerical validation of the proposed mechanism is conducted on the arm-reaching task under divergent force field. The preliminary computational results align with the experimental observations from the past literature of computational neuroscience.more » « less
-
Motor systems in animals are highly dependent on sensory information for optimal control and precision, with mechanosensory feedback from the somatosensory system playing a critical role. These mechanosensory pathways are woven into the descending feedforward pathways and local central pattern generator circuits that control and generate movement, respectively. Somatosensory feedback in mammals and insects, the two animal classes this review touches upon, is complex due to the increased demands that limbed locomotion, weight-bearing, and corrective movements place on sensorimotor control. In this review, we outline the salient features of the proprioceptive and exteroceptive sensory feedback pathways animals rely on for controlling movement and highlight some of the key principles of sensory feedback that are shared across the animal kingdom.more » « less
-
Abstract Action and perception are closely linked in many behaviors necessitating a close coordination between sensory and motor neural processes so as to achieve a well-integrated smoothly evolving task performance. To investigate the detailed nature of these sensorimotor interactions, and their role in learning and executing the skilled motor task of speaking, we analyzed ECoG recordings of responses in the high-γ band (70–150 Hz) in human subjects while they listened to, spoke, or silently articulated speech. We found elaborate spectrotemporally modulated neural activity projecting in both “forward” (motor-to-sensory) and “inverse” directions between the higher-auditory and motor cortical regions engaged during speaking. Furthermore, mathematical simulations demonstrate a key role for the forward projection in “learning” to control the vocal tract, beyond its commonly postulated predictive role during execution. These results therefore offer a broader view of the functional role of the ubiquitous forward projection as an important ingredient in learning, rather than just control, of skilled sensorimotor tasks.more » « less
-
null (Ed.)People with moderate-to-severe cerebral palsy (CP) have the greatest need for postural control research yet are usually excluded from research due to deficits in sitting ability. We use a support system that allows us to quantify and model postural mechanisms in nonambulatory children with CP. A continuous external bench tilt stimulus was used to evoke trunk postural responses in seven sitting children with CP (ages 2.5 to 13 yr) in several test sessions. Eight healthy adults were also included. Postural sway was analyzed with root mean square (RMS) sway and RMS sway velocity, along with frequency response functions (FRF, gain and phase) and coherence functions across two different stimulus amplitudes. A feedback model (including sensorimotor noise, passive, reflexive, and sensory integration mechanisms) was developed to hypothesize how postural control mechanisms are organized and function. Experimental results showed large RMS sway, FRF gains, and variability compared with adults. Modeling suggested that many subjects with CP adopted “simple” control with major contributions from a passive and reflexive mechanism and only a small contribution from active sensory integration. In contrast, mature trunk postural control includes major contributions from sensory integration and sensory reweighting. Relative to their body size, subjects with CP showed significantly lower damping, three to five times larger corrective torque, and much higher sensorimotor noise compared with the healthy mature system. Results are the first characterization of trunk postural responses and the first attempt at system identification in moderate-to-severe CP, an important step toward developing and evaluating more targeted interventions. NEW & NOTEWORTHY Cerebral palsy (CP) is the most common cause of motor disability in children. People with moderate-to-severe CP are typically nonambulatory and have major impairments in trunk postural control. We present the first systems identification study to investigate postural responses to external stimulus in this population and hypothesize at how the atypical postural control system functions with use of a feedback model. People with moderate-to-severe CP may use a simple control system with significant sensorimotor noise.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1007/s00422-022-00922-z
