skip to main content


Title: Reinforcement Learning Assist-as-needed Control for Robot Assisted Gait Training
The primary goal of an assist-as-needed (AAN) controller is to maximize subjects' active participation during motor training tasks while allowing moderate tracking errors to encourage human learning of a target movement. Impedance control is typically employed by AAN controllers to create a compliant force-field around the desired motion trajectory. To accommodate different individuals with varying motor abilities, most of the existing AAN controllers require extensive manual tuning of the control parameters, resulting in a tedious and time-consuming process. In this paper, we propose a reinforcement learning AAN controller that can autonomously reshape the force-field in real-time based on subjects' training performances. The use of action-dependent heuristic dynamic programming enables a model-free implementation of the proposed controller. To experimentally validate the controller, a group of healthy individuals participated in a gait training session wherein they were asked to learn a modified gait pattern with the help of a powered ankle-foot orthosis. Results indicated the potential of the proposed control strategy for robot-assisted gait training.  more » « less
Award ID(s):
1944203
NSF-PAR ID:
10216891
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob)
Page Range / eLocation ID:
785 to 790
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Background

    This research focused on the development of a motor imagery (MI) based brain–machine interface (BMI) using deep learning algorithms to control a lower-limb robotic exoskeleton. The study aimed to overcome the limitations of traditional BMI approaches by leveraging the advantages of deep learning, such as automated feature extraction and transfer learning. The experimental protocol to evaluate the BMI was designed as asynchronous, allowing subjects to perform mental tasks at their own will.

    Methods

    A total of five healthy able-bodied subjects were enrolled in this study to participate in a series of experimental sessions. The brain signals from two of these sessions were used to develop a generic deep learning model through transfer learning. Subsequently, this model was fine-tuned during the remaining sessions and subjected to evaluation. Three distinct deep learning approaches were compared: one that did not undergo fine-tuning, another that fine-tuned all layers of the model, and a third one that fine-tuned only the last three layers. The evaluation phase involved the exclusive closed-loop control of the exoskeleton device by the participants’ neural activity using the second deep learning approach for the decoding.

    Results

    The three deep learning approaches were assessed in comparison to an approach based on spatial features that was trained for each subject and experimental session, demonstrating their superior performance. Interestingly, the deep learning approach without fine-tuning achieved comparable performance to the features-based approach, indicating that a generic model trained on data from different individuals and previous sessions can yield similar efficacy. Among the three deep learning approaches compared, fine-tuning all layer weights demonstrated the highest performance.

    Conclusion

    This research represents an initial stride toward future calibration-free methods. Despite the efforts to diminish calibration time by leveraging data from other subjects, complete elimination proved unattainable. The study’s discoveries hold notable significance for advancing calibration-free approaches, offering the promise of minimizing the need for training trials. Furthermore, the experimental evaluation protocol employed in this study aimed to replicate real-life scenarios, granting participants a higher degree of autonomy in decision-making regarding actions such as walking or stopping gait.

     
    more » « less
  2. Abstract Background Few studies have systematically investigated robust controllers for lower limb rehabilitation exoskeletons (LLREs) that can safely and effectively assist users with a variety of neuromuscular disorders to walk with full autonomy. One of the key challenges for developing such a robust controller is to handle different degrees of uncertain human-exoskeleton interaction forces from the patients. Consequently, conventional walking controllers either are patient-condition specific or involve tuning of many control parameters, which could behave unreliably and even fail to maintain balance. Methods We present a novel, deep neural network, reinforcement learning-based robust controller for a LLRE based on a decoupled offline human-exoskeleton simulation training with three independent networks, which aims to provide reliable walking assistance against various and uncertain human-exoskeleton interaction forces. The exoskeleton controller is driven by a neural network control policy that acts on a stream of the LLRE’s proprioceptive signals, including joint kinematic states, and subsequently predicts real-time position control targets for the actuated joints. To handle uncertain human interaction forces, the control policy is trained intentionally with an integrated human musculoskeletal model and realistic human-exoskeleton interaction forces. Two other neural networks are connected with the control policy network to predict the interaction forces and muscle coordination. To further increase the robustness of the control policy to different human conditions, we employ domain randomization during training that includes not only randomization of exoskeleton dynamics properties but, more importantly, randomization of human muscle strength to simulate the variability of the patient’s disability. Through this decoupled deep reinforcement learning framework, the trained controller of LLREs is able to provide reliable walking assistance to patients with different degrees of neuromuscular disorders without any control parameter tuning. Results and conclusion A universal, RL-based walking controller is trained and virtually tested on a LLRE system to verify its effectiveness and robustness in assisting users with different disabilities such as passive muscles (quadriplegic), muscle weakness, or hemiplegic conditions without any control parameter tuning. Analysis of the RMSE for joint tracking, CoP-based stability, and gait symmetry shows the effectiveness of the controller. An ablation study also demonstrates the strong robustness of the control policy under large exoskeleton dynamic property ranges and various human-exoskeleton interaction forces. The decoupled network structure allows us to isolate the LLRE control policy network for testing and sim-to-real transfer since it uses only proprioception information of the LLRE (joint sensory state) as the input. Furthermore, the controller is shown to be able to handle different patient conditions without the need for patient-specific control parameter tuning. 
    more » « less
  3. null (Ed.)
    In this work, we introduce a novel approach to assistive exoskeleton (or powered orthosis) control which avoids needing task and gait phase information. Our approach is based on directly designing the Hamiltonian dynamics of the target closed-loop behavior, shaping the energy of the human and the robot. Relative to previous energy shaping controllers for assistive exoskeletons, we introduce ground reaction force and torque information into the target behavior definition, reformulate the kinematics so as to avoid explicit matching conditions due to under-actuation, and avoid the need to switch between swing and stance energy shapes. Our controller introduces new states into the target Hamiltonian energy that represent a virtual second leg that is connected to the physical leg using virtual springs. The impulse the human imparts to the physical leg is amplified and applied to the virtual leg, but the ground reaction force acts only on the physical leg. A state transformation allows the proposed control to be available using only encoders, an IMU, and ground reaction force sensors. We prove that this controller is stable and passive when acted on by the ground reaction force and demonstrate the controller's strength amplifying behavior in a simulation. A linear analysis based on small signal assumptions allows us to explain the relationship between our tuning parameters and the frequency domain amplification bandwidth. 
    more » « less
  4. Powered prostheses aim to mimic the missing biological limb with controllers that are finely tuned to replicate the nominal gait pattern of non-amputee individuals. Unfortunately, this control approach poses a problem with real-world ambulation, which includes tasks such as crossing over obstacles, where the prosthesis trajectory must be modified to provide adequate foot clearance and ensure timely foot placement. Here, we show an indirect volitional control approach that enables prosthesis users to walk at different speeds while smoothly and continuously crossing over obstacles of different sizes without explicit classification of the environment. At the high level, the proposed controller relies on a heuristic algorithm to continuously change the maximum knee flexion angle and the swing duration in harmony with the user’s residual limb. At the low level, minimum-jerk planning is used to continuously adapt the swing trajectory while maximizing smoothness. Experiments with three individuals with above-knee amputation show that the proposed control approach allows for volitional control of foot clearance, which is necessary to negotiate environmental barriers. Our study suggests that a powered prosthesis controller with intrinsic, volitional adaptability may provide prosthesis users with functionality that is not currently available, facilitating real-world ambulation.

     
    more » « less
  5. null (Ed.)
    Powered ankle exoskeletons that apply assistive torques with optimized timing and magnitude can reduce metabolic cost by ∼10% compared to normal walking. However, finding individualized optimal control parameters is time consuming and must be done independently for different walking modes (e.g., speeds, slopes). Thus, there is a need for exoskeleton controllers that are capable of continuously adapting torque assistance in concert with changing locomotor demands. One option is to use a biologically inspired, model-based control scheme that can capture the adaptive behavior of the human plantarflexors during natural gait. Here, based on previously demonstrated success in a powered ankle-foot prosthesis, we developed an ankle exoskeleton controller that uses a neuromuscular model (NMM) comprised of a Hill type musculotendon driven by a simple positive force feedback reflex loop. To examine the effects of NMM reflex parameter settings on (i) ankle exoskeleton mechanical performance and (ii) users’ physiological response, we recruited nine healthy, young adults to walk on a treadmill at a fixed speed of 1.25 m/s while donning bilateral tethered robotic ankle exoskeletons. To quantify exoskeleton mechanics, we measured exoskeleton torque and power output across a range of NMM controller Gain (0.8–2.0) and Delay (10–40 ms) settings, as well as a High Gain/High Delay (2.0/40 ms) combination. To quantify users’ physiological response, we compared joint kinematics and kinetics, ankle muscle electromyography and metabolic rate between powered and unpowered/zero-torque conditions. Increasing NMM controller reflex Gain caused increases in average ankle exoskeleton torque and net power output, while increasing NMM controller reflex Delay caused a decrease in net ankle exoskeleton power output. Despite systematic reduction in users’ average biological ankle moment with exoskeleton mechanical assistance, we found no NMM controller Gain or Delay settings that yielded changes in metabolic rate. Post hoc analyses revealed weak association at best between exoskeleton and biological mechanics and changes in users’ metabolic rate. Instead, changes in users’ summed ankle joint muscle activity with powered assistance correlated with changes in their metabolic energy use, highlighting the potential to utilize muscle electromyography as a target for on-line optimization in next generation adaptive exoskeleton controllers. 
    more » « less