More Like this

1. Data-Driven Latent Space Representation for Robust Bipedal Locomotion Learning

   https://doi.org/10.1109/ICRA57147.2024.10610978  

   Castillo, Guillermo A; Weng, Bowen; Zhang, Wei; Hereid, Ayonga (May 2024, IEEE)

   This paper presents a novel framework for learning robust bipedal walking by combining a data-driven state representation with a Reinforcement Learning (RL) based locomotion policy. The framework utilizes an autoencoder to learn a low-dimensional latent space that captures the complex dynamics of bipedal locomotion from existing locomotion data. This reduced dimensional state representation is then used as states for training a robust RL-based gait policy, eliminating the need for heuristic state selections or the use of template models for gait planning. The results demonstrate that the learned latent variables are disentangled and directly correspond to different gaits or speeds, such as moving forward, backward, or walking in place. Compared to traditional template model-based approaches, our framework exhibits superior performance and robustness in simulation. The trained policy effectively tracks a wide range of walking speeds and demonstrates good generalization capabilities to unseen scenarios. 

   more » « less
2. Template Model Inspired Task Space Learning for Robust Bipedal Locomotion

   https://doi.org/10.1109/IROS55552.2023.10341263  

   Castillo, Guillermo A.; Weng, Bowen; Yang, Shunpeng; Zhang, Wei; Hereid, Ayonga (October 2023, IEEE)

   This work presents a hierarchical framework for bipedal locomotion that combines a Reinforcement Learning (RL)-based high-level (HL) planner policy for the online generation of task space commands with a model-based low-level (LL) controller to track the desired task space trajectories. Different from traditional end-to-end learning approaches, our HL policy takes insights from the angular momentum-based linear inverted pendulum (ALIP) to carefully design the observation and action spaces of the Markov Decision Process (MDP). This simple yet effective design creates an insightful mapping between a low-dimensional state that effectively captures the complex dynamics of bipedal locomotion and a set of task space outputs that shape the walking gait of the robot. The HL policy is agnostic to the task space LL controller, which increases the flexibility of the design and generalization of the framework to other bipedal robots. This hierarchical design results in a learning-based framework with improved performance, data efficiency, and robustness compared with the ALIP model-based approach and state-of-the-art learning-based frameworks for bipedal locomotion. The proposed hierarchical controller is tested in three different robots, Rabbit, a five-link underactuated planar biped; Walker2D, a seven-link fully-actuated planar biped; and Digit, a 3D humanoid robot with 20 actuated joints. The trained policy naturally learns human-like locomotion behaviors and is able to effectively track a wide range of walking speeds while preserving the robustness and stability of the walking gait even under adversarial conditions. 

   more » « less
3. Learning Terrain-Aware Bipedal Locomotion via Reduced-Dimensional Perceptual Representations

   https://doi.org/10.1109/TCST.2026.3664022  

   Castillo, Guillermo A; Lodha, Himanshu; Hereid, Ayonga (January 2026, IEEE Transactions on Control Systems Technology)

   This work introduces a hierarchical strategy for terrain-aware bipedal locomotion that integrates reduced-dimensional perceptual representations to enhance the reinforcement learning (RL)-based high-level (HL) policies for real-time gait generation. Unlike end-to-end approaches, our framework leverages latent terrain encodings via a convolutional variational autoencoder (CNN-VAE) alongside reduced-order robot dynamics, optimizing the locomotion decision process with a compact state. We systematically analyze the impact of latent space dimensionality on learning efficiency and policy robustness. In addition, we extend our method to be history-aware, incorporating sequences of recent terrain observations into the latent representation to improve robustness. To address real-world feasibility, we introduce a distillation method to learn the latent representation directly from depth camera images and provide preliminary hardware validation by comparing simulated and real sensor data. We further validate our framework using the high-fidelity agility robotics (ARs) simulator, incorporating realistic sensor noise, state estimation, and actuator dynamics. The results confirm the robustness and adaptability of our method, underscoring its potential for hardware deployment. 

   more » « less
4. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control

   https://doi.org/10.1177/02783649241285161  

   Li, Zhongyu; Peng, Xue Bin; Abbeel, Pieter; Levine, Sergey; Berseth, Glen; Sreenath, Koushil (April 2025, The International Journal of Robotics Research)

   This paper presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal robots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing. Our RL-based controller incorporates a novel dual-history architecture, utilizing both a long-term and short-term input/output (I/O) history of the robot. This control architecture, when trained through the proposed end-to-end RL approach, consistently outperforms other methods across a diverse range of skills in both simulation and the real world. The study also delves into the adaptivity and robustness introduced by the proposed RL system in developing locomotion controllers. We demonstrate that the proposed architecture can adapt to both time-invariant dynamics shifts and time-variant changes, such as contact events, by effectively using the robot’s I/O history. Additionally, we identify task randomization as another key source of robustness, fostering better task generalization and compliance to disturbances. The resulting control policies can be successfully deployed on Cassie, a torque-controlled human-sized bipedal robot. This work pushes the limits of agility for bipedal robots through extensive real-world experiments. We demonstrate a diverse range of locomotion skills, including: robust standing, versatile walking, fast running with a demonstration of a 400-meter dash, and a diverse set of jumping skills, such as standing long jumps and high jumps. 

   more » « less
5. Center of mass kinematic reconstruction during steady-state walking using optimized template models

   https://doi.org/10.1371/journal.pone.0313156  

   Kelly, David J; Wensing, Patrick M (November 2024, PLOS ONE)

   Template models, such as the Bipedal Spring-Loaded Inverted Pendulum and the Virtual Pivot Point, have been widely used as low-dimensional representations of the complex dynamics in legged locomotion. Despite their ability to qualitatively match human walking characteristics like M-shaped ground reaction force (GRF) profiles, they often exhibit discrepancies when compared to experimental data, notably in overestimating vertical center of mass (CoM) displacement and underestimating gait event timings (touchdown/ liftoff). This paper hypothesizes that the constant leg stiffness of these models explains the majority of these discrepancies. The study systematically investigates the impact of stiffness variations on the fidelity of model fittings to human data, where an optimization framework is employed to identify optimal leg stiffness trajectories. The study also quantifies the effects of stiffness variations on salient characteristics of human walking (GRF profiles and gait event timing). The optimization framework was applied to 24 subjects walking at 40% to 145% preferred walking speed (PWS). The findings reveal that despite only modifying ground forces in one direction, variable leg stiffness models exhibited a >80% reduction in CoM error across both the B-SLIP and VPP models, while also improving prediction of human GRF profiles. However, the accuracy of gait event timing did not consistently show improvement across all conditions. The resulting stiffness profiles mimic walking characteristics of ankle push-off during double support and reduced CoM vaulting during single support. 

   more » « less