skip to main content


Title: Asynchronous parallel reinforcement learning for optimizing propulsive performance in fin ray control
Abstract

Fish fin rays constitute a sophisticated control system for ray-finned fish, facilitating versatile locomotion within complex fluid environments. Despite extensive research on the kinematics and hydrodynamics of fish locomotion, the intricate control strategies in fin-ray actuation remain largely unexplored. While deep reinforcement learning (DRL) has demonstrated potential in managing complex nonlinear dynamics; its trial-and-error nature limits its application to problems involving computationally demanding environmental interactions. This study introduces a cutting-edge off-policy DRL algorithm, interacting with a fluid–structure interaction (FSI) environment to acquire intricate fin-ray control strategies tailored for various propulsive performance objectives. To enhance training efficiency and enable scalable parallelism, an innovative asynchronous parallel training (APT) strategy is proposed, which fully decouples FSI environment interactions and policy/value network optimization. The results demonstrated the success of the proposed method in discovering optimal complex policies for fin-ray actuation control, resulting in a superior propulsive performance compared to the optimal sinusoidal actuation function identified through a parametric grid search. The merit and effectiveness of the APT approach are also showcased through comprehensive comparison with conventional DRL training strategies in numerical experiments of controlling nonlinear dynamics.

 
more » « less
PAR ID:
10560695
Author(s) / Creator(s):
; ; ; ;
Publisher / Repository:
Springer Science + Business Media
Date Published:
Journal Name:
Engineering with Computers
ISSN:
0177-0667
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Fish locomotion emerges from diverse interactions among deformable structures, surrounding fluids and neuromuscular activations, i.e. fluid–structure interactions (FSI) controlled by fish's motor systems. Previous studies suggested that such motor-controlled FSI may possess embodied traits. However, their implications in motor learning, neuromuscular control, gait generation, and swimming performance remain to be uncovered. Using robot models, we studied the embodied traits in fish-inspired swimming. We developed modular robots with various designs and used central pattern generators (CPGs) to control the torque acting on robot body. We used reinforcement learning to learn CPG parameters for maximizing the swimming speed. The results showed that motor frequency converged faster than other parameters, and the emergent swimming gaits were robust against disruptions applied to motor control. For all robots and frequencies tested, swimming speed was proportional to the mean undulation velocity of body and caudal-fin combined, yielding an invariant, undulation-based Strouhal number. The Strouhal number also revealed two fundamental classes of undulatory swimming in both biological and robotic fishes. The robot actuators were also demonstrated to function as motors, virtual springs and virtual masses. These results provide novel insights in understanding fish-inspired locomotion.

     
    more » « less
  2. In this study, the effects of antagonistic muscle actuation on the propulsion of a bilaminar-structure fish fin ray were investigated using a two-dimensional computational flow–structure interaction (FSI) model. The structure and material properties of the model were based on the realistic biological data of the sunfish fin. The effect of muscle actuation was modelled using root displacement offset between the two hemitrichs. Parametric FSI simulations were conducted by assuming a sinusoidal function of the offset over a cycle and varying the amplitude and phase difference between the actuations and pitching/plunging motions. The results show that the phase of muscle actuation is a critical factor affecting its effects. Three performance regions can be identified with different phase ranges, including a thrust-favour region, an efficiency-favour region and a thrust-efficiency-unfavour region. In each region, the relationships among the root actuations, fin-ray kinematics, vortex dynamics and resulting performance are studied and discussed. Furthermore, a strong positive correlation between the trailing–leading amplitude ratio and thrust coefficient as well as a negative relationship between the efficiency and angle of attack at the centre of mass of the fin ray are observed.

     
    more » « less
  3. Undulatory fin motions in fish-like robots are typically created using intricate arrays of servo motors. Motor arrays offer impressive versatility in terms of kinematics, but their complexity leads to constraints on size, hydrodynamic force production, and power consumption, particularly when studying propulsive performance at high-frequencies. Here we present an alternative design that uses a single motor and a tunable rotary cam-train system to achieve a spectrum of fin motions running from oscillation (wavenumber < 1) to undulation (wavenumber > 1). Our platform enables thrust, lift, power, and wake measurements at prescribed pitch amplitudes, frequencies, and wavenumbers. We demonstrated the platform’s oscillating and undulating capabilities via force and wake measurements in a water tank. Studies of fin wavenumber offer design insights for fish-like underwater robots, particularly those with stingray-inspired designs. 
    more » « less
  4. Gorb, S. (Ed.)
    Through computational fluid dynamics (CFD) simulations of a model manta ray body, the hydrodynamic role of manta-like bioinspired flapping is investigated. The manta ray model motion is reconstructed from synchronized high-resolution videos of manta ray swimming. Rotation angles of the model skeletal joints are altered to scale the pitching and bending, resulting in eight models with different pectoral fin pitching and bending ratios. Simulations are performed using an in-house developed immersed boundary method-based numerical solver. Pectoral fin pitching ratio (PR) is found to have significant implications in the thrust and efficiency of the manta model. This occurs due to more optimal vortex formation and shedding caused by the lower pitching ratio. Leading edge vortexes (LEVs) formed on the bottom of the fin, a characteristic of the higher PR cases, produced parasitic low pressure that hinders thrust force. Lowering the PR reduces the influence of this vortex while another LEV that forms on the top surface of the fin strengthens it. A moderately high bending ratio (BR) can slightly reduce power consumption. Finally, by combining a moderately high BR = 0.83 with PR = 0.67, further performance improvements can be made. This enhanced understanding of manta-inspired propulsive mechanics fills a gap in our understanding of the manta-like mobuliform locomotion. This motivates a new generation of manta-inspired robots that can mimic the high speed and efficiency of their biological counterpart.

     
    more » « less
  5. MDPI (Ed.)
    Through computational fluid dynamics (CFD) simulations of a model manta ray body, the hydrodynamic role of manta-like bioinspired flapping is investigated. The manta ray model motion is reconstructed from synchronized high-resolution videos of manta ray swimming. Rotation angles of the model skeletal joints are altered to scale the pitching and bending, resulting in eight models with different pectoral fin pitching and bending ratios. Simulations are performed using an in-house developed immersed boundary method-based numerical solver. Pectoral fin pitching ratio (PR) is found to have significant implications in the thrust and efficiency of the manta model. This occurs due to more optimal vortex formation and shedding caused by the lower pitching ratio. Leading edge vortexes (LEVs) formed on the bottom of the fin, a characteristic of the higher PR cases, produced parasitic low pressure that hinders thrust force. Lowering the PR reduces the influence of this vortex while another LEV that forms on the top surface of the fin strengthens it. A moderately high bending ratio (BR) can slightly reduce power consumption. Finally, by combining a moderately high BR = 0.83 with PR = 0.67, further performance improvements can be made. This enhanced understanding of manta-inspired propulsive mechanics fills a gap in our understanding of the manta-like mobuliform locomotion. This motivates a new generation of manta-inspired robots that can mimic the high speed and efficiency of their biological counterpart 
    more » « less