skip to main content


Title: Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning
  more » « less
NSF-PAR ID:
10368592
Author(s) / Creator(s):
 ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Advanced Intelligent Systems
Volume:
4
Issue:
10
ISSN:
2640-4567
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Shared control of mobile robots integrates manual input with auxiliary autonomous controllers to improve the overall system performance. However, prior work that seeks to find the optimal shared control ratio needs an accurate human model, which is usually challenging to obtain. In this study, the authors develop an extended Twin Delayed Deep Deterministic Policy Gradient (DDPG) (TD3X)‐based shared control framework that learns to assist a human operator in teleoperating mobile robots optimally. The robot's states, shared control ratio in the previous time step, and human's control input is used as inputs to the reinforcement learning (RL) agent, which then outputs the optimal shared control ratio between human input and autonomous controllers without knowing the human model. Noisy softmax policies are developed to make the TD3X algorithm feasible under the constraint of a shared control ratio. Furthermore, to accelerate the training process and protect the robot, a navigation demonstration policy and a safety guard are developed. A neural network (NN) structure is developed to maintain the correlation of sensor readings among heterogeneous input data and improve the learning speed. In addition, an extended DAGGER (DAGGERX) human agent is developed for training the RL agent to reduce human workload. Robot simulations and experiments with humans in the loop are conducted. The results show that the DAGGERX human agent can simulate real human inputs in the worst‐case scenarios with a mean square error of 0.0039. Compared to the original TD3 agent, the TD3X‐based shared control system decreased the average collision number from 387.3 to 44.4 in a simplistic environment and 394.2 to 171.2 in a more complex environment. The maximum average return increased from 1043 to 1187 with a faster converge speed in the simplistic environment, while the performance is equally good in the complex environment because of the use of an advanced human agent. In the human subject tests, participants' average perceived workload was significantly lower in shared control than that in exclusively manual control (26.90 vs. 40.07,p = 0.013).

     
    more » « less
  2.  
    more » « less
  3. Smooth camber morphing aircraft offer increased control authority and improved aerodynamic efficiency. Smart material actuators have become a popular driving force for shape changes, capable of adhering to weight and size constraints and allowing for simplicity in mechanical design. As a step towards creating uncrewed aerial vehicles (UAVs) capable of autonomously responding to flow conditions, this work examines a multifunctional morphing airfoil’s ability to follow commands in various flows. We integrated an airfoil with a morphing trailing edge consisting of an antagonistic pair of macro fiber composites (MFCs), serving as both skin and actuator, and internal piezoelectric flex sensors to form a closed loop composite system. Closed loop feedback control is necessary to accurately follow deflection commands due to the hysteretic behavior of MFCs. Here we used a deep reinforcement learning algorithm, Proximal Policy Optimization, to control the morphing airfoil. Two neural controllers were trained in a simulation developed through time series modeling on long short-term memory recurrent neural networks. The learned controllers were then tested on the composite wing using two state inference methods in still air and in a wind tunnel at various flow speeds. We compared the performance of our neural controllers to one using traditional position-derivative feedback control methods. Our experimental results validate that the autonomous neural controllers were faster and more accurate than traditional methods. This research shows that deep learning methods can overcome common obstacles for achieving sufficient modeling and control when implementing smart composite actuators in an autonomous aerospace environment.

     
    more » « less
  4. Abstract A magnetic object subject to an external rotating magnetic field would be rotated due to the alignment tendency between its internal magnetization and the field. Based on this principle, 12 shapes of swimming microrobots around 1 mm long were designed and 3D-printed using biodegradable materials Poly (ethylene glycol) diacrylate (PEDGA). Their surface was decorated with superparamagnetic iron oxide nanoparticles to provide magnetic responsivity. An array of 12 permanent magnets generated a rotating uniform magnetic field (∼100 mT) to impose magnetic torque, which induces a tumbling motion in the microrobot. We developed a dynamic model that captured the behavior of swimming microrobots of different shapes and showed good agreement with experimental results. Among these 12 shapes, we found that microrobots with equal length, width, and depth performed better. The observed translational speed of the hollow cube microrobot can exceed 17.84 mm s −1 (17.84 body lengths/s) under a rotating magnetic field of 5.26 Hz. These microrobots could swim to the targeted sites in a simplified vessel branch. And a finite element model was created to simulate the motion of the swimming microrobot under a flow rate of 0.062 m s −1 . 
    more » « less
  5. This study investigates the motion characteristics of soft alginate microrobots in complex fluidic environments utilizing wireless magnetic fields for actuation. The aim is to explore the diverse motion modes that arise due to shear forces in viscoelastic fluids by employing snowman-shaped microrobots. Polyacrylamide (PAA), a water-soluble polymer, is used to create a dynamic environment with non-Newtonian fluid properties. Microrobots are fabricated via an extrusion-based microcentrifugal droplet method, successfully demonstrating the feasibility of both wiggling and tumbling motions. Specifically, the wiggling motion primarily results from the interplay between the viscoelastic fluid environment and the microrobots’ non-uniform magnetization. Furthermore, it is discovered that the viscoelasticity properties of the fluid influence the motion behavior of the microrobots, leading to non-uniform behavior in complex environments for microrobot swarms. Through velocity analysis, valuable insights into the relationship between applied magnetic fields and motion characteristics are obtained, facilitating a more realistic understanding of surface locomotion for targeted drug delivery purposes while accounting for swarm dynamics and non-uniform behavior.

     
    more » « less