Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time‐varying magnetic field generated from a three‐axis array of electromagnets. The reinforcement learning agent learns successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulate the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.
This content will become publicly available on July 4, 2024
- NSF-PAR ID:
- 10437914
- Date Published:
- Journal Name:
- Smart Materials and Structures
- Volume:
- 32
- Issue:
- 8
- ISSN:
- 0964-1726
- Page Range / eLocation ID:
- 085014
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
-
This paper demonstrates a manipulation of snowman-shaped soft microrobots under a uniform rotating magnetic field. Each microsnowman robot consists of two biocompatible alginate microspheres with embedded magnetic nanoparticles. The soft microsnowmen were fabricated using a microfluidic device by following a centrifuge-based microfluidic droplet method. Under a uniform rotating magnetic field, the microsnowmen were rolled on the substrate surface, and the velocity response for increasing magnetic field frequencies was analyzed. Then, a microsnowman was rolled to follow different paths, which demonstrated directional controllability of the microrobot. Moreover, swarms of microsnowmen and single alginate microrobots were manipulated under the rotating magnetic field, and their velocity responses were analyzed for comparison.more » « less
-
Nano/microrobotic swimmers have many possible biomedical applications such as drug delivery and micro-manipulation. This paper examines one of the most promising classes of these: rigid magnetic microrobots that are propelled through bulk fluid by rotation induced by a rotating magnetic field. Propulsion corresponds to steadily rotating and translating solutions of the dynamics of such microrobots that co-rotate with the magnetic field. To be observed in experiments and be amenable to steering control, such solutions must also be stable to perturbations. In this paper, we analytically derive a criterion for the stability of such steadily rotating solutions for a microrobot made of soft magnetic materials, which have a magnetization that depends on the applied field. This result generalizes previous stability criteria we obtained for microrobots with a permanent magnetization.more » « less
-
Abstract The design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are few dynamic simulation tools available to accurately predict the motion or performance of untethered microrobots as they move over a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid body dynamic simulators, which implicitly assume that contact between two bodies can be modeled as point contact, are not suitable. In this paper, we present techniques for simulating the motion of microrobots where there can be intermittent and non-point contact between the robot and the substrate. We use these techniques to study the motion of tumbling microrobots of different shapes and select shapes that are optimal for improving locomotion performance. Simulation results are verified using experimental data on linear velocity, maximum climbable incline angle, and microrobot trajectory. Microrobots with improved geometry were fabricated, but limitations in the fabrication process resulted in unexpected manufacturing errors and material/size scale adjustments. The developed simulation model can incorporate these limitations and emulate their effect on the microrobot’s motion, reproducing the experimental behavior of the tumbling microrobots, further showcasing the effectiveness of having such a dynamic model.more » « less
-
more » « less
The implementation of two‐photon polymerization (TPP) in the microrobotics community has permitted the fabrication of complex 3D structures at the microscale, creating novel platforms with potential biomedical applications for minimizing procedure invasiveness and diagnosis accuracy. Although advanced functionalities for manipulation and drug delivery tasks have been explored, one remaining challenge is achieving improved visualization, identification, and accurate closed‐loop control of microscale robots. To enable this, distinguishable identifying and trackable features must be included on the microrobot. Toward this end, the construction of micro‐ and nanoscale patterns using TPP is demonstrated for the first time on microrobot surfaces with the intent of mimicking color‐expressing nanostructures present on beetles or butterflies. The patterns provide identification and tracking targets due to their vivid color expression under visible light. Helical and rectangular microrobots are designed with the topical patterns and further functionalized with magnetic materials to be externally actuated by magnetic fields. Vision‐based tracking of a 20 μm × 30 μm colored feature on a 100 μm‐long helical microrobot using a fixed angular position light source during microrobotic motion is shown. This versatile structural color patterning approach shows great potential for the visual differentiation of various microrobots and tracking for improved closed‐loop control.
