An official website of the United States government
Tracking microrobots is challenging, considering their minute size and high speed. As the field progresses towards developing microrobots for biomedical applications and conducting mechanistic studies in physiologically relevant media (e.g., collagen), this challenge is exacerbated by the dense surrounding environments with feature size and shape comparable to microrobots. Herein, we report Motion Enhanced Multi-level Tracker (MEMTrack), a robust pipeline for detecting and tracking microrobots using synthetic motion features, deep learning-based object detection, and a modified Simple Online and Real-time Tracking (SORT) algorithm with interpolation for tracking. Our object detection approach combines different models based on the object's motion pattern. We trained and validated our model using bacterial micro-motors in collagen (tissue phantom) and tested it in collagen and aqueous media. We demonstrate that MEMTrack accurately tracks even the most challenging bacteria missed by skilled human annotators, achieving precision and recall of 77% and 48% in collagen and 94% and 35% in liquid media, respectively. Moreover, we show that MEMTrack can quantify average bacteria speed with no statistically significant difference from the laboriously-produced manual tracking data. MEMTrack represents a significant contribution to microrobot localization and tracking, and opens the potential for vision-based deep learning approaches to microrobot control in dense and low-contrast settings. All source code for training and testing MEMTrack and reproducing the results of the paper have been made publicly available this https URL.
more »
« less
Motion Enhanced Multi‐Level Tracker (MEMTrack): A Deep Learning‐Based Approach to Microrobot Tracking in Dense and Low‐Contrast Environments
Tracking microrobots is challenging due to their minute size and high speed. In biomedical applications, this challenge is exacerbated by the dense surrounding environments with feature sizes and shapes comparable to microrobots. Herein, Motion Enhanced Multi‐level Tracker (MEMTrack) is introduced for detecting and tracking microrobots in dense and low‐contrast environments. Informed by the physics of microrobot motion, synthetic motion features for deep learning‐based object detection and a modified Simple Online and Real‐time Tracking (SORT)algorithm with interpolation are used for tracking. MEMTrack is trained and tested using bacterial micromotors in collagen (tissue phantom), achieving precision and recall of 76% and 51%, respectively. Compared to the state‐of‐the‐art baseline models, MEMTrack provides a minimum of 2.6‐fold higher precision with a reasonably high recall. MEMTrack's generalizability to unseen (aqueous) media and its versatility in tracking microrobots of different shapes, sizes, and motion characteristics are shown. Finally, it is shown that MEMTrack localizes objects with a root‐mean‐square error of less than 1.84 μm and quantifies the average speed of all tested systems with no statistically significant difference from the laboriously produced manual tracking data. MEMTrack significantly advances microrobot localization and tracking in dense and low‐contrast settings and can impact fundamental and translational microrobotic research.
more »
« less
- PAR ID:
- 10496423
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Advanced Intelligent Systems
- Volume:
- 6
- Issue:
- 4
- ISSN:
- 2640-4567
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract A magnetic object subject to an external rotating magnetic field would be rotated due to the alignment tendency between its internal magnetization and the field. Based on this principle, 12 shapes of swimming microrobots around 1 mm long were designed and 3D-printed using biodegradable materials Poly (ethylene glycol) diacrylate (PEDGA). Their surface was decorated with superparamagnetic iron oxide nanoparticles to provide magnetic responsivity. An array of 12 permanent magnets generated a rotating uniform magnetic field (∼100 mT) to impose magnetic torque, which induces a tumbling motion in the microrobot. We developed a dynamic model that captured the behavior of swimming microrobots of different shapes and showed good agreement with experimental results. Among these 12 shapes, we found that microrobots with equal length, width, and depth performed better. The observed translational speed of the hollow cube microrobot can exceed 17.84 mm s −1 (17.84 body lengths/s) under a rotating magnetic field of 5.26 Hz. These microrobots could swim to the targeted sites in a simplified vessel branch. And a finite element model was created to simulate the motion of the swimming microrobot under a flow rate of 0.062 m s −1 .more » « less
-
The implementation of two‐photon polymerization (TPP) in the microrobotics community has permitted the fabrication of complex 3D structures at the microscale, creating novel platforms with potential biomedical applications for minimizing procedure invasiveness and diagnosis accuracy. Although advanced functionalities for manipulation and drug delivery tasks have been explored, one remaining challenge is achieving improved visualization, identification, and accurate closed‐loop control of microscale robots. To enable this, distinguishable identifying and trackable features must be included on the microrobot. Toward this end, the construction of micro‐ and nanoscale patterns using TPP is demonstrated for the first time on microrobot surfaces with the intent of mimicking color‐expressing nanostructures present on beetles or butterflies. The patterns provide identification and tracking targets due to their vivid color expression under visible light. Helical and rectangular microrobots are designed with the topical patterns and further functionalized with magnetic materials to be externally actuated by magnetic fields. Vision‐based tracking of a 20 μm × 30 μm colored feature on a 100 μm‐long helical microrobot using a fixed angular position light source during microrobotic motion is shown. This versatile structural color patterning approach shows great potential for the visual differentiation of various microrobots and tracking for improved closed‐loop control.more » « less
-
Abstract The design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are few dynamic simulation tools available to accurately predict the motion or performance of untethered microrobots as they move over a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid body dynamic simulators, which implicitly assume that contact between two bodies can be modeled as point contact, are not suitable. In this paper, we present techniques for simulating the motion of microrobots where there can be intermittent and non-point contact between the robot and the substrate. We use these techniques to study the motion of tumbling microrobots of different shapes and select shapes that are optimal for improving locomotion performance. Simulation results are verified using experimental data on linear velocity, maximum climbable incline angle, and microrobot trajectory. Microrobots with improved geometry were fabricated, but limitations in the fabrication process resulted in unexpected manufacturing errors and material/size scale adjustments. The developed simulation model can incorporate these limitations and emulate their effect on the microrobot’s motion, reproducing the experimental behavior of the tumbling microrobots, further showcasing the effectiveness of having such a dynamic model.more » « less
-
This study investigates the motion characteristics of soft alginate microrobots in complex fluidic environments utilizing wireless magnetic fields for actuation. The aim is to explore the diverse motion modes that arise due to shear forces in viscoelastic fluids by employing snowman-shaped microrobots. Polyacrylamide (PAA), a water-soluble polymer, is used to create a dynamic environment with non-Newtonian fluid properties. Microrobots are fabricated via an extrusion-based microcentrifugal droplet method, successfully demonstrating the feasibility of both wiggling and tumbling motions. Specifically, the wiggling motion primarily results from the interplay between the viscoelastic fluid environment and the microrobots’ non-uniform magnetization. Furthermore, it is discovered that the viscoelasticity properties of the fluid influence the motion behavior of the microrobots, leading to non-uniform behavior in complex environments for microrobot swarms. Through velocity analysis, valuable insights into the relationship between applied magnetic fields and motion characteristics are obtained, facilitating a more realistic understanding of surface locomotion for targeted drug delivery purposes while accounting for swarm dynamics and non-uniform behavior.more » « less
