An official website of the United States government
Real-time three-dimensional single-particle tracking (RT-3D-SPT) allows continuous detection of individual freely diffusing objects with high spatiotemporal precision by applying closed-loop active feedback in an optical microscope. However, the current tracking speed in RT-3D-SPT is primarily limited by the response time of control actuators, impeding long-term observation of fast diffusive objects such as single molecules. Here, we present an RT-3D-SPT system with improved tracking performance by replacing the XY piezoelectric stage with a galvo scanning mirror with an approximately five-time faster response rate (~5 kHz). Based on the previously developed 3D single-molecule active real-time tracking (3D-SMART), this new implementation with a fast-responding galvo mirror eliminates the mechanical movement of the sample and allows more rapid response to particle motion. The improved tracking performance of the galvo mirror-based implementation is verified through simulation and proof-of-principle experiments. Fluorescent nanoparticles and ~ 1 kB double-stranded DNA molecules were tracked via both the original piezoelectric stage and new galvo mirror implementations. With the new galvo-based implementation, notable increases in tracking duration, localization precision, and the degree to which the objects are locked to the center of the detection volume were observed. These results suggest faster control response elements can expand RT-3D-SPT to a broader range of chemical and biological systems.
more »
« less
Miniaturization and geometric optimization of SteamVR active optical trackers
Active tracking enables higher precision in tracking the positions, orientations, and states of the virtualized objects. STEAMVR Lighthouse tracking base-stations can be used for tracking specific objects. However, current solutions are bulky and costly. The overall goal of this research work was to reduce the size and cost of active VR trackers to enable their attachment to ever smaller physical tools and objects to be tracked in the real world and displayed in a virtual reality environment.
more »
« less
- Award ID(s):
- 1918045
- PAR ID:
- 10432264
- Editor(s):
- Kress, Bernard C.; Peroz, Christophe
- Date Published:
- Journal Name:
- Proc. SPIE 12449, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV
- Volume:
- 12449
- Issue:
- 21
- Page Range / eLocation ID:
- 129
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The eukaryotic cell's cytoskeleton is a prototypical example of an active material: objects embedded within it are driven by molecular motors acting on the cytoskeleton, leading to anomalous diffusive behavior. Experiments tracking the behavior of cell-attached objects have observed anomalous diffusion with a distribution of displacements that is non-Gaussian, with heavy tails. This has been attributed to “cytoquakes” or other spatially extended collective effects. We show, using simulations and analytical theory, that a simple continuum active gel model driven by fluctuating force dipoles naturally creates heavy power-law tails in cytoskeletal displacements. We predict that this power law exponent should depend on the geometry and dimensionality of where force dipoles are distributed through the cell; we find qualitatively different results for force dipoles in a 3D cytoskeleton and a quasi-two-dimensional cortex. We then discuss potential applications of this model both in cells and in synthetic active gels.more » « less
-
Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging’s depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object’s depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object’s bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object’s depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.more » « less
-
This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and performs online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning features of objects it has interacted with and updates their reconstructed models. Whenever an instance of a previously manipulated object reappears, the system aims to first recognize it and then register its previously reconstructed model given the current observation. This step greatly reduces object shape uncertainty allowing the system to even reason for parts of objects, which are currently not observable. This also results in better manipulation efficiency as it reduces the need for active perception of the target object during manipulation. To get a reusable reconstructed model, the proposed pipeline adopts: i) TSDF for object representation, and ii) a variant of the standard particle filter algorithm for pose estimation and tracking of the partial object model. Furthermore, an effective way to construct and maintain a dataset of manipulated objects is presented. A sequence of real-world manipulation experiments is performed. They show how future manipulation tasks become more effective and efficient by reusing reconstructed models of previously manipulated objects, which were generated during their prior manipulation, instead of treating objects as novel every time.more » « less
-
Zhou, Jianhong; Osten, Wolfgang; Nikolaev, Dmitry P. (Ed.)Despite recent advances in deep learning, object detection and tracking still require considerable manual and computational effort. First, we need to collect and create a database of hundreds or thousands of images of the target objects. Next we must annotate or curate the images to indicate the presence and position of the target objects within those images. Finally, we must train a CNN (convolution neural network) model to detect and locate the target objects in new images. This training is usually computationally intensive, consists of thousands of epochs, and can take tens of hours for each target object. Even after the model training in completed, there is still a chance of failure if the real-time tracking and object detection phases lack sufficient accuracy, precision, and/or speed for many important applications. Here we present a system and approach which minimizes the computational expense of the various steps in the training and real-time tracking process outlined above of for applications in the development of mixed-reality science laboratory experiences by using non-intrusive object-encoding 2D QR codes that are mounted directly onto the surfaces of the lab tools to be tracked. This system can start detecting and tracking it immediately and eliminates the laborious process of acquiring and annotating a new training dataset for every new lab tool to be tracked.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1117/12.2652267
