skip to main content

Explore Research Products in the NSF-PAR

Title: Sensor-aided camera calibration for three dimensional digital image correlation measurements
Stereovision systems can extract full-field three-dimensional (3D) displacements of structures by processing the images collected with two synchronized cameras. To obtain accurate measurements, the cameras must be calibrated to account for lens distortion (i.e., intrinsic parameters) and compute the cameras’ relative position and orientation (i.e., extrinsic parameters). Traditionally, calibration is performed by taking photos of a calibration object (e.g., a checkerboard) with the two cameras. Because the calibration object must be similar in size to the targeted structure, measurements on large-scale structures are highly impractical. This research proposes a multi-sensor board with three inertial measurement units and a laser distance meter to compute the extrinsic parameters of a stereovision system and streamline the calibration procedure. In this paper, the performances of the proposed sensor-based calibration are compared with the accuracy of the traditional image-based calibration procedure. Laboratory experiments show that cameras calibrated with the multi-sensor board measure displacements with 95% accuracy compared to displacements obtained from cameras calibrated with the traditional procedure. The results of this study indicate that the sensor-based approach can increase the applicability of 3D digital image correlation measurements to large-scale structures while reducing the time and complexity of the calibration.  more » « less
Award ID(s):
2018992
NSF-PAR ID:
10456045
Author(s) / Creator(s):
; ; ; ; ;
Editor(s):
Fromme, Paul; Su, Zhongqing
Date Published:
Journal Name:
Health Monitoring of Structural and Biological Systems XVII
Volume:
12488
Page Range / eLocation ID:
77
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ( , Health Monitoring of Structural and Biological Systems XVI)
    Fromme, Paul ; Su, Zhongqing (Ed.)
    Three-dimensional digital image correlation (3D-DIC) has become a strong alternative to traditional contact-based techniques for structural health monitoring. 3D-DIC can extract the full-field displacement of a structure from a set of synchronized stereo images. Before performing 3D-DIC, a complex calibration process must be completed to obtain the stereovision system’s extrinsic parameters (i.e., cameras’ distance and orientation). The time required for the calibration depends on the dimensions of the targeted structure. For example, for large-scale structures, the calibration may take several hours. Furthermore, every time the cameras’ position changes, a new calibration is required to recalculate the extrinsic parameters. The approach proposed in this research allows determining the 3D-DIC extrinsic parameters using the data measured with commercially available sensors. The system utilizes three Inertial Measurement Units with a laser distance meter to compute the relative orientation and distance between the cameras. In this paper, an evaluation of the sensitivity of the newly developed sensor suite is provided by assessing the errors in the measurement of the extrinsic parameters. Analytical simulations performed on a 7.5 x 5.7 m field of view using the data retrieved from the sensors show that the proposed approach provides an accuracy of ~10-6 m and a promising way to reduce the complexity of 3D-DIC calibration. 
    more » « less
  2. ; ; ; ( , Frontiers in Neuroscience)
    Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates and insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision 1 . 
    more » « less
  3. ; ; ; ; ; ( , Advances in neural information processing systems)
    S. Koyejo ; S. Mohamed ; A. Agarwal ; D. Belgrave ; K. Cho ; A. Oh (Ed.)
    Labeling articulated objects in unconstrained settings has a wide variety of applications including entertainment, neuroscience, psychology, ethology, and many fields of medicine. Large offline labeled datasets do not exist for all but the most common articulated object categories (e.g., humans). Hand labeling these landmarks within a video sequence is a laborious task. Learned landmark detectors can help, but can be error-prone when trained from only a few examples. Multi-camera systems that train fine-grained detectors have shown significant promise in detecting such errors, allowing for self-supervised solutions that only need a small percentage of the video sequence to be hand-labeled. The approach, however, is based on calibrated cameras and rigid geometry, making it expensive, difficult to manage, and impractical in real-world scenarios. In this paper, we address these bottlenecks by combining a non-rigid 3D neural prior with deep flow to obtain high-fidelity landmark estimates from videos with only two or three uncalibrated, handheld cameras. With just a few annotations (representing 1−2 % of the frames), we are able to produce 2D results comparable to state-of-the-art fully supervised methods, along with 3D reconstructions that are impossible with other existing approaches. Our Multi-view Bootstrapping in the Wild (MBW) approach demonstrates impressive results on standard human datasets, as well as tigers, cheetahs, fish, colobus monkeys, chimpanzees, and flamingos from videos captured casually in a zoo. We release the codebase for MBW as well as this challenging zoo dataset consisting of image frames of tail-end distribution categories with their corresponding 2D and 3D labels generated from minimal human intervention. 
    more » « less
  4. ; ; ; ; ; ( , Advances in neural information processing systems)
    Labeling articulated objects in unconstrained settings have a wide variety of applications including entertainment, neuroscience, psychology, ethology, and many fields of medicine. Large offline labeled datasets do not exist for all but the most common articulated object categories (e.g., humans). Hand labeling these landmarks within a video sequence is a laborious task. Learned landmark detectors can help, but can be error-prone when trained from only a few examples. Multi-camera systems that train fine-grained detectors have shown significant promise in detecting such errors, allowing for self-supervised solutions that only need a small percentage of the video sequence to be hand-labeled. The approach, however, is based on calibrated cameras and rigid geometry, making it expensive, difficult to manage, and impractical in real-world scenarios. In this paper, we address these bottlenecks by combining a non-rigid 3D neural prior with deep flow to obtain high-fidelity landmark estimates from videos with only two or three uncalibrated, handheld cameras. With just a few annotations (representing 1-2% of the frames), we are able to produce 2D results comparable to state-of-the-art fully supervised methods, along with 3D reconstructions that are impossible with other existing approaches. Our Multi-view Bootstrapping in the Wild (MBW) approach demonstrates impressive results on standard human datasets, as well as tigers, cheetahs, fish, colobus monkeys, chimpanzees, and flamingos from videos captured casually in a zoo. We release the codebase for MBW as well as this challenging zoo dataset consisting image frames of tail-end distribution categories with their corresponding 2D, 3D labels generated from minimal human intervention. 
    more » « less
  5. ; ; ; ; ; ( , ACM Transactions on Design Automation of Electronic Systems)

    Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. However, the typical three-stage pipeline (i.e., training, pruning, and retraining (fine-tuning)) significantly increases the overall training time. In this article, we develop a systematic weight-pruning optimization approach based on surrogate Lagrangian relaxation (SLR), which is tailored to overcome difficulties caused by the discrete nature of the weight-pruning problem. We further prove that our method ensures fast convergence of the model compression problem, and the convergence of the SLR is accelerated by using quadratic penalties. Model parameters obtained by SLR during the training phase are much closer to their optimal values as compared to those obtained by other state-of-the-art methods. We evaluate our method on image classification tasks using CIFAR-10 and ImageNet with state-of-the-art multi-layer perceptron based networks such as MLP-Mixer; attention-based networks such as Swin Transformer; and convolutional neural network based models such as VGG-16, ResNet-18, ResNet-50, ResNet-110, and MobileNetV2. We also evaluate object detection and segmentation tasks on COCO, the KITTI benchmark, and the TuSimple lane detection dataset using a variety of models. Experimental results demonstrate that our SLR-based weight-pruning optimization approach achieves a higher compression rate than state-of-the-art methods under the same accuracy requirement and also can achieve higher accuracy under the same compression rate requirement. Under classification tasks, our SLR approach converges to the desired accuracy × faster on both of the datasets. Under object detection and segmentation tasks, SLR also converges 2× faster to the desired accuracy. Further, our SLR achieves high model accuracy even at the hardpruning stage without retraining, which reduces the traditional three-stage pruning into a two-stage process. Given a limited budget of retraining epochs, our approach quickly recovers the model’s accuracy.

     
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email