Disentangling the sources of visual motion in a dynamic scene during self-movement or ego motion is important for autonomous navigation and tracking. In the dynamic image segments of a video frame containing independently moving objects, optic flow relative to the next frame is the sum of the motion fields generated due to camera and object motion. The traditional ego-motion estimation methods assume the scene to be static, and the recent deep learning-based methods do not separate pixel velocities into object- and ego-motion components. We propose a learning-based approach to predict both ego-motion parameters and object-motion field (OMF) from image sequences using a convolutional autoencoder while being robust to variations due to the unconstrained scene depth. This is achieved by: 1) training with continuous ego-motion constraints that allow solving for ego-motion parameters independently of depth and 2) learning a sparsely activated overcomplete ego-motion field (EMF) basis set, which eliminates the irrelevant components in both static and dynamic segments for the task of ego-motion estimation. In order to learn the EMF basis set, we propose a new differentiable sparsity penalty function that approximates the number of nonzero activations in the bottleneck layer of the autoencoder and enforces sparsity more effectively than L1- and L2-norm-based penalties. Unlike the existing direct ego-motion estimation methods, the predicted global EMF can be used to extract OMF directly by comparing it against the optic flow. Compared with the state-of-the-art baselines, the proposed model performs favorably on pixelwise object- and ego-motion estimation tasks when evaluated on real and synthetic data sets of dynamic scenes.
more »
« less
Visual Vibration Tomography: Estimating Interior Material Properties from Monocular Video
An object’s interior material properties, while invisible to the human eye, determine motion observed on its surface. We propose an approach that esti- mates heterogeneous material properties of an object directly from a monoc- ular video of its surface vibrations. Specifically, we estimate Young’s modulus and density throughout a 3D object with known geometry. Knowledge of how these values change across the object is useful for characterizing defects and simulating how the object will interact with different environments. Traditional non-destructive testing approaches, which generally estimate homogenized material properties or the presence of defects, are expensive and use specialized instruments. We propose an approach that leverages monocular video to (1) measure an object’s sub-pixel motion and decompose this motion into image-space modes, and (2) directly infer spatially-varying Young’s modulus and density values from the observed image-space modes. On both simulated and real videos, we demonstrate that our approach is able to image material properties simply by analyzing surface motion. In particular, our method allows us to identify unseen defects on a 2D drum head from real, high-speed video.
more »
« less
- NSF-PAR ID:
- 10378315
- Publisher / Repository:
- IEEE
- Date Published:
- Journal Name:
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Page Range / eLocation ID:
- 16210 to 16219
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Tracking the 6D pose of objects in video sequences is important for robot manipulation. This task, however, in- troduces multiple challenges: (i) robot manipulation involves significant occlusions; (ii) data and annotations are troublesome and difficult to collect for 6D poses, which complicates machine learning solutions, and (iii) incremental error drift often accu- mulates in long term tracking to necessitate re-initialization of the object’s pose. This work proposes a data-driven opti- mization approach for long-term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object’s model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained only with synthetic data can work effectively over real images. Comprehensive experiments over benchmarks - existing ones as well as a new dataset with significant occlusions related to object manipulation - show that the proposed approach achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach is also the most computationally efficient among the alternatives and achieves a tracking frequency of 90.9Hz.more » « less
-
more » « less
Abstract Arctic icebergs, unconstrained sea ice floes, oil slicks, mangrove drifters, lost cargo containers, and other flotsam are known to move at 2%–4% of the prevailing wind velocity relative to the water, despite vast differences in the material properties, shapes, and sizes of objects. Here, we revisit the roles of density, aspect ratio, and skin and form drag in determining how an object is driven by winds and water currents. Idealized theoretical considerations show that although substantial differences exist for end members of the parameter space (either very thin or thick and very light or dense objects), most realistic cases of floating objects drift at approximately 3% of the free-stream wind velocity (measured outside an object’s surface boundary layer) relative to the water. This relationship, known as a long-standing rule of thumb for the drift of various types of floating objects, arises from the square root of the ratio of the density of air to that of water. We support our theoretical findings with flume experiments using floating objects with a range of densities and shapes.
-
Identifying the material properties of unknown media is an important scientific/engineering challenge in areas as varied as in-vivo tissue health diagnostics and metamaterial characterization. Currently, techniques exist to retrieve the material parameters of large unknown media from elastic wave scattering in free-space using analytical or numerical methods. However, applying these methods to small samples on the order of few wavelengths in diameter is challenging, as the fields scattered by these samples become significantly contaminated by diffraction from the sample edges. Here, we propose a method to extract the material parameters of small samples using convolutional neural networks trained to learn the mapping between far-field echoes and the material parameters. Networks were trained with synthetic time domain echo data obtained by simulating the free-space scattering of sound from unknown media underwater. Results show that neural networks can accurately predict effective material parameters such as mass density, bulk modulus, and shear modulus even when small training sets are used. Furthermore, we demonstrate in experiments executed in a water tank that the networks trained with synthetic data can accurately estimate the material properties of fabricated metamaterial samples from single-point echo measurements performed in the far-field. This work highlights the effectiveness of our approach to identify unknown media using far-field acoustic reflection dominated by diffraction fields and will open a new avenue toward acoustic sensing techniques.more » « less
-
Abstract We acquired a unique ambient vibration dataset from Castleton Tower, a 120 m high bedrock monolith located near Moab, Utah, to resolve dynamic and material properties of the landform. We identified the first two resonant modes at 0.8 and 1.0 Hz, which consist of mutually perpendicular, linearly polarized horizontal ground motion at the top of the tower. Damping ratios for these modes were low at ∼1%. We successfully reproduced field data in 3D numerical eigenfrequency simulation implementing a Young’s modulus of 7 GPa, a value ∼30% lower than measured on core samples. Our analysis confirms that modal deformation at the first resonant frequencies closely resembles that of a cantilever beam. The outcome is that with basic estimates of geometry and material properties, the resonant frequencies of other freestanding rock monoliths can be estimated a priori. Such estimates are crucial to evaluate the response of rock towers to external vibration inputs.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/CVPR52688.2022.01575
