An official website of the United States government
Segmentation of moving objects in dynamic scenes is a key process in scene understanding for navigation tasks. Classical cameras suffer from motion blur in such scenarios rendering them effete. On the contrary, event cameras, because of their high temporal resolution and lack of motion blur, are tailor-made for this problem. We present an approach for monocular multi-motion segmentation, which combines bottom-up feature tracking and top-down motion compensation into a unified pipeline, which is the first of its kind to our knowledge. Using the events within a time-interval, our method segments the scene into multiple motions by splitting and merging. We further speed up our method by using the concept of motion propagation and cluster keyslices.The approach was successfully evaluated on both challenging real-world and synthetic scenarios from the EV-IMO, EED, and MOD datasets and outperformed the state-of-the-art detection rate by 12%, achieving a new state-of-the-art average detection rate of 81.06%, 94.2% and 82.35% on the aforementioned datasets. To enable further research and systematic evaluation of multi-motion segmentation, we present and open-source a new dataset/benchmark called MOD++, which includes challenging sequences and extensive data stratification in-terms of camera and object motion, velocity magnitudes, direction, and rotational speeds.
more »
« less
A dynamic scale-mixture model of motion in natural scenes
Some of the most important tasks of visual and motor systems involve estimating the motion of objects and tracking them over time. Such systems evolved to meet the behavioral needs of the organism in its natural environment, and may therefore be adapted to the statistics of motion it is likely to encounter. By tracking the movement of individual points in movies of natural scenes, we begin to identify common properties of natural motion across scenes. As expected, objects in natural scenes move in a persistent fashion, with velocity correlations lasting hundreds of milliseconds. More subtly, but crucially, we find that the observed velocity distributions are heavy-tailed and can be modeled as a Gaussian scale-mixture. Extending this model to the time domain leads to a dynamic scale-mixture model, consisting of a Gaussian process multiplied by a positive scalar quantity with its own independent dynamics. Dynamic scaling of velocity arises naturally as a consequence of changes in object distance from the observer, and may approximate the effects of changes in other parameters governing the motion in a given scene. This modeling and estimation framework has implications for the neurobiology of sensory and motor systems, which need to cope with these fluctuations in scale in order to represent motion efficiently and drive fast and accurate tracking behavior.
more »
« less
- Award ID(s):
- 2235451
- PAR ID:
- 10609196
- Publisher / Repository:
- bioRxiv
- Date Published:
- Format(s):
- Medium: X
- Institution:
- bioRxiv
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In recent years, satellites capable of capturing videos have been developed and launched to provide high definition satellite videos that enable applications far beyond the capabilities of remotely sensed imagery. Moving object detection and moving object tracking are among the most essential and challenging tasks, but existing studies have mainly focused on vehicles. To accurately detect and then track more complex moving objects, specifically airplanes, we need to address the challenges posed by the new data. First, slow-moving airplanes may cause foreground aperture problem during detection. Second, various disturbances, especially parallax motion, may cause false detection. Third, airplanes may perform complex motions, which requires a rotation-invariant and scale-invariant tracking algorithm. To tackle these difficulties, we first develop an Improved Gaussian-based Background Subtractor (IPGBBS) algorithm for moving airplane detection. This algorithm adopts a novel strategy for background and foreground adaptation, which can effectively deal with the foreground aperture problem. Then, the detected moving airplanes are tracked by a Primary Scale Invariant Feature Transform (P-SIFT) keypoint matching algorithm. The P-SIFT keypoint of an airplane exhibits high distinctiveness and repeatability. More importantly, it provides a highly rotation-invariant and scale-invariant feature vector that can be used in the matching process to determine the new locations of the airplane in the frame sequence. The method was tested on a satellite video with eight moving airplanes. Compared with state-of-the-art algorithms, our IPGBBS algorithm achieved the best detection accuracy with the highest F1 score of 0.94 and also demonstrated its superiority on parallax motion suppression. The P-SIFT keypoint matching algorithm could successfully track seven out of the eight airplanes. Based on the tracking results, movement trajectories of the airplanes and their dynamic properties were also estimated.more » « less
-
Abstract Non-Line-Of-Sight (NLOS) imaging aims at recovering the 3D geometry of objects that are hidden from the direct line of sight. One major challenge with this technique is the weak available multibounce signal limiting scene size, capture speed, and reconstruction quality. To overcome this obstacle, we introduce a multipixel time-of-flight non-line-of-sight imaging method combining specifically designed Single Photon Avalanche Diode (SPAD) array detectors with a fast reconstruction algorithm that captures and reconstructs live low-latency videos of non-line-of-sight scenes with natural non-retroreflective objects. We develop a model of the signal-to-noise-ratio of non-line-of-sight imaging and use it to devise a method that reconstructs the scene such that signal-to-noise-ratio, motion blur, angular resolution, and depth resolution are all independent of scene depth suggesting that reconstruction of very large scenes may be possible.more » « less
-
We report a minimal microtubule-based motile system displaying signatures of unconventional diffusion. The system consists of a single model cargo driven by an ensemble of N340K NCD motors along a single microtubule. Despite the absence of cytosolic or cytoskeleton complexity, the system shows complex behavior, characterized by sub-diffusive motion for short time lag scales and linear mean squared displacement dependence for longer time lags. The latter is also shown to have non-Gaussian character and cannot be ascribed to a canonical diffusion process. We use single particle tracking and analysis at varying temperatures and motor concentrations to identify the origin of these behaviors as enzymatic activity of mutant NCD. Our results show that signatures of non-Gaussian diffusivities can arise as a result of an active process and suggest that some immotility of cargos observed in cells may reflect the ensemble workings of mechanochemical enzymes and need not always reflect the properties of the cytoskeletal network or the cytosol.more » « less
-
Abstract The stochastic trajectories of molecules in living cells, as well as the dynamics in many other complex systems, often exhibit memory in their path over long periods of time. In addition, these systems can show dynamic heterogeneities due to which the motion changes along the trajectories. Such effects manifest themselves as spatiotemporal correlations. Despite the broad occurrence of heterogeneous complex systems in nature, their analysis is still quite poorly understood and tools to model them are largely missing. We contribute to tackling this problem by employing an integral representation of Mandelbrot’s fractional Brownian motion that is compliant with varying motion parameters while maintaining long memory. Two types of switching fractional Brownian motion are analysed, with transitions arising from a Markovian stochastic process and scale-free intermittent processes. We obtain simple formulas for classical statistics of the processes, namely the mean squared displacement and the power spectral density. Further, a method to identify switching fractional Brownian motion based on the distribution of displacements is described. A validation of the model is given for experimental measurements of the motion of quantum dots in the cytoplasm of live mammalian cells that were obtained by single-particle tracking.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Posted Content:
- https://doi.org/10.1101/2023.10.19.563101
