An official website of the United States government
Video scene analysis is a well-investigated area where researchers have devoted efforts to detect and classify people and objects in the scene. However, real-life scenes are more complex: the intrinsic states of the objects (e.g., machine operating states or human vital signals) are often overlooked by vision-based scene analysis. Recent work has proposed a radio frequency (RF) sensing technique, wireless vibrometry, that employs wireless signals to sense subtle vibrations from the objects and infer their internal states. We envision that the combination of video scene analysis with wireless vibrometry form a more comprehensive understanding of the scene, namely "rich scene analysis". However, the RF sensors used in wireless vibrometry only provide time series, and it is challenging to associate these time series data with multiple real-world objects. We propose a real-time RF-vision sensor fusion system, Capricorn, that efficiently builds a cross-modal correspondence between visual pixels and RF time series to better understand the complex natures of a scene. The vision sensors in Capricorn model the surrounding environment in 3D and obtain the distances of different objects. In the RF domain, the distance is proportional to the signal time-of-flight (ToF), and we can leverage the ToF to separate the RF time series corresponding to each object. The RF-vision sensor fusion in Capricorn brings multiple benefits. The vision sensors provide environmental contexts to guide the processing of RF data, which helps us select the most appropriate algorithms and models. Meanwhile, the RF sensor yields additional information that is originally invisible to vision sensors, providing insight into objects' intrinsic states. Our extensive evaluations show that Capricorn real-timely monitors multiple appliances' operating status with an accuracy of 97%+ and recovers vital signals like respirations from multiple people. A video (https://youtu.be/b-5nav3Fi78) demonstrates the capability of Capricorn.
more »
« less
Validation of a Novel Stereo Vibrometry Technique for Spiderweb Signal Analysis
From courtship rituals, to prey identification, to displays of rivalry, a spider’s web vibrates with a symphony of information. Examining the modality of information being transmitted and how spiders interact with this information could lead to new understanding how spiders perceive the world around them through their webs, and new biological and engineering techniques that leverage this understanding. Spiders interact with their webs through a variety of body motions, including abdominal tremors, bounces, and limb jerks along threads of the web. These signals often create a large enough visual signature that the web vibrations can be analyzed using video vibrometry on high-speed video of the communication exchange. Using video vibrometry to examine these signals has numerous benefits over the conventional method of laser vibrometry, such as the ability to analyze three-dimensional vibrations and the ability to take measurements from anywhere in the web, including directly from the body of the spider itself. In this study, we developed a method of three-dimensional vibration analysis that combines video vibrometry with stereo vision, and verified this method against laser vibrometry on a black widow spiderweb that was experiencing rivalry signals from two female spiders.
more »
« less
- Award ID(s):
- 1504428
- PAR ID:
- 10394242
- Date Published:
- Journal Name:
- Insects
- Volume:
- 13
- Issue:
- 4
- ISSN:
- 2075-4450
- Page Range / eLocation ID:
- 310
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Webs play many essential roles in spider biology, including communication, prey capture, locomotion, and reproduction. One interesting morphological feature of many spiders is the cribellum, a plate located near the silk-producing structures called spinnerets, and used to create a special type of matted silk that captures prey mechanically, instead of with glue droplets used by many orb-weaving spiders. The cribellum is hypothesized to have been present in the ancestor of all araneomorph spiders, but lost multiple times over the course of spider evolution. One group of spiders, the ‘marronoids’, shows a pattern of repeated loss and gain of this structure, placing them at a transitional position in the evolution of spider webs, with further implications for the web capture strategy, and other ecological conditions such as water-associated habitat. Studying the timing of the loss of the cribellum may yield insight to the cryptic ecology and morphology of the marranoid clade, and more broadly, araneomorph spiders. We use comparative phylogenetic methods to identify ancestral states of morphological and behavioral characters, and examine divergence dates with fossil calibrations. To do this, 98 representative spiders from the marronoid clade were coded by zoogeographic region, distribution proximity to a body of water and type, web type, and observed aquatic behavior. The morphology of the cribellum and spinnerets was assessed using 42 characters with multiple states. We identified patterns of evolution of the cribellum and aquatic habitat associations in the context of phylogeny, and geologic time.more » « less
-
Spiders use various combinations of silks, adhesives, and behaviors to ensnare prey. One common but difficult-to-catch prey is moths. They easily escape typical orb-webs because their bodies are covered in tiny sacrificial scales that flake off when in contact with the web’s adhesives. This defense is defeated by spiders of the sub-family of Cyrtarachninae—moth-catching specialists who combine changes in orb-web structure, predatory behavior, and chemistry of the aggregate glue placed in those webs. The most extreme changes in web structure are shown by the bolas spiders which create only one or two glue droplets at the end of a single thread. They prey on male moths by releasing pheromones to draw them close. Here, we confirm the hypothesis that the spinning behavior of the spider is directly used to spin its glue droplets using a high-speed video camera to observe the captured behavior of the bolas spider Cladomelea akermani as it actively spins its body and bolas. We use the kinematics of the spider and bolas to begin to quantify and model the physical and mechanical properties of the bolas during prey capture. We then examine why this species chooses to spin its body, an energetically costly behavior, during prey capture. We test the hypothesis that spinning helps to spread pheromones by creating a computational fluid dynamics model of airflow within an open field and comparing it to that of airflow within a tree, a common environment for bolas spiders that do not spin. Spinning in an open environment creates turbulent air, spreading pheromones further and creating a pocket of pheromones. Conversely, spinning within a tree does little to affect the natural airflow.more » « less
-
Intelligent systems commonly employ vision sensors like cameras to analyze a scene. Recent work has proposed a wireless sensing technique, wireless vibrometry, to enrich the scene analysis generated by vision sensors. Wireless vibrometry employs wireless signals to sense subtle vibrations from the objects and infer their internal states. However, it is difficult for pure Radio-Frequency (RF) sensing systems to obtain objects' visual appearances (e.g., object types and locations), especially when an object is inactive. Thus, most existing wireless vibrometry systems assume that the number and the types of objects in the scene are known. The key to getting rid of these presumptions is to build a connection between wireless sensor time series and vision sensor images. We present Capricorn, a vision-guided wireless vibrometry system. In Capricorn, the object type information from vision sensors guides the wireless vibrometry system to select the most appropriate signal processing pipeline. The object tracking capability in computer vision also helps wireless systems efficiently detect and separate vibrations from multiple objects in real time.more » « less
-
Spiders use various combinations of silks, adhesives, and behaviors to ensnare and trap prey. A common but difficult to catch prey in most spider habitats are moths. They easily escape typical orb-webs because their bodies are covered in sacrificial scales that flake off when in contact with the web’s adhesives. This defense is defeated by spiders of the sub-family of Cyrtarachninae, moth-catching specialists who combine changes in orb-web structure, predatory behavior, and chemistry of the aggregate glue placed in those webs. The most extreme changes in web structure are shown by bolas spiders, who create a solitary capture strand containing only one or two glue droplets at the end of a single thread. They prey on male moths by releasing pheromones to draw them within range of their bolas, which they flick to ensnare the moth. We used a high-speed video camera to capture the behavior of the bolas spider Mastophora hutchinsoni. We calculated the kinematics of spiders and moths in the wild to model the physical and mechanical properties of the bolas during prey capture, the behavior of the moth, and how these factors lead to successful prey capture. We created a numerical model to explain the mechanical behavior of the bolas silk during prey capture. Our kinematic analysis shows that the material properties of the aggregate glue bolas of M. hutchinsoni are distinct from that of the other previously analyzed moth-specialist, Cyrtarachne akirai. The spring-like behavior of the M. hutchinsoni bolas suggests it spins a thicker liquid.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/insects13040310
