Video scene analysis is a well-investigated area where researchers have devoted efforts to detect and classify people and objects in the scene. However, real-life scenes are more complex: the intrinsic states of the objects (e.g., machine operating states or human vital signals) are often overlooked by vision-based scene analysis. Recent work has proposed a radio frequency (RF) sensing technique, wireless vibrometry, that employs wireless signals to sense subtle vibrations from the objects and infer their internal states. We envision that the combination of video scene analysis with wireless vibrometry form a more comprehensive understanding of the scene, namely "rich scene analysis". However, the RF sensors used in wireless vibrometry only provide time series, and it is challenging to associate these time series data with multiple real-world objects. We propose a real-time RF-vision sensor fusion system, Capricorn, that efficiently builds a cross-modal correspondence between visual pixels and RF time series to better understand the complex natures of a scene. The vision sensors in Capricorn model the surrounding environment in 3D and obtain the distances of different objects. In the RF domain, the distance is proportional to the signal time-of-flight (ToF), and we can leverage the ToF to separate the RF time series corresponding to each object. The RF-vision sensor fusion in Capricorn brings multiple benefits. The vision sensors provide environmental contexts to guide the processing of RF data, which helps us select the most appropriate algorithms and models. Meanwhile, the RF sensor yields additional information that is originally invisible to vision sensors, providing insight into objects' intrinsic states. Our extensive evaluations show that Capricorn real-timely monitors multiple appliances' operating status with an accuracy of 97%+ and recovers vital signals like respirations from multiple people. A video (https://youtu.be/b-5nav3Fi78) demonstrates the capability of Capricorn.
more »
« less
Dispersed passive RF-sensing for 3D structural health monitoring
We propose a sensing system comprising a large network of tiny, battery-less, Radio Frequency (RF)-powered sensors that use backscatter communication. The sensors use an entirely passive technique to 'sense' the parameters of the wireless channel between themselves. Since the material properties influence RF channels, this fine-grain sensing can uncover multiple material properties both at a large scale and fine spatial resolution. In this paper, we study the feasibility of the proposed passive technique for monitoring parameters of material in which the sensors are embedded. We performed a set of experiments where the sensor-to-sensor wireless channel parameters are well-defined using physics-based modeling, and we compared the theoretical and experimentally obtained values. For some material parameters of interest, like humidity or strain, the relationship with the observed wireless channel parameters have to be modeled relying on data-driven approaches. The initial experiments show an observable difference in the sensor-to-sensor channel phase with variation in the applied weights.
more »
« less
- NSF-PAR ID:
- 10466328
- Date Published:
- Journal Name:
- ITU Journal on Future and Evolving Technologies
- Volume:
- 3
- Issue:
- 2
- ISSN:
- 2616-8375
- Page Range / eLocation ID:
- 535 to 544
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Intelligent systems commonly employ vision sensors like cameras to analyze a scene. Recent work has proposed a wireless sensing technique, wireless vibrometry, to enrich the scene analysis generated by vision sensors. Wireless vibrometry employs wireless signals to sense subtle vibrations from the objects and infer their internal states. However, it is difficult for pure Radio-Frequency (RF) sensing systems to obtain objects' visual appearances (e.g., object types and locations), especially when an object is inactive. Thus, most existing wireless vibrometry systems assume that the number and the types of objects in the scene are known. The key to getting rid of these presumptions is to build a connection between wireless sensor time series and vision sensor images. We present Capricorn, a vision-guided wireless vibrometry system. In Capricorn, the object type information from vision sensors guides the wireless vibrometry system to select the most appropriate signal processing pipeline. The object tracking capability in computer vision also helps wireless systems efficiently detect and separate vibrations from multiple objects in real time.more » « less
-
The wireless signal propagates via multipath arising from different reflections and penetration between a transmitter and receiver. Extracting multipath profiles (e.g., delay and Doppler along each path) from received signals enables many important applications, such as channel prediction and crossband channel estimation (i.e., estimating the channel on a different frequency). The benefit of multipath estimation further increases with mobility since the channel in that case is less stable and more important to track. Yet high-speed mobility poses significant challenges to multipath estimation. In this paper, instead of using time-frequency domain channel representation, we leverage the delay-Doppler domain representation to accurately extract and predict multipath properties. Specifically, we use impulses in the delay-Doppler domain as pilots to estimate the multipath parameters and apply the multipath information to predicting wireless channels as an example application. Our design rationale is that mobility is more predictable than the wireless channel since mobility has inertial while the wireless channel is the outcome of a complicated interaction between mobility, multipath, and noise. We evaluate our approach via both acoustic and RF experiments, including vehicular experiments using USRP. Our results show that the estimated multipath matches the ground truth, and the resulting channel prediction is more accurate than the traditional channel prediction schemes.more » « less
-
Large scale networks of intelligent sensors that can function without any batteries will have enormous implications in applications that range from smart spaces to structural and environmental monitoring. RF tags present an amenable platform for sensor integration as the backscatter communication offers low energy cost of communication. Current RF tags either use extremely low-power sensors or perform tasks of tag localization and identification based on the strength of the backscatter signal. We present a technique for estimation of amplitude and phase of the tag-to-tag channel that can be performed with very limited computational and energy resources. This enables monitoring of the interactions between tagged objects and activities around tags, as well as assessment of a variety of engineering structures. Experimental results demonstrate high resolution in the amplitude and phase channel measurement at a distances ranging from 22 cm to 1.34 m.more » « less
-
Conventional catheter- or probe-based in vivo biomedical sensing is uncomfortable, inconvenient, and sometimes infeasible for longterm monitoring. Existing implantable sensors often require an invasive procedure for sensor placement. Untethered soft robots with the capability to deliver the sensor to the desired monitoring point hold great promise for minimally invasive biomedical sensing. Inspired by the locomotion modes of snakes, we present here a soft kirigami robot for sensor deployment and real-time wireless sensing. The locomotion mechanism of the soft robot is achieved by kirigami patterns that offer asymmetric tribological properties that mimic the skin of the snake. The robot exhibits good deployability, excellent load capacity (up to 150 times its own weight), high-speed locomotion (0.25 body length per step), and wide environmental adaptability with multimodal movements (obstacle crossing, locomotion in wet and dry conditions, climbing, and inverted crawling). When integrated with passive sensors, the versatile soft robot can locomote inside the human body, deliver the passive sensor to the desired location, and hold the sensor in place for real-time monitoring in a minimally invasive manner. The proof-of-concept prototype demonstrates that the platform can perform real-time impedance monitoring for the diagnosis of gastroesophageal reflux disease.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.52953/ZQYZ8264
