More Like this

1. Multi-Frequency RF Sensor Fusion for Word-Level Fluent ASL Recognition

   https://doi.org/10.1109/JSEN.2021.3078339  

   Gurbuz, Sevgi Z. ; Mahbubur Rahman, M. ; Kurtoglu, Emre ; Malaia, Evie ; Gurbuz, Ali C. ; Griffin, Darrin J. ; Crawford, Chris ( May 2021 , IEEE Sensors Journal)

   null (Ed.)

   Deaf spaces are unique indoor environments designed to optimize visual communication and Deaf cultural expression. However, much of the technological research geared towards the deaf involve use of video or wearables for American sign language (ASL) translation, with little consideration for Deaf perspective on privacy and usability of the technology. In contrast to video, RF sensors offer the avenue for ambient ASL recognition while also preserving privacy for Deaf signers. Methods: This paper investigates the RF transmit waveform parameters required for effective measurement of ASL signs and their effect on word-level classification accuracy attained with transfer learning and convolutional autoencoders (CAE). A multi-frequency fusion network is proposed to exploit data from all sensors in an RF sensor network and improve the recognition accuracy of fluent ASL signing. Results: For fluent signers, CAEs yield a 20-sign classification accuracy of %76 at 77 GHz and %73 at 24 GHz, while at X-band (10 Ghz) accuracy drops to 67%. For hearing imitation signers, signs are more separable, resulting in a 96% accuracy with CAEs. Further, fluent ASL recognition accuracy is significantly increased with use of the multi-frequency fusion network, which boosts the 20-sign fluent ASL recognition accuracy to 95%, surpassing conventional feature level fusion by 12%. Implications: Signing involves finer spatiotemporal dynamics than typical hand gestures, and thus requires interrogation with a transmit waveform that has a rapid succession of pulses and high bandwidth. Millimeter wave RF frequencies also yield greater accuracy due to the increased Doppler spread of the radar backscatter. Comparative analysis of articulation dynamics also shows that imitation signing is not representative of fluent signing, and not effective in pre-training networks for fluent ASL classification. Deep neural networks employing multi-frequency fusion capture both shared, as well as sensor-specific features and thus offer significant performance gains in comparison to using a single sensor or feature-level fusion. 

   more » « less
2. Radar Based Joint Human Activity and Agility Recognition via Multi Input Multi Task Learning

   https://doi.org/10.1109/RadarConf2458775.2024.10548013  

   Kurtoğlu, Emre ; Amin, Moeness G ; Gurbuz, Sevgi Z ( May 2024 , IEEE)

   Radar-based recognition of human activities of daily living has been a focus of research for over a decade. Current techniques focus on generalized motion recognition of any person and rely on massive amounts of data to characterize generic human activity. However, human gait is actually a person-specific biometric, correlated with health and agility, which depends on a person’s mobility ethogram. This paper proposes a multi-input multi-task deep learning framework for jointly learning a person’s agility and activity. As a proof of concept, we consider three categories of agility represented by slow, fast and nominal motion articulations and show that joint consideration of agility and activity can lead to improved activity classification accuracy and estimation of agility. To the best of our knowledge, this work represents the first work considering personalized motion recognition and agility characterization using radar. 

   more » « less
3. Physics-Aware Design of Multi-Branch GAN for Human RF Micro-Doppler Signature Synthesis

   https://doi.org/10.1109/RadarConf2147009.2021.9455194  

   Rahman, M. Mahbubur ; Gurbuz, Sevgi Z. ; Amin, Moeness G. ( May 2021 , IEEE Radar Conference)

   null (Ed.)

   Generative adversarial networks (GANs) have been recently proposed for the synthesis of RF micro-Doppler signatures to mitigate the problem of low sample support and enable the training of deeper neural networks (DNNs) for improved RF signal classification. However, when applied to human micro-Doppler signatures for gait analysis, GANs suffer from systemic kinematic discrepancies that degrade performance. As a solution to this problem, this paper proposes the design of a physics-aware loss function and multi-branch GAN architecture. Our results show that RF gait signatures synthesized using the proposed approached have greater correlation and similarity to measured RF gait signatures, while also improving the accuracy in classifying five different gaits. 

   more » « less
4. Multi-Frequency RF Sensor Data Adaptation for Motion Recognition with Multi-Modal Deep Learning

   https://doi.org/10.1109/RadarConf2147009.2021.9455204  

   Mahbubur Rahman, M. ; Gurbuz, Sevgi Z. ( May 2021 , IEEE Radar Conference)

   null (Ed.)

   The widespread availability of low-cost RF sensors has made it easier to construct RF sensor networks for motion recognition, as well as increased the availability of RF data across a variety of frequencies, waveforms, and transmit parameters. However, it is not effective to directly use disparate RF sensor data for the training of deep neural networks, as the phenomenological differences in the data result in significant performance degradation. In this paper, we consider two approaches for the exploitation of multi-frequency RF data: 1) a single sensor case, where adversarial domain adaptation is used to transform the data from one RF sensor to resemble that of another, and 2) a multi-sensor case, where a multi-modal neural network is designed for joint target recognition using measurements from all sensors. Our results show that the developed approaches offer effective techniques for leveraging multi-frequency RF sensor data for target recognition. 

   more » « less
5. RF sensing of personalized mobility: accounting for temporal variability in ethogram-based classification

   https://doi.org/10.1117/12.3013747  

   Kurtoglu, Emre ; Salehin, Sultanus ; Amin, Moeness G ; Gurbuz, Sevgi Zubeyde ( June 2024 , SPIE)

   Hedden, Abigail S ; Mazzaro, Gregory J (Ed.)

   Human activity recognition (HAR) with radar-based technologies has become a popular research area in the past decade. However, the objective of these studies are often to classify human activity for anyone; thus, models are trained using data spanning as broad a swath of people and mobility profiles as possible. In contrast, applications of HAR and gait analysis to remote health monitoring require characterization of the person-specific qualities of a person’s activities and gait, which greatly depends on age, health and agility. In fact, the speed or agility with which a person moves can be an important health indicator. In this study, we propose a multi-input multi-task deep learning framework to simultaneously learn a person’s activity and agility. In this initial study, we consider three different agility states: slow, nominal, and fast. It is shown that joint learning of agility and activity improves the classification accuracy for both activity and agility recognition tasks. To the best of our knowledge, this study is the first work considering both agility characterization and personalized activity recognition using RF sensing. 

   more » « less