More Like this

1. Near real-time ASL recognition using a millimeter wave radar

   https://doi.org/10.1117/12.2588616  

   Adeoluwa, Oladipupo ; Kearney, Sean J. ; Kurtoglu, Emre ; Connors, Charles ; Gurbuz, Sevgi Zubeyde ( April 2021 , Proceedings of SPIE (International Society of Optics and Photonics))

   Raynal, Ann M. ; Ranney, Kenneth I. (Ed.)

   Most research in technologies for the Deaf community have focused on translation using either video or wearable devices. Sensor-augmented gloves have been reported to yield higher gesture recognition rates than camera-based systems; however, they cannot capture information expressed through head and body movement. Gloves are also intrusive and inhibit users in their pursuit of normal daily life, while cameras can raise concerns over privacy and are ineffective in the dark. In contrast, RF sensors are non-contact, non-invasive and do not reveal private information even if hacked. Although RF sensors are unable to measure facial expressions or hand shapes, which would be required for complete translation, this paper aims to exploit near real-time ASL recognition using RF sensors for the design of smart Deaf spaces. In this way, we hope to enable the Deaf community to benefit from advances in technologies that could generate tangible improvements in their quality of life. More specifically, this paper investigates near real-time implementation of machine learning and deep learning architectures for the purpose of sequential ASL signing recognition. We utilize a 60 GHz RF sensor which transmits a frequency modulation continuous wave (FMWC waveform). RF sensors can acquire a unique source of information that is inaccessible to optical or wearable devices: namely, a visual representation of the kinematic patterns of motion via the micro-Doppler signature. Micro-Doppler refers to frequency modulations that appear about the central Doppler shift, which are caused by rotational or vibrational motions that deviate from principle translational motion. In prior work, we showed that fractal complexity computed from RF data could be used to discriminate signing from daily activities and that RF data could reveal linguistic properties, such as coarticulation. We have also shown that machine learning can be used to discriminate with 99% accuracy the signing of native Deaf ASL users from that of copysigning (or imitation signing) by hearing individuals. Therefore, imitation signing data is not effective for directly training deep models. But, adversarial learning can be used to transform imitation signing to resemble native signing, or, alternatively, physics-aware generative models can be used to synthesize ASL micro-Doppler signatures for training deep neural networks. With such approaches, we have achieved over 90% recognition accuracy of 20 ASL signs. In natural environments, however, near real-time implementations of classification algorithms are required, as well as an ability to process data streams in a continuous and sequential fashion. In this work, we focus on extensions of our prior work towards this aim, and compare the efficacy of various approaches for embedding deep neural networks (DNNs) on platforms such as a Raspberry Pi or Jetson board. We examine methods for optimizing the size and computational complexity of DNNs for embedded micro-Doppler analysis, methods for network compression, and their resulting sequential ASL recognition performance. 

   more » « less
2. Word-Level ASL Recognition and Trigger Sign Detection with RF Sensors

   https://doi.org/10.1109/ICASSP39728.2021.9414063  

   Rahman, M. Mahbubur ; Kurtoglu, Emre ; Mdrafi, Robiulhossain ; Gurbuz, Ali C. ; Malaia, Evie ; Crawford, Chris ; Griffin, Darrin ; Gurbuz, Sevgi Z. ( June 2021 , IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)))

   null (Ed.)

   Current research in the recognition of American Sign Language (ASL) has focused on perception using video or wearable gloves. However, deaf ASL users have expressed concern about the invasion of privacy with video, as well as the interference with daily activity and restrictions on movement presented by wearable gloves. In contrast, RF sensors can mitigate these issues as it is a non-contact ambient sensor that is effective in the dark and can penetrate clothes, while only recording speed and distance. Thus, this paper investigates RF sensing as an alternative sensing modality for ASL recognition to facilitate interactive devices and smart environments for the deaf and hard-of-hearing. In particular, the recognition of up to 20 ASL signs, sequential classification of signing mixed with daily activity, and detection of a trigger sign to initiate human-computer interaction (HCI) via RF sensors is presented. Results yield %91.3 ASL word-level classification accuracy, %92.3 sequential recognition accuracy, 0.93 trigger recognition rate. 

   more » « less
3. Multi-Frequency RF Sensor Fusion for Word-Level Fluent ASL Recognition

   https://doi.org/10.1109/JSEN.2021.3078339  

   Gurbuz, Sevgi Z. ; Mahbubur Rahman, M. ; Kurtoglu, Emre ; Malaia, Evie ; Gurbuz, Ali C. ; Griffin, Darrin J. ; Crawford, Chris ( May 2021 , IEEE Sensors Journal)

   null (Ed.)

   Deaf spaces are unique indoor environments designed to optimize visual communication and Deaf cultural expression. However, much of the technological research geared towards the deaf involve use of video or wearables for American sign language (ASL) translation, with little consideration for Deaf perspective on privacy and usability of the technology. In contrast to video, RF sensors offer the avenue for ambient ASL recognition while also preserving privacy for Deaf signers. Methods: This paper investigates the RF transmit waveform parameters required for effective measurement of ASL signs and their effect on word-level classification accuracy attained with transfer learning and convolutional autoencoders (CAE). A multi-frequency fusion network is proposed to exploit data from all sensors in an RF sensor network and improve the recognition accuracy of fluent ASL signing. Results: For fluent signers, CAEs yield a 20-sign classification accuracy of %76 at 77 GHz and %73 at 24 GHz, while at X-band (10 Ghz) accuracy drops to 67%. For hearing imitation signers, signs are more separable, resulting in a 96% accuracy with CAEs. Further, fluent ASL recognition accuracy is significantly increased with use of the multi-frequency fusion network, which boosts the 20-sign fluent ASL recognition accuracy to 95%, surpassing conventional feature level fusion by 12%. Implications: Signing involves finer spatiotemporal dynamics than typical hand gestures, and thus requires interrogation with a transmit waveform that has a rapid succession of pulses and high bandwidth. Millimeter wave RF frequencies also yield greater accuracy due to the increased Doppler spread of the radar backscatter. Comparative analysis of articulation dynamics also shows that imitation signing is not representative of fluent signing, and not effective in pre-training networks for fluent ASL classification. Deep neural networks employing multi-frequency fusion capture both shared, as well as sensor-specific features and thus offer significant performance gains in comparison to using a single sensor or feature-level fusion. 

   more » « less
4. Word-level Sign Language Recognition Using Linguistic Adaptation of 77 GHz FMCW Radar Data

   https://doi.org/10.1109/RadarConf2147009.2021.9455190  

   Rahman, M. Mahbubur ; Mdrafi, Robiulhossain ; Gurbuz, Ali C. ; Malaia, Evie ; Crawford, Chris ; Griffin, Darrin ; Gurbuz, Sevgi Z. ( May 2021 , IEEE Radar Conference)

   null (Ed.)

   Over the years, there has been much research in both wearable and video-based American Sign Language (ASL) recognition systems. However, the restrictive and invasive nature of these sensing modalities remains a significant disadvantage in the context of Deaf-centric smart environments or devices that are responsive to ASL. This paper investigates the efficacy of RF sensors for word-level ASL recognition in support of human-computer interfaces designed for deaf or hard-of-hearing individuals. A principal challenge is the training of deep neural networks given the difficulty in acquiring native ASL signing data. In this paper, adversarial domain adaptation is exploited to bridge the physical/kinematic differences between the copysigning of hearing individuals (repetition of sign motion after viewing a video), and native signing of Deaf individuals who are fluent in sign language. Domain adaptation results are compared with those attained by directly synthesizing ASL signs using generative adversarial networks (GANs). Kinematic improvements to the GAN architecture, such as the insertion of micro-Doppler signature envelopes in a secondary branch of the GAN, are utilized to boost performance. Word-level classification accuracy of 91.3% is achieved for 20 ASL words. 

   more » « less
5. A Linguistic Perspective on Radar Micro-Doppler Analysis of American Sign Language

   https://doi.org/10.1109/RADAR42522.2020.9114818  

   Gurbuz, Sevgi Z. ; Gurbuz, Ali C. ; Malaia, Evie A. ; Griffin, Darrin J. ; Crawford, Chris ; Rahman, M. Mahbubur ; Aksu, Ridvan ; Kurtoglu, Emre ; Mdrafi, Robiulhossain ; Anbuselvam, Ajaymehul ; et al ( April 2020 , 2020 IEEE International Radar Conference (RADAR))

   Although users of American Sign Language (ASL) comprise a significant minority in the U.S. and Canada, people in the Deaf community have been unable to benefit from many new technologies, which depend upon vocalized speech, and are designed for hearing individuals. While video has led to tremendous advances in ASL recognition, concerns over invasion of privacy have limited its use for in-home smart environments. This work presents initial work on the use of RF sensors, which can protect user privacy, for the purpose of ASL recognition. The new offerings of 2D/3D RF data representations and optical flow are presented. The fractal complexity of ASL is shown to be greater than that of daily activities - a relationship consistent with linguistic analysis conducted using video. 

   more » « less