Search for: All records

Recent advances in robotics have accelerated their widespread use in nontraditional domains such as law enforcement. The inclusion of robotics allows for the introduction of time and space in dangerous situations, and protects law enforcement officers (LEOs) from the many potentially dangerous situations they encounter. In this paper, a teleoperated robot prototype was designed and tested to allow LEOs to remotely and transparently communicate and interact with others. The robot featured near face-to-face interactivity and accuracy across multiple verbal and non-verbal modes using screens, microphones, and speakers. In cooperation with multiple law enforcement agencies, results are presented on this dynamic and integrative teleoperated communicative robot platform in terms of attitudes towards robots, trust in robot operation, and trust in human-robot-human interaction and communication. 

Deaf spaces are unique indoor environments designed to optimize visual communication and Deaf cultural expression. However, much of the technological research geared towards the deaf involve use of video or wearables for American sign language (ASL) translation, with little consideration for Deaf perspective on privacy and usability of the technology. In contrast to video, RF sensors offer the avenue for ambient ASL recognition while also preserving privacy for Deaf signers. Methods: This paper investigates the RF transmit waveform parameters required for effective measurement of ASL signs and their effect on word-level classification accuracy attained with transfer learning and convolutional autoencoders (CAE). A multi-frequency fusion network is proposed to exploit data from all sensors in an RF sensor network and improve the recognition accuracy of fluent ASL signing. Results: For fluent signers, CAEs yield a 20-sign classification accuracy of %76 at 77 GHz and %73 at 24 GHz, while at X-band (10 Ghz) accuracy drops to 67%. For hearing imitation signers, signs are more separable, resulting in a 96% accuracy with CAEs. Further, fluent ASL recognition accuracy is significantly increased with use of the multi-frequency fusion network, which boosts the 20-sign fluent ASL recognition accuracy to 95%, surpassing conventional feature level fusion by 12%. Implications: Signing involves finer spatiotemporal dynamics than typical hand gestures, and thus requires interrogation with a transmit waveform that has a rapid succession of pulses and high bandwidth. Millimeter wave RF frequencies also yield greater accuracy due to the increased Doppler spread of the radar backscatter. Comparative analysis of articulation dynamics also shows that imitation signing is not representative of fluent signing, and not effective in pre-training networks for fluent ASL classification. Deep neural networks employing multi-frequency fusion capture both shared, as well as sensor-specific features and thus offer significant performance gains in comparison to using a single sensor or feature-level fusion. 

Many technologies for human-computer interaction have been designed for hearing individuals and depend upon vocalized speech, precluding users of American Sign Language (ASL) in the Deaf community from benefiting from these advancements. While great strides have been made in ASL recognition with video or wearable gloves, the use of video in homes has raised privacy concerns, while wearable gloves severely restrict movement and infringe on daily life. Methods: This paper proposes the use of RF sensors for HCI applications serving the Deaf community. A multi-frequency RF sensor network is used to acquire non-invasive, non-contact measurements of ASL signing irrespective of lighting conditions. The unique patterns of motion present in the RF data due to the micro-Doppler effect are revealed using time-frequency analysis with the Short-Time Fourier Transform. Linguistic properties of RF ASL data are investigated using machine learning (ML). Results: The information content, measured by fractal complexity, of ASL signing is shown to be greater than that of other upper body activities encountered in daily living. This can be used to differentiate daily activities from signing, while features from RF data show that imitation signing by non-signers is 99% differentiable from native ASL signing. Feature-level fusion of RF sensor network data is used to achieve 72.5% accuracy in classification of 20 native ASL signs. Implications: RF sensing can be used to study dynamic linguistic properties of ASL and design Deaf-centric smart environments for non-invasive, remote recognition of ASL. ML algorithms should be benchmarked on native, not imitation, ASL data. 

Although users of American Sign Language (ASL) comprise a significant minority in the U.S. and Canada, people in the Deaf community have been unable to benefit from many new technologies, which depend upon vocalized speech, and are designed for hearing individuals. While video has led to tremendous advances in ASL recognition, concerns over invasion of privacy have limited its use for in-home smart environments. This work presents initial work on the use of RF sensors, which can protect user privacy, for the purpose of ASL recognition. The new offerings of 2D/3D RF data representations and optical flow are presented. The fractal complexity of ASL is shown to be greater than that of daily activities - a relationship consistent with linguistic analysis conducted using video.