More Like this

1. Transfer Learning on Small Datasets for Improved Fall Detection

   https://doi.org/10.3390/s23031105  

   Maray, Nader; Ngu, Anne Hee; Ni, Jianyuan; Debnath, Minakshi; Wang, Lu (February 2023, Sensors)

   Falls in the elderly are associated with significant morbidity and mortality. While numerous fall detection devices incorporating AI and machine learning algorithms have been developed, no known smartwatch-based system has been used successfully in real-time to detect falls for elderly persons. We have developed and deployed a SmartFall system on a commodity-based smartwatch which has been trialled by nine elderly participants. The system, while being usable and welcomed by the participants in our trials, has two serious limitations. The first limitation is the inability to collect a large amount of personalized data for training. When the fall detection model, which is trained with insufficient data, is used in the real world, it generates a large amount of false positives. The second limitation is the model drift problem. This means an accurate model trained using data collected with a specific device performs sub-par when used in another device. Therefore, building one model for each type of device/watch is not a scalable approach for developing smartwatch-based fall detection system. To tackle those issues, we first collected three datasets including accelerometer data for fall detection problem from different devices: the Microsoft watch (MSBAND), the Huawei watch, and the meta-sensor device. After that, a transfer learning strategy was applied to first explore the use of transfer learning to overcome the small dataset training problem for fall detection. We also demonstrated the use of transfer learning to generalize the model across the heterogeneous devices. Our preliminary experiments demonstrate the effectiveness of transfer learning for improving fall detection, achieving an F1 score higher by over 10% on average, an AUC higher by over 0.15 on average, and a smaller false positive prediction rate than the non-transfer learning approach across various datasets collected using different devices with different hardware specifications. 

   more » « less
2. SoundWatch: deep learning for sound accessibility on smartwatches

   https://doi.org/10.1145/3531447  

   Jain, Dhruv; Ngo, Hung; Patel, Pratyush; Goodman, Steven; Nguyen, Khoa; Grossman-Kahn, Rachel; Findlater, Leah; Froehlich, Jon (May 2022, Communications of the ACM)

   Smartwatches have the potential to provide glanceable, always-available sound feedback to people who are deaf or hard of hearing (DHH). We present SoundWatch, a smartwatch-based deep learning application to sense, classify, and provide feedback about sounds occurring in the environment. To design SoundWatch, we first examined four low-resource sound classification models across four device architectures: watch-only, watch+phone, watch+phone+cloud, and watch+cloud. We found that the best model, VGG-lite, performed similar to the state of the art for nonportable devices although requiring substantially less memory (∼1/3^rd) and that the watch+phone architecture provided the best balance among CPU, memory, network usage, and latency. Based on these results, we built and conducted a lab evaluation of our smartwatch app with eight DHH participants. We found support for our sound classification app but also uncovered concerns with misclassifications, latency, and privacy. 

   more » « less
3. Collaborative Fall Detection using a Wearable Device and a Companion Robot

   Liang, Fei; Hernandez, Ricardo; Lu, Jiaxing; Moore, Jackson; Sheng, Weihua; Zhang, Senlin (June 2021, IEEE International Conference on Robotics and Automation)

   null (Ed.)

   Older adults who age in place face many health problems and need to be taken care of. Fall is a serious problem among elderly people. In this paper, we present the design and implementation of collaborative fall detection using a wearable device and a companion robot. First, we developed a wearable device by integrating a camera, an accelerometer and a microphone. Second, a companion robot communicates with the wearable device to conduct collaborative fall detection. The robot is also able to contact caregivers in case of emergency. The collaborative fall detection method consists of motion data based preliminary detection on the wearable device and video-based final detection on the companion robot. Both convolutional neural network (CNN) and long short-term memory (LSTM) are used for video-based fall detection. The experimental results show that the overall accuracy of video-based algorithm is 84%. We also investigated the relation between the accuracy and the number of image frames. Our method improves the accuracy of fall detection while maximizing the battery life of the wearable device. In addition, our method significantly increases the sensing range of the companion robot. 

   more » « less
4. Personalized Fall Detection System

   Ngu, Anne H; Metsis, Vangelis; Coyne, Shaun; Chung, Brian; Pai, Rachel; Chang, Joshua (January 2020, 5th IEEE PerCom Workshop on Pervasive Health Technologies)

   This paper explores the personalization of smartwatch-based fall detection models trained using a combination of deep neural networks with ensemble techniques. Deep neural networks face practical challenges when used for fall detection, which in general tend to have limited training samples and imbalanced datasets. Moreover, many motions generated by a wrist-worn watch can be mistaken for a fall. Obtaining a large amount of real-world labeled fall data is impossible as fall is a rare event. However, it is easy to collect a large number of non-fall data samples from users. In this paper, we aim to mitigate the scarcity of training data in fall detection by first training a generic deep learning ensemble model, optimized for high recall, and then enhancing the precision of the model, by collecting personalized false positive samples from individual users, via feedback from the SmartFall App. We performed real-world experiments with five volunteers and concluded that a personalized fall detection model significantly outperforms generic fall detection models, especially in terms of precision. We further validated the performance of personalization by using a new metric for evaluating the accuracy of the model via normalizing false positive rates with regard to the number of spikes of acceleration over time. 

   more » « less
5. Experimental Study of Long Short-Term Memory and Transformer Models for Fall Detection on Smartwatches

   https://doi.org/10.3390/s24196235  

   Haque, Syed Tousiful; Debnath, Minakshi; Yasmin, Awatif; Mahmud, Tarek; Ngu, Anne_Hee Hiong (October 2024, Sensors)

   Falls are the second leading cause of unintentional injury deaths worldwide. While numerous wearable fall detection devices incorporating AI models have been developed, none of them are used successfully in a fall detection application running on commodity-based smartwatches in real time. The system misses some falls, and generates an annoying amount of False Positives for practical use. We have investigated and experimented with an LSTM model for fall detection on a smartwatch. Even though the LSTM model has high accuracy during offline testing, the good performance of offline LSTM models cannot be translated to the equivalence of real-time performance. Transformers, on the other hand, can learn long-sequence data and patterns intrinsic to the data due to their self-attention mechanism. This paper compares three variants of LSTM and two variants of Transformer models for learning fall patterns. We trained all models using fall and activity data from three datasets, and the real-time testing of the model was performed using the SmartFall App. Our findings showed that in the offline training, the CNN-LSTM model was better than the Transformer model for all the datasets. However, the Transformer is a preferable choice for deployment in real-time fall detection applications. 

   more » « less