In this paper, we propose a machine learning-based multi-stream framework to recognize American Sign Language (ASL) manual signs and nonmanual gestures (face and head movements) in real time from RGB-D videos. Our approach is based on 3D Convolutional Neural Networks (3D CNNs) by fusing the multi-modal features including hand gestures, facial expressions, and body poses from multiple channels (RGB, Depth, Motion, and Skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3D CNN model. We collected a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera. Each video consists of 100 ASL manual signs, along with RGB channel, Depth maps, Skeleton joints, Face features, and HD face. The dataset is fully annotated for each semantic region (i.e. the time duration of each sign that the human signer performs). Our proposed method achieves 92.88% accuracy for recognizing 100 ASL sign glosses in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on a large-scale dataset, ChaLearn IsoGD, achieving the state-of-the-art results.
- Home
- Search Results
- Page 1 of 1
Search for: All records
-
Total Resources4
- Resource Type
-
01000030000
- More
- Availability
-
31
- Author / Contributor
- Filter by Author / Creator
-
-
Jing, Longlong (4)
-
Tian, Yingli (3)
-
Akin, Oguz (1)
-
He, Ju (1)
-
Huenerfauth, Matt (1)
-
Huo, Yumei (1)
-
Li, Lihong (1)
-
Li, Yingwei (1)
-
She, Qi (1)
-
Tan, Jiaxing (1)
-
Vahdani, Elahe (1)
-
Xiao, Junfei (1)
-
Yuille, Alan (1)
-
Zhang, Lin (1)
-
Zhou, Zongwei (1)
-
#Tyler Phillips, Kenneth E. (0)
-
#Willis, Ciara (0)
-
& Abreu-Ramos, E. D. (0)
-
& Abramson, C. I. (0)
-
& Abreu-Ramos, E. D. (0)
-
- Filter by Editor
-
-
& Spizer, S. M. (0)
-
& . Spizer, S. (0)
-
& Ahn, J. (0)
-
& Bateiha, S. (0)
-
& Bosch, N. (0)
-
& Brennan K. (0)
-
& Brennan, K. (0)
-
& Chen, B. (0)
-
& Chen, Bodong (0)
-
& Drown, S. (0)
-
& Ferretti, F. (0)
-
& Higgins, A. (0)
-
& J. Peters (0)
-
& Kali, Y. (0)
-
& Ruiz-Arias, P.M. (0)
-
& S. Spitzer (0)
-
& Sahin. I. (0)
-
& Spitzer, S. (0)
-
& Spitzer, S.M. (0)
-
(submitted - in Review for IEEE ICASSP-2024) (0)
-
-
Have feedback or suggestions for a way to improve these results?
!
Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher.
Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?
Some links on this page may take you to non-federal websites. Their policies may differ from this site.
-
Free, publicly-accessible full text available March 1, 2025
-
Xiao, Junfei ; Jing, Longlong ; Zhang, Lin ; He, Ju ; She, Qi ; Zhou, Zongwei ; Yuille, Alan ; Li, Yingwei ( , 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR))
-
Jing, Longlong ; Tian, Yingli ( , IEEE Transactions on Pattern Analysis and Machine Intelligence)
-
Tan, Jiaxing ; Jing, Longlong ; Huo, Yumei ; Li, Lihong ; Akin, Oguz ; Tian, Yingli ( , Computerized Medical Imaging and Graphics)