More Like this

1. Speech-Driven Expressive Talking Lips with Conditional Sequential Generative Adversarial Networks

   https://doi.org/10.1109/TAFFC.2019.2916031  

   Sadoughi, Najmeh; Busso, Carlos (May 2019, IEEE Transactions on Affective Computing)

   null (Ed.)

   Articulation, emotion, and personality play strong roles in the orofacial movements. To improve the naturalness and expressiveness of virtual agents(VAs), it is important that we carefully model the complex interplay between these factors. This paper proposes a conditional generative adversarial network, called conditional sequential GAN(CSG), which learns the relationship between emotion, lexical content and lip movements in a principled manner. This model uses a set of spectral and emotional speech features directly extracted from the speech signal as conditioning inputs, generating realistic movements. A key feature of the approach is that it is a speech-driven framework that does not require transcripts. Our experiments show the superiority of this model over three state-of-the-art baselines in terms of objective and subjective evaluations. When the target emotion is known, we propose to create emotionally dependent models by either adapting the base model with the target emotional data (CSG-Emo-Adapted), or adding emotional conditions as the input of the model(CSG-Emo-Aware). Objective evaluations of these models show improvements for the CSG-Emo-Adapted compared with the CSG model, as the trajectory sequences are closer to the original sequences. Subjective evaluations show significantly better results for this model compared with the CSG model when the target emotion is happiness. 

   more » « less
2. A Data-Driven Predictive Model of Individual-Specific Effects of FES on Human Gait Dynamics

   Drnach L., Allen (May 2019, International Conference on Robotics and Automation (ICRA))

   Modeling individual-specific gait dynamics based on kinematic data could aid development of gait rehabilitation robotics by enabling robots to predict the user’s gait kinematics with and without external inputs, such as mechanical or electrical perturbations. Here we address a current limitation of data driven gait models, which do not yet predict human responses to perturbations. We used Switched Linear Dynamical Systems (SLDS) to model joint angle kinematic data from healthy individuals walking on a treadmill during normal gait and during gait perturbed by functional electrical stimulation (FES) to the ankle muscles. Our SLDS models were able to predict the time-evolution of joint kinematics in each of four gait phases, as well as across an entire gait cycle. Because the SLDS dynamics matrices encoded significant coupling across joints, we compared the SLDS predictions to that of a kinematic model, where the joint angles were independent. Gait kinematics predicted by SLDS and kinematic models were similar over time horizons of a few milliseconds, but SLDS models provided better predictions of gait kinematics over time horizons of up to a second. We also demonstrated that SLDS models can infer and predict individual-specific responses to FES during swing phase. As such, SLDS models may be a promising approach for online estimation and control of and human gait dynamics, allowing robotic control strategies to be tailored to an individual’s specific gait coordination patterns. 

   more » « less
3. A Data-Driven Predictive Model of Individual-Specific Effects of FES on Human Gait Dynamics.

   Drnach, L.; Allen, J.L.; Ting, L.H. (January 2019, ICRA research monographs)

   Modeling individual-specific gait dynamics based on kinematic data could aid the development of gait rehabilitation robotics by enabling robots to predict the user’s gait kinematics with and without external inputs, such as mechanical or electrical perturbations. Here we address a current limitation of data-driven gait models, which do not yet predict human gait dynamics nor responses to perturbations. We used Switched Linear Dynamical Systems (SLDS) to model joint angle kinematic data from healthy individuals walking on a treadmill during normal gait and during gait perturbed by functional electrical stimulation (FES) to the ankle muscles. Our SLDS models were able to generate joint angle trajectories in each of four gait phases, as well as across an entire gait cycle, given initial conditions and gait phase information. Because the SLDS dynamics matrices encoded significant coupling across joints that differed across individuals, we compared the SLDS predictions to that of a kinematic model, where the joint angles were independent. Joint angle trajectories generated by SLDS and kinematic models were similar over time horizons of a few milliseconds, but SLDS models provided better predictions of gait kinematics over time horizons of up to a second. We also demonstrated that SLDS models can infer and predict individual-specific responses to FES during swing phase. As such, SLDS models may be a promising approach for online estimation and control of and human gait dynamics, allowing robotic control strategies to be tailored to an individual’s specific gait coordination patterns. 

   more » « less
4. Superposition principle applies to human walking with two simultaneous interventions

   https://doi.org/10.1038/s41598-021-86840-9  

   Rasouli, Fatemeh; Kim, Seok Hun; Reed, Kyle B. (December 2021, Scientific Reports)

   null (Ed.)

   Abstract Gait rehabilitation therapies provide adjusted sensory inputs to modify and retrain walking patterns in an impaired gait. Asymmetric walking is a common gait abnormality, especially among stroke survivors. Physical therapy interventions using adaptation techniques (such as treadmill training, auditory stimulation, visual biofeedback, etc.) train gait toward symmetry. However, a single rehabilitation therapy comes up short of affecting all aspects of gait performance. Multiple-rehabilitation therapy applies simultaneous stimuli to affect a wider range of gait parameters and create flexible training regiments. Understanding gait responses to individual and jointly applied stimuli is important for developing improved and efficient therapies. In this study, 16 healthy subjects participated in a four-session experiment to study gait kinetics and spatiotemporal outcomes under training. Each session consisted of two stimuli, treadmill training and auditory stimulation, with symmetric or asymmetric ratios between legs. The study hypothesizes a linear model for gait response patterns. We found that the superposition principle largely applies to the gait response under two simultaneous stimuli. The linear models developed in this study fit the actual data from experiments with the r-squared values of 0.95 or more. 

   more » « less
5. AugToAct: scaling complex human activity recognition with few labels

   https://doi.org/10.1145/3360774.3360831  

   Faridee, Abu Zaher; Khan, Md Abdullah; Pathak, Nilavra; Roy, Nirmalya (November 2019, 16th ACM/EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous), 2019)

   Human activity recognition (HAR) from wearable sensor data has recently gained widespread adoption in a number of fields. However, recognizing complex human activities, postural and rhythmic body movements (e.g., dance, sports) is challenging due to the lack of domain-specific labeling information, the perpetual variability in human movement kinematics profiles due to age, sex, dexterity and the level of professional training. In this paper, we propose a deep activity recognition model to work with limited labeled data, both for simple and complex human activities. To mitigate the intra- and inter-user spatio-temporal variability of movements, we posit novel data augmentation and domain normalization techniques. We depict a semi-supervised technique that learns noise and transformation invariant feature representation from sparsely labeled data to accommodate intra-personal and inter-user variations of human movement kinematics. We also postulate a transfer learning approach to learn domain invariant feature representations by minimizing the feature distribution distance between the source and target domains. We showcase the improved performance of our proposed framework, AugToAct, using a public HAR dataset. We also design our own data collection, annotation and experimental setup on complex dance activity recognition steps and kinematics movements where we achieved higher performance metrics with limited label data compared to simple activity recognition tasks. 

   more » « less