skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Closed-Loop Tracking and Regulation of Emotional Valence State From Facial Electromyogram Measurements
Affective studies provide essential insights to address emotion recognition and tracking. In traditional open-loop structures, a lack of knowledge about the internal emotional state makes the system incapable of adjusting stimuli parameters and automatically responding to changes in the brain. To address this issue, we propose to use facial electromyogram measurements as biomarkers to infer the internal hidden brain state as feedback to close the loop. In this research, we develop a systematic way to track and control emotional valence, which codes emotions as being pleasant or obstructive. Hence, we conduct a simulation study by modeling and tracking the subject's emotional valence dynamics using state-space approaches. We employ Bayesian filtering to estimate the person-specific model parameters along with the hidden valence state, using continuous and binary features extracted from experimental electromyogram measurements. Moreover, we utilize a mixed-filter estimator to infer the secluded brain state in a real-time simulation environment. We close the loop with a fuzzy logic controller in two categories of regulation: inhibition and excitation. By designing a control action, we aim to automatically reflect any required adjustments within the simulation and reach the desired emotional state levels. Final results demonstrate that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, we envision future outcomes of this research to support alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare.  more » « less
Award ID(s):
1942585 1755780 2226123
PAR ID:
10322318
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Frontiers in Computational Neuroscience
Volume:
16
ISSN:
1662-5188
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Hypercortisolism or Cushing's disease, which corresponds to the excessive levels of cortisol hormone, is associated with tiredness and fatigue during the day and disturbed sleep at night. Our goal is to employ a wearable brain machine interface architecture to regulate one's energy levels in hypercortisolism. In the present simulation study, we generate multi-day cortisol profile data for ten subjects both in healthy and disease conditions. To relate an internal hidden cognitive energy state to one's cortisol secretion patterns, we employ a state-space model. Particularly, we consider circadian upper and lower bound envelopes on cortisol levels, and timings of hypothalamic pulsatile activity underlying cortisol secretions as continuous and binary observations, respectively. To estimate the hidden cognitive energy-related state, we use Bayesian filtering. In our proposed architecture, we infer one's cognitive energy-related state using wearable devices rather than monitoring the brain activity directly and close the loop utilizing fuzzy control. To model actuation in the real-time closed-loop architecture, we simulate two types of medications that result in increasing and decreasing the energy levels in the body. Finally, we close the loop using a knowledge-based control approach. The results on ten simulated profiles verify how the proposed architecture is able to track the energy state and regulate it using hypothetical medications. In a simulation study based on experimental data, we illustrate the feasibility of designing a wearable brain machine interface architecture for energy regulation in hypercortisolism. This simulation study is a first step toward the ultimate goal of managing hypercortisolism in real-world situations. 
    more » « less
  2. null (Ed.)
    Real-time adaptation is imperative to the control of robots operating in complex, dynamic environments. Adaptive control laws can endow even nonlinear systems with good trajectory tracking performance, provided that any uncertain dynamics terms are linearly parameterizable with known nonlinear features. However, it is often difficult to specify such features a priori, such as for aerodynamic disturbances on rotorcraft or interaction forces between a manipulator arm and various objects. In this paper, we turn to data-driven modeling with neural networks to learn, offline from past data, an adaptive controller with an internal parametric model of these nonlinear features. Our key insight is that we can better prepare the controller for deployment with control-oriented meta-learning of features in closed-loop simulation, rather than regression-oriented meta-learning of features to fit input-output data. Specifically, we meta-learn the adaptive controller with closed-loop tracking simulation as the base-learner and the average tracking error as the meta-objective. With a nonlinear planar rotorcraft subject to wind, we demonstrate that our adaptive controller outperforms other controllers trained with regression-oriented meta-learning when deployed in closed-loop for trajectory tracking control. 
    more » « less
  3. Our daily observations tell us that the delivery of social sentiments and emotions differs between strangers and romantic partners. This work explores how relationship status influences our delivery and perception of social touches and emotions, by evaluating the physics of contact interactions. In a study with human participants, strangers and romantically involved touchers delivered emotional messages to receivers’ forearms. Physical contact interactions were measured using a customized 3D tracking system. The results indicate that strangers and romantic receivers recognize emotional messages with similar accuracy, but with higher levels of valence and arousal between romantic partners. Further investigation into the contact interactions which underlie the higher levels of valence and arousal reveals that a toucher tunes their strategy with their romantic partner. For example, when stroking, romantic touchers use velocities preferential to C-tactile afferents, and maintain contact for longer durations with larger contact areas. Notwithstanding, while we show that relationship intimacy influences the deployment of touch strategies, such impact is relatively subtle compared to distinctions between gestures, emotional messages, and individual preferences. 
    more » « less
  4. Recognizing emotions using few attribute dimensions such as arousal, valence and dominance provides the flexibility to effectively represent complex range of emotional behaviors. Conventional methods to learn these emotional descriptors primarily focus on separate models to recognize each of these attributes. Recent work has shown that learning these attributes together regularizes the models, leading to better feature representations. This study explores new forms of regularization by adding unsupervised auxiliary tasks to reconstruct hidden layer representations. This auxiliary task requires the denoising of hidden representations at every layer of an auto-encoder. The framework relies on ladder networks that utilize skip connections between encoder and decoder layers to learn powerful representations of emotional dimensions. The results show that ladder networks improve the performance of the system compared to baselines that individually learn each attribute, and conventional denoising autoencoders. Furthermore, the unsupervised auxiliary tasks have promising potential to be used in a semi-supervised setting, where few labeled sentences are available. 
    more » « less
  5. Smiling and laughter are typically associated with amusement. If they occur under negative emotions, systems responding naively may confuse an uncomfortable smile or laugh with an amused state. We present a passive text and video elicitation task and collect spontaneous laughter and smiles in reaction to amusing and negative experiences, using standard, ubiquitous sensors (webcam and microphone), along with participant self-ratings. While we rely on a state-of-the-art smile recognizer, for laughter recognition our transfer learning architecture enhanced on modest data outperforms other models with up to 85% accuracy (F1 = 0.86), suggesting this technique as promising for improving affect models. Subsequently, we analyze and automatically predict laughter as amused vs. negative. However, contrasting with prior findings for acted data, for this spontaneously elicited dataset classifying laughter by emotional valence is not satisfactory. 
    more » « less