Affective studies provide essential insights to address emotion recognition and tracking. In traditional open-loop structures, a lack of knowledge about the internal emotional state makes the system incapable of adjusting stimuli parameters and automatically responding to changes in the brain. To address this issue, we propose to use facial electromyogram measurements as biomarkers to infer the internal hidden brain state as feedback to close the loop. In this research, we develop a systematic way to track and control emotional valence, which codes emotions as being pleasant or obstructive. Hence, we conduct a simulation study by modeling and tracking the subject's emotional valence dynamics using state-space approaches. We employ Bayesian filtering to estimate the person-specific model parameters along with the hidden valence state, using continuous and binary features extracted from experimental electromyogram measurements. Moreover, we utilize a mixed-filter estimator to infer the secluded brain state in a real-time simulation environment. We close the loop with a fuzzy logic controller in two categories of regulation: inhibition and excitation. By designing a control action, we aim to automatically reflect any required adjustments within the simulation and reach the desired emotional state levels. Final results demonstrate that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, we envision future outcomes of this research to support alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare.
more »
« less
Modeling EEG Dynamics of Self-Imagery Emotions: a Pilot Study
Electroencephalography (EEG)-based emotion classification has drawn increasing attention yet EEG signals associated with emotional responses are still elusive. This study applies a multi-model adaptive mixture independent component analysis (AMICA) as an unsupervised approach to identify and characterize emotional states. Empirical results showed that the AMICA was able to learn distinct models that accounted for four self-imagery emotions. While large-scale analyses and careful examinations are needed, the pilot study offers evidence for AMICA as a promising, data-driven approach to model EEG dynamics of self-imagery emotions.
more »
« less
- Award ID(s):
- 1719130
- NSF-PAR ID:
- 10107955
- Date Published:
- Journal Name:
- International IEEE/EMBS Conference on Neural Engineering
- ISSN:
- 1948-3554
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.
-
more » « less
Background As mobile health (mHealth) studies become increasingly productive owing to the advancements in wearable and mobile sensor technology, our ability to monitor and model human behavior will be constrained by participant receptivity. Many health constructs are dependent on subjective responses, and without such responses, researchers are left with little to no ground truth to accompany our ever-growing biobehavioral data. This issue can significantly impact the quality of a study, particularly for populations known to exhibit lower compliance rates. To address this challenge, researchers have proposed innovative approaches that use machine learning (ML) and sensor data to modify the timing and delivery of surveys. However, an overarching concern is the potential introduction of biases or unintended influences on participants’ responses when implementing new survey delivery methods.
Objective This study aims to demonstrate the potential impact of an ML-based ecological momentary assessment (EMA) delivery system (using receptivity as the predictor variable) on the participants’ reported emotional state. We examine the factors that affect participants’ receptivity to EMAs in a 10-day wearable and EMA–based emotional state–sensing mHealth study. We study the physiological relationships indicative of receptivity and affect while also analyzing the interaction between the 2 constructs.
Methods We collected data from 45 healthy participants wearing 2 devices measuring electrodermal activity, accelerometer, electrocardiography, and skin temperature while answering 10 EMAs daily, containing questions about perceived mood. Owing to the nature of our constructs, we can only obtain ground truth measures for both affect and receptivity during responses. Therefore, we used unsupervised and supervised ML methods to infer affect when a participant did not respond. Our unsupervised method used k-means clustering to determine the relationship between physiology and receptivity and then inferred the emotional state during nonresponses. For the supervised learning method, we primarily used random forest and neural networks to predict the affect of unlabeled data points as well as receptivity.
Results Our findings showed that using a receptivity model to trigger EMAs decreased the reported negative affect by >3 points or 0.29 SDs in our self-reported affect measure, scored between 13 and 91. The findings also showed a bimodal distribution of our predicted affect during nonresponses. This indicates that this system initiates EMAs more commonly during states of higher positive emotions.
Conclusions Our results showed a clear relationship between affect and receptivity. This relationship can affect the efficacy of an mHealth study, particularly those that use an ML algorithm to trigger EMAs. Therefore, we propose that future work should focus on a smart trigger that promotes EMA receptivity without influencing affect during sampled time points.
-
Background The physical and emotional well-being of women is critical for healthy pregnancy and birth outcomes. The Two Happy Hearts intervention is a personalized mind-body program coached by community health workers that includes monitoring and reflecting on personal health, as well as practicing stress management strategies such as mindful breathing and movement. Objective The aims of this study are to (1) test the daily use of a wearable device to objectively measure physical and emotional well-being along with subjective assessments during pregnancy, and (2) explore the user’s engagement with the Two Happy Hearts intervention prototype, as well as understand their experiences with various intervention components. Methods A case study with a mixed design was used. We recruited a 29-year-old woman at 33 weeks of gestation with a singleton pregnancy. She had no medical complications or physical restrictions, and she was enrolled in the Medi-Cal public health insurance plan. The participant engaged in the Two Happy Hearts intervention prototype from her third trimester until delivery. The Oura smart ring was used to continuously monitor objective physical and emotional states, such as resting heart rate, resting heart rate variability, sleep, and physical activity. In addition, the participant self-reported her physical and emotional health using the Two Happy Hearts mobile app–based 24-hour recall surveys (sleep quality and level of physical activity) and ecological momentary assessment (positive and negative emotions), as well as the Perceived Stress Scale, Center for Epidemiologic Studies Depression Scale, and State-Trait Anxiety Inventory. Engagement with the Two Happy Hearts intervention was recorded via both the smart ring and phone app, and user experiences were collected via Research Electronic Data Capture satisfaction surveys. Objective data from the Oura ring and subjective data on physical and emotional health were described. Regression plots and Pearson correlations between the objective and subjective data were presented, and content analysis was performed for the qualitative data. Results Decreased resting heart rate was significantly correlated with increased heart rate variability (r=–0.92, P<.001). We found significant associations between self-reported responses and Oura ring measures: (1) positive emotions and heart rate variability (r=0.54, P<.001), (2) sleep quality and sleep score (r=0.52, P<.001), and (3) physical activity and step count (r=0.77, P<.001). In addition, deep sleep appeared to increase as light and rapid eye movement sleep decreased. The psychological measures of stress, depression, and anxiety appeared to decrease from baseline to post intervention. Furthermore, the participant had a high completion rate of the components of the Two Happy Hearts intervention prototype and shared several positive experiences, such as an increased self-efficacy and a normal delivery. Conclusions The Two Happy Hearts intervention prototype shows promise for potential use by underserved pregnant women.more » « less
-
more » « less
Abstract People may experience emotions before interacting with automated agents to seek information and support. However, existing literature has not well examined how human emotional states affect their interaction experience with agents or how automated agents should react to emotions. This study proposes to test how participants perceive an empathetic agent (chatbot) vs. a non-empathetic one under various emotional states (i.e., positive, neutral, negative) when the chatbot mediates the initial screening process for student advising. Participants are prompted to recall a previous emotional experience and have text-based conversations with the chatbot. The study confirms the importance of presenting empathetic cues in the design of automated agents to support human-agent collaboration. Participants who recall a positive experience are more sensitive to the chatbot’s empathetic behavior. The empathetic behavior of the chatbot improves participants’ satisfaction and makes those who recall a neutral experience feel more positive during the interaction. The results reveal that participants’ emotional states are likely to influence their tendency to self-disclose, interaction experience, and perception of the chatbot’s empathetic behavior. The study also highlights the increasing need for emotional acknowledgment of people who experience positive emotions so that design efforts need to be designated according to people’s dynamic emotional states.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
