Inferring emotions from others’ non-verbal behavior is a pervasive and fundamental task in social interactions. Typically, real-life encounters imply the co-location of interactants, i.e., their embodiment within a shared spatial-temporal continuum in which the trajectories of the interaction partner’s Expressive Body Movement (EBM) create mutual social affordances. Shared Virtual Environments (SVEs) and Virtual Characters (VCs) are increasingly used to study social perception, allowing to reconcile experimental stimulus control with ecological validity. However, it remains unclear whether display modalities that enable co-presence have an impact on observers responses to VCs’ expressive behaviors. Drawing upon ecological approaches to social perception, we reasoned that sharing the space with a VC should amplify affordances as compared to a screen display, and consequently alter observers’ perceptions of EBM in terms of judgment certainty, hit rates, perceived expressive qualities (arousal and valence), and resulting approach and avoidance tendencies. In a between-subject design, we compared the perception of 54 10-s animations of VCs performing three daily activities (painting, mopping, sanding) in three emotional states (angry, happy, sad)—either displayed in 3D as a co-located VC moving in shared space, or as a 2D replay on a screen that was also placed in the SVEs. Results confirm the effective experimental control of the variable of interest, showing that perceived co-presence was significantly affected by the display modality, while perceived realism and immersion showed no difference. Spatial presence and social presence showed marginal effects. Results suggest that the display modality had a minimal effect on emotion perception. A weak effect was found for the expression “happy,” for which unbiased hit rates were higher in the 3D condition. Importantly, low hit rates were observed for all three emotion categories. However, observers judgments significantly correlated for category assignment and across all rating dimensions, indicating universal decoding principles. While category assignment was erroneous, though, ratings of valence and arousal were consistent with expectations derived from emotion theory. The study demonstrates the value of animated VCs in emotion perception studies and raises new questions regarding the validity of category-based emotion recognition measures.
This content will become publicly available on July 1, 2024
- Award ID(s):
- 1908115
- NSF-PAR ID:
- 10475033
- Publisher / Repository:
- NSF-PAR
- Date Published:
- Journal Name:
- IEEE Transactions on Haptics
- Volume:
- 16
- Issue:
- 3
- ISSN:
- 1939-1412
- Page Range / eLocation ID:
- 449 to 454
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
-
Touch as a modality in social communication has been getting more attention with recent developments in wearable technology and an increase in awareness of how limited physical contact can lead to touch starvation and feelings of depression. Although several mediated touch methods have been developed for conveying emotional support, the transfer of emotion through mediated touch has not been widely studied. This work addresses this need by exploring emotional communication through a novel wearable haptic system. The system records physical touch patterns through an array of force sensors, processes the recordings using novel gesture-based algorithms to create actuator control signals, and generates mediated social touch through an array of voice coil actuators. We conducted a human subject study ( N = 20) to understand the perception and emotional components of this mediated social touch for common social touch gestures, including poking, patting, massaging, squeezing, and stroking. Our results show that the speed of the virtual gesture significantly alters the participants' ratings of valence, arousal, realism, and comfort of these gestures with increased speed producing negative emotions and decreased realism. The findings from the study will allow us to better recognize generic patterns from human mediated touch perception and determine how mediated social touch can be used to convey emotion. Our system design, signal processing methods, and results can provide guidance in future mediated social touch design.more » « less
-
more » « less
Background As mobile health (mHealth) studies become increasingly productive owing to the advancements in wearable and mobile sensor technology, our ability to monitor and model human behavior will be constrained by participant receptivity. Many health constructs are dependent on subjective responses, and without such responses, researchers are left with little to no ground truth to accompany our ever-growing biobehavioral data. This issue can significantly impact the quality of a study, particularly for populations known to exhibit lower compliance rates. To address this challenge, researchers have proposed innovative approaches that use machine learning (ML) and sensor data to modify the timing and delivery of surveys. However, an overarching concern is the potential introduction of biases or unintended influences on participants’ responses when implementing new survey delivery methods.
Objective This study aims to demonstrate the potential impact of an ML-based ecological momentary assessment (EMA) delivery system (using receptivity as the predictor variable) on the participants’ reported emotional state. We examine the factors that affect participants’ receptivity to EMAs in a 10-day wearable and EMA–based emotional state–sensing mHealth study. We study the physiological relationships indicative of receptivity and affect while also analyzing the interaction between the 2 constructs.
Methods We collected data from 45 healthy participants wearing 2 devices measuring electrodermal activity, accelerometer, electrocardiography, and skin temperature while answering 10 EMAs daily, containing questions about perceived mood. Owing to the nature of our constructs, we can only obtain ground truth measures for both affect and receptivity during responses. Therefore, we used unsupervised and supervised ML methods to infer affect when a participant did not respond. Our unsupervised method used k-means clustering to determine the relationship between physiology and receptivity and then inferred the emotional state during nonresponses. For the supervised learning method, we primarily used random forest and neural networks to predict the affect of unlabeled data points as well as receptivity.
Results Our findings showed that using a receptivity model to trigger EMAs decreased the reported negative affect by >3 points or 0.29 SDs in our self-reported affect measure, scored between 13 and 91. The findings also showed a bimodal distribution of our predicted affect during nonresponses. This indicates that this system initiates EMAs more commonly during states of higher positive emotions.
Conclusions Our results showed a clear relationship between affect and receptivity. This relationship can affect the efficacy of an mHealth study, particularly those that use an ML algorithm to trigger EMAs. Therefore, we propose that future work should focus on a smart trigger that promotes EMA receptivity without influencing affect during sampled time points.
-
more » « less
Respiratory sinus arrhythmia (RSA), an index of the parasympathetic nervous system, has recently gained attention as a physiological component of regulatory processes, social connectedness, and health. Within the context of romantic relationships, studies have operationalized and conceptualized RSA in disparate ways, obscuring a clear pattern of findings. This systematic review synthesizes the rapidly developing literature and clarifies the role of RSA in romantic relationships. We evaluate support for three conceptual hypotheses: (1) resting baseline RSA is associated with better quality relationships; (2) phasic RSA is reflective of changes in threat and connection during couple interactions; and (3) physiological linkage in RSA between romantic partners relates to positive or negative relationship functioning depending on the nature of the linkage (e.g., in‐phase vs. antiphase). We identified 26 empirical studies that tested associations between RSA and an index of romantic relationships (i.e., relationship satisfaction). Our findings show that higher RSA is not uniformly “good” for relationships. Higher resting baseline RSA was contemporaneously associated with better quality relationships, yet higher baseline RSA was also unexpectedly associated with relationship violence. Short‐term decreases in RSA were found during relationship conflict, though the opposite—phasic increases in RSA during positive romantic partner interactions—was not found due to mixed empirical support. As expected, evidence for RSA linkage was found, though the connection between linkage and relationship functioning depends on the context in which it was measured. We discuss methodological limitations and directions for future research.
-
null (Ed.)Considerable amount of laboratory and survey‐based research finds that people show disproportional compassionate and affective response to the scope of human mortality risk. According to research on “psychic numbing,” it is often the case that the more who die, the less we care. In the present article, we examine the extent of this phenomenon in verbal behavior, using large corpora of natural language to quantify the affective reactions to loss of life. We analyze valence, arousal, and specific emotional content of over 100,000 mentions of death in news articles and social media posts, and find that language shows an increase in valence (i.e., decreased negative affect) and a decrease in arousal when describing mortality of larger numbers of people. These patterns are most clearly reflected in specific emotions of joy and (in a reverse fashion) of fear and anger. Our results showcase a novel methodology for studying affective decision making, and highlight the robustness and real‐world relevance of psychic numbing. They also offer new insights regarding the psychological underpinnings of psychic numbing, as well as possible interventions for reducing psychic numbing and overcoming social and psychological barriers to action in the face of the world's most serious threats.more » « less
