More Like this

1. Impact of Interaction Context on the Student Affect-Learning Relationship in Child-Robot Interaction

   https://doi.org/10.1145/3319502.3374822  

   Chen, Huili; Park, Hae Won; Zhang, Xiajie; Breazeal, Cynthia (March 2020, ACM/IEEE International Conference on Human-Robot Interaction)

   Prior work in affect-aware educational robots has often relied on a common belief that the relationship between student affect and learning is independent of agent behaviors (child’s/robot’s) or unidirectional (positive/negative but not both) throughout the entire student-robot interaction.We argue that the student affect-learning relationship should be interpreted in two contexts: (1) social learning paradigm and (2) sub-events within child-robot interaction. In our paper, we examine two different social learning paradigms where children interact with a robot that acts either as a tutor or a tutee. Sub-events within child-robot interaction are defined as task-related events occurring in specific phases of an interaction (e.g., when the child/robot gets a wrong answer). We examine subevents at a macro level (entire interaction) and a micro level (within specific sub-events). In this paper, we provide an in-depth correlation analysis of children’s facial affect and vocabulary learning. We found that children’s affective displays became more predictive of their vocabulary learning when children interacted with a tutee robot who did not scaffold their learning. Additionally, children’s affect displayed during micro-level events was more predictive of their learning than during macro-level events. Last, we found that the affect-learning relationship is not unidirectional, but rather is modulated by context, i.e., several affective states facilitated student learning when displayed in some sub-events but inhibited learning when displayed in others. These findings indicate that both social learning paradigm and sub-events within interaction modulate student affect-learning relationship. 

   more » « less
2. Evaluating Temporal Patterns in Applied Infant Affect Recognition. 10th International Conference on Affective Computing and Intelligent Interaction

   Chang*, Allen; Klein*, Lauren; Rosales, Marcelo R.; Deng, Weiyang; Smith, Beth A.; Matarić, Maja (October 2022, 10th International Conference on Affective Computing and Intelligent Interaction)

   Agents must monitor their partners' affective states continuously in order to understand and engage in social interactions. However, methods for evaluating affect recognition do not account for changes in classification performance that may occur during occlusions or transitions between affective states. This paper addresses temporal patterns in affect classification performance in the context of an infant-robot interaction, where infants’ affective states contribute to their ability to participate in a therapeutic leg movement activity. To support robustness to facial occlusions in video recordings, we trained infant affect recognition classifiers using both facial and body features. Next, we conducted an in-depth analysis of our best-performing models to evaluate how performance changed over time as the models encountered missing data and changing infant affect. During time windows when features were extracted with high confidence, a unimodal model trained on facial features achieved the same optimal performance as multimodal models trained on both facial and body features. However, multimodal models outperformed unimodal models when evaluated on the entire dataset. Additionally, model performance was weakest when predicting an affective state transition and improved after multiple predictions of the same affective state. These findings emphasize the benefits of incorporating body features in continuous affect recognition for infants. Our work highlights the importance of evaluating variability in model performance both over time and in the presence of missing data when applying affect recognition to social interactions. 

   more » « less
3. Analyzing the Effect of Affective Priming on Emotional Annotations

   https://doi.org/10.1109/ACII59096.2023.10388184  

   Martinez-Lucas, Luz; Salman, Ali; Leem, Seong-Gyun; Upadhyay, Shreya G; Lee, Chi-Chun; Busso, Carlos (September 2023, IEEE)

   na (Ed.)

   In the field of affective computing, emotional annotations are highly important for both the recognition and synthesis of human emotions. Researchers must ensure that these emotional labels are adequate for modeling general human perception. An unavoidable part of obtaining such labels is that human annotators are exposed to known and unknown stimuli before and during the annotation process that can affect their perception. Emotional stimuli cause an affective priming effect, which is a pre-conscious phenomenon in which previous emotional stimuli affect the emotional perception of a current target stimulus. In this paper, we use sequences of emotional annotations during a perceptual evaluation to study the effect of affective priming on emotional ratings of speech. We observe that previous emotional sentences with extreme emotional content push annotations of current samples to the same extreme. We create a sentence-level bias metric to study the effect of affective priming on speech emotion recognition (SER) modeling. The metric is used to identify subsets in the database with more affective priming bias intentionally creating biased datasets. We train and test SER models using the full and biased datasets. Our results show that although the biased datasets have low inter-evaluator agreements, SER models for arousal and dominance trained with those datasets perform the best. For valence, the models trained with the less-biased datasets perform the best. 

   more » « less
4. Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions

   https://doi.org/10.1038/s41598-019-52891-2  

   Siddharth, Siddharth; Jung, Tzyy-Ping; Sejnowski, Terrence J. (November 2019, Scientific Reports)

   Abstract Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers. 

   more » « less
5. Attributing responsibility for performance failure on worker-robot trust in construction collaborative tasks

   https://doi.org/10.35490/EC3.2023.205  

   Chang, Woei-Chyi; Ryan, Sophia Marie; Hasanzadeh, Sogand; Esmaeili, Behzad (July 2023, European Council on Computing in Construction)

   Recent advances in construction automation increased the need for cooperation between workers and robots, where workers have to face both success and failure in human-robot collaborative work, ultimately affecting their trust in robots. This study simulated a worker-robot bricklaying collaborative task to examine the impacts of blame targets (responsibility attributions) on trust and trust transfer in multi-robots-human interaction. The findings showed that workers’ responsibility attributions to themselves or robots significantly affect their trust in the robot. Further, in a multi-robots-human interaction, observing one robot’s failure to complete the task will affect the trust in the other devices, aka., trust transfer. 

   more » « less