More Like this

1. Modeling Aesthetics and Emotions in Visual Content: From Vincent van Gogh to Robotics and Vision

   Wang, James Z. ( October 2020 , Proceedings of the Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, in conjunction with the ACM International Conference on Multimedia)

   As inborn characteristics, humans possess the ability to judge visual aesthetics, feel the emotions from the environment, and comprehend others’ emotional expressions. Many exciting applications become possible if robots or computers can be empowered with similar capabilities. Modeling aesthetics, evoked emotions, and emotional expressions automatically in unconstrained situations, however, is daunting due to the lack of a full understanding of the relationship between low-level visual content and high-level aesthetics or emotional expressions. With the growing availability of data, it is possible to tackle these problems using machine learning and statistical modeling approaches. In the talk, I provide an overview of our research in the last two decades on data-driven analyses of visual artworks and digital visual content for modeling aesthetics and emotions. First, I discuss our analyses of styles in visual artworks. Art historians have long observed the highly characteristic brushstroke styles of Vincent van Gogh and have relied on discerning these styles for authenticating and dating his works. In our work, we compared van Gogh with his contemporaries by statistically analyzing a massive set of automatically extracted brushstrokes. A novel extraction method is developed by exploiting an integration of edge detection and clustering-based segmentation. Evidence substantiates that van Gogh’smore »brushstrokes are strongly rhythmic. Next, I describe an effort to model the aesthetic and emotional characteristics in visual contents such as photographs. By taking a data-driven approach, using the Internet as the data source, we show that computers can be trained to recognize various characteristics that are highly relevant to aesthetics and emotions. Future computer systems equipped with such capabilities are expected to help millions of users in unimagined ways. Finally, I highlight our research on automated recognition of bodily expression of emotion. We propose a scalable and reliable crowdsourcing approach for collecting in-the-wild perceived emotion data for computers to learn to recognize the body language of humans. Comprehensive statistical analysis revealed many interesting insights from the dataset. A system to model the emotional expressions based on bodily movements, named ARBEE (Automated Recognition of Bodily Expression of Emotion), has also been developed and evaluated.« less
2. Micro-expressions in Animated Agents

   https://doi.org/10.54941/ahfe1001081  

   Hou, Tianyu ; Adamo, Nicoletta ; Villani, Nicholas ( January 2022 , AHFE International)

   The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other basedmore »on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/

   « less
3. Maternal reactions to children’s negative emotions: Relations to children's socio-emotional development among European American and Chinese immigrant children

   https://doi.org/10.1177/1363461520905997  

   Yang, Yang ; Song, Qingfang ; Doan, Stacey N. ; Wang, Qi ( February 2020 , Transcultural Psychiatry)

   This study examined the relations between maternal reactions to children’s negative emotions and children’s socio-emotional outcomes, including psychological adjustment, emotion knowledge, and coping strategies. European American and Chinese immigrant mothers reported on their reactions to children’s ( N = 117, M = 7.14 years) negative emotions and on children’s psychological adjustment. One year later, children were interviewed for emotion knowledge and mothers reported on children’s use of coping strategies. Mothers from the two cultural groups reported the same level of supportive reactions to their children’s negative emotions, whereas Chinese immigrant mothers more often adopted what are commonly considered to be non-supportive strategies than did European American mothers. Whereas supportive maternal reactions were associated with better child outcomes in both cultures, maternal non-supportive reactions were negatively associated with children’s functioning for European American children but not for Chinese immigrant children. The findings shed critical light on the functional meaning of parenting practices in specific cultural contexts in shaping developmental outcomes.
4. The Centrality of Black Identity for Black Students in Engineering

   Mobley, Catherine ; Brawner, Catherine E. ; Brent, Rebecca ; Orr, Marisa K. ( January 2021 , Collaborative Network for Engineering and Computing Diversity (CoNECD) Conference)

   Introduction and Theoretical Frameworks Our study draws upon several theoretical foundations to investigate and explain the educational experiences of Black students majoring in ME, CpE, and EE: intersectionality, critical race theory, and community cultural wealth theory. Intersectionality explains how gender operates together with race, not independently, to produce multiple, overlapping forms of discrimination and social inequality (Crenshaw, 1989; Collins, 2013). Critical race theory recognizes the unique experiences of marginalized groups and strives to identify the micro- and macro-institutional sources of discrimination and prejudice (Delgado & Stefancic, 2001). Community cultural wealth integrates an asset-based perspective to our analysis of engineering education to assist in the identification of factors that contribute to the success of engineering students (Yosso, 2005). These three theoretical frameworks are buttressed by our use of Racial Identity Theory, which expands understanding about the significance and meaning associated with students’ sense of group membership. Sellers and colleagues (1997) introduced the Multidimensional Model of Racial Identity (MMRI), in which they indicated that racial identity refers to the “significance and meaning that African Americans place on race in defining themselves” (p. 19). The development of this model was based on the reality that individuals vary greatly in the extent to whichmore »they attach meaning to being a member of the Black racial group. Sellers et al. (1997) posited that there are four components of racial identity: 1. Racial salience: “the extent to which one’s race is a relevant part of one’s self-concept at a particular moment or in a particular situation” (p. 24). 2. Racial centrality: “the extent to which a person normatively defines himself or herself with regard to race” (p. 25). 3. Racial regard: “a person’s affective or evaluative judgment of his or her race in terms of positive-negative valence” (p. 26). This element consists of public regard and private regard. 4. Racial ideology: “composed of the individual’s beliefs, opinions and attitudes with respect to the way he or she feels that the members of the race should act” (p. 27). The resulting 56-item inventory, the Multidimensional Inventory of Black Identity (MIBI), provides a robust measure of Black identity that can be used across multiple contexts. Research Questions Our 3-year, mixed-method study of Black students in computer (CpE), electrical (EE) and mechanical engineering (ME) aims to identify institutional policies and practices that contribute to the retention and attrition of Black students in electrical, computer, and mechanical engineering. Our four study institutions include historically Black institutions as well as predominantly white institutions, all of which are in the top 15 nationally in the number of Black engineering graduates. We are using a transformative mixed-methods design to answer the following overarching research questions: 1. Why do Black men and women choose and persist in, or leave, EE, CpE, and ME? 2. What are the academic trajectories of Black men and women in EE, CpE, and ME? 3. In what way do these pathways vary by gender or institution? 4. What institutional policies and practices promote greater retention of Black engineering students? Methods This study of Black students in CpE, EE, and ME reports initial results from in-depth interviews at one HBCU and one PWI. We asked students about a variety of topics, including their sense of belonging on campus and in the major, experiences with discrimination, the impact of race on their experiences, and experiences with microaggressions. For this paper, we draw on two methodological approaches that allowed us to move beyond a traditional, linear approach to in-depth interviews, allowing for more diverse experiences and narratives to emerge. First, we used an identity circle to gain a better understanding of the relative importance to the participants of racial identity, as compared to other identities. The identity circle is a series of three concentric circles, surrounding an “inner core” representing one’s “core self.” Participants were asked to place various identities from a provided list that included demographic, family-related, and school-related identities on the identity circle to reflect the relative importance of the different identities to participants’ current engineering education experiences. Second, participants were asked to complete an 8-item survey which measured the “centrality” of racial identity as defined by Sellers et al. (1997). Following Enders’ (2018) reflection on the MMRI and Nigrescence Theory, we chose to use the measure of racial centrality as it is generally more stable across situations and best “describes the place race holds in the hierarchy of identities an individual possesses and answers the question ‘How important is race to me in my life?’” (p. 518). Participants completed the MIBI items at the end of the interview to allow us to learn more about the participants’ identification with their racial group, to avoid biasing their responses to the Identity Circle, and to avoid potentially creating a stereotype threat at the beginning of the interview. This paper focuses on the results of the MIBI survey and the identity circles to investigate whether these measures were correlated. Recognizing that Blackness (race) is not monolithic, we were interested in knowing the extent to which the participants considered their Black identity as central to their engineering education experiences. Combined with discussion about the identity circles, this approach allowed us to learn more about how other elements of identity may shape the participants’ educational experiences and outcomes and revealed possible differences in how participants may enact various points of their identity. Findings For this paper, we focus on the results for five HBCU students and 27 PWI students who completed the MIBI and identity circle. The overall MIBI average for HBCU students was 43 (out of a possible 56) and the overall MIBI scores ranged from 36-51; the overall MIBI average for the PWI students was 40; the overall MIBI scores for the PWI students ranged from 24-51. Twenty-one students placed race in the inner circle, indicating that race was central to their identity. Five placed race on the second, middle circle; three placed race on the third, outer circle. Three students did not place race on their identity circle. For our cross-case qualitative analysis, we will choose cases across the two institutions that represent low, medium and high MIBI scores and different ranges of centrality of race to identity, as expressed in the identity circles. Our final analysis will include descriptive quotes from these in-depth interviews to further elucidate the significance of race to the participants’ identities and engineering education experiences. The results will provide context for our larger study of a total of 60 Black students in engineering at our four study institutions. Theoretically, our study represents a new application of Racial Identity Theory and will provide a unique opportunity to apply the theories of intersectionality, critical race theory, and community cultural wealth theory. Methodologically, our findings provide insights into the utility of combining our two qualitative research tools, the MIBI centrality scale and the identity circle, to better understand the influence of race on the education experiences of Black students in engineering.« less
5. Sexist Slurs: Reinforcing Feminine Stereotypes Online

   https://doi.org/10.1007/s11199-019-01095-z  

   Felmlee, Diane ; Inara Rodis, Paulina ; Zhang, Amy ( November 2019 , Sex Roles)

   Social media platforms are accused repeatedly of creating environments in which women are bullied and harassed. We argue that online aggression toward women aims to reinforce traditional feminine norms and stereotypes. In a mixed methods study, we find that this type of aggression on Twitter is common and extensive and that it can spread far beyond the original target. We locate over 2.9 million tweets in one week that contain instances of gendered insults (e.g., “bitch,” “cunt,” “slut,” or “whore”)—averaging 419,000 sexist slurs per day. The vast majority of these tweets are negative in sentiment. We analyze the social networks of the conversations that ensue in several cases and demonstrate how the use of “replies,” “retweets,” and “likes” can further victimize a target. Additionally, we develop a sentiment classifier that we use in a regression analysis to compare the negativity of sexist messages. We find that words in a message that reinforce feminine stereotypes inflate the negative sentiment of tweets to a significant and sizeable degree. These terms include those insulting someone’s appearance (e.g., “ugly”), intellect (e.g., “stupid”), sexual experience (e.g., “promiscuous”), mental stability (e.g., “crazy”), and age (“old”). Messages enforcing beauty norms tend to be particularly negative. In sum,more »hostile, sexist tweets are strategic in nature. They aim to promote traditional, cultural beliefs about femininity, such as beauty ideals, and they shame victims by accusing them of falling short of these standards. Harassment on social media constitutes an everyday, routine occurrence, with researchers finding 9,764,583 messages referencing bullying on Twitter over the span of two years (Bellmore et al. 2015). In other words, Twitter users post over 13,000 bullying-related messages on a daily basis. Forms of online aggression also carry with them serious, negative consequences. Repeated research documents that bullying victims suffer from a host of deleterious outcomes, such as low self-esteem (Hinduja and Patchin 2010), emotional and psychological distress (Ybarra et al. 2006), and negative emotions (Faris and Felmlee 2014; Juvonen and Gross 2008). Compared to those who have not been attacked, victims also tend to report more incidents of suicide ideation and attempted suicide (Hinduja and Patchin 2010). Several studies document that the targets of cyberbullying are disproportionately women (Backe et al. 2018; Felmlee and Faris 2016; Hinduja and Patchin 2010; Pew Research Center 2017), although there are exceptions depending on definitions and venues. Yet, we know little about the content or pattern of cyber aggression directed toward women in online forums. The purpose of the present research, therefore, is to examine in detail the practice of aggressive messaging that targets women and femininity within the social media venue of Twitter. Using both qualitative and quantitative analyses, we investigate the role of gender norm regulation in these patterns of cyber aggression.« less