skip to main content


Search for: All records

Award ID contains: 1851591

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

  1. Human-robot interaction has played an increasingly significant role in more recent research involving the Theory of Mind (ToM). As the use of robot facilitators increases, questions arise regarding the implications of their involvement in a research setting. This work addresses the effects of a humanoid robot facilitator in a ToM assessment. This paper analyzes subjects’ performances on tasks meant to test ToM as those tasks are delivered by human or robot facilitators. Various modalities of data were collected: performance on ToM tasks, subjects’ perceptions of the robot, results from a ToM survey, and response duration. This paper highlights the effects of human-robot interactions in ToM assessments, which ultimately leads to a discussion on the effectiveness of using robot facilitators in future human-subject research. 
    more » « less
  2. We explore the extent to which empathetic reactions are elicited when subjects view 3D motion-capture driven avatar faces compared to viewing human faces. Through a remote study, we captured subjects' facial reactions when viewing avatar and humans faces, and elicited self reported feedback regarding empathy. Avatar faces varied by gender and realism. Results show no sign of facial mimicry; only mimicking of slight facial movements with no solid consistency. Participants tended to empathize with avatars when they could adequately identify the stimulus' emotion. As avatar realism increased, it negatively impacted the subjects' feelings towards the stimuli. 
    more » « less
  3. In order to build more human-like cognitive agents, systems capable of detecting various human emotions must be designed to respond appropriately. Confusion, the combination of an emotional and cognitive state, is under-explored. In this paper, we build upon prior work to develop models that detect confusion from three modalities: video (facial features), audio (prosodic features), and text (transcribed speech features). Our research improves the data collection process by allowing for continuous (as opposed to discrete) annotation of confusion levels. We also craft models based on recurrent neural networks (RNNs) given their ability to predict sequential data. In our experiments, we find that text and video modalities are the most important in predicting confusion while the explored audio features are relatively unimportant predictors of confusion in our data. 
    more » « less
  4. We provide an experience report about a remote framework for early undergraduate research experiences, which was thematically focused on sensing humans computationally. The framework included three complementary components. First, students experienced a team-based research cycle online, spanning formulating research questions, conducting literature review, performing fully remote human subject data collection experiments and data processing, analyzing and making inference over acquired data with computational experimentation, and disseminating findings. Second, the virtual program offered a set of professional development activities targeted to developing skills and knowledge for graduate school and research career trajectories. Third, it offered interactional and cohort-networking programming for community-building. We discuss not only the unique challenges of the virtual format and the steps put in place to address them but also the opportunities that being online afforded to innovate undergraduate research training remotely. We evaluate the remote training intervention through the organizing team’s post-program reflection and the students’ perceptions conveyed in exit interviews and a mid-program focus group. In addition to outlining lessons learned about more or less successful framework elements, we offer recommendations for applying the framework at other institutions as well as how to transfer activities to in-person formats. 
    more » « less
  5. Automated journalism technology is transforming news production and changing how audiences perceive the news. As automated text-generation models advance, it is important to understand how readers perceive human-written and machine-generated content. This study used OpenAI’s GPT-2 text-generation model (May 2019 release) and articles from news organizations across the political spectrum to study participants’ reactions to human- and machine-generated articles. As participants read the articles, we collected their facial expression and galvanic skin response (GSR) data together with self-reported perceptions of article source and content credibility. We also asked participants to identify their political affinity and assess the articles’ political tone to gain insight into the relationship between political leaning and article perception. Our results indicate that the May 2019 release of OpenAI’s GPT-2 model generated articles that were misidentified as written by a human close to half the time, while human-written articles were identified correctly as written by a human about 70 percent of the time. 
    more » « less
  6. Smiling and laughter are typically associated with amusement. If they occur under negative emotions, systems responding naively may confuse an uncomfortable smile or laugh with an amused state. We present a passive text and video elicitation task and collect spontaneous laughter and smiles in reaction to amusing and negative experiences, using standard, ubiquitous sensors (webcam and microphone), along with participant self-ratings. While we rely on a state-of-the-art smile recognizer, for laughter recognition our transfer learning architecture enhanced on modest data outperforms other models with up to 85% accuracy (F1 = 0.86), suggesting this technique as promising for improving affect models. Subsequently, we analyze and automatically predict laughter as amused vs. negative. However, contrasting with prior findings for acted data, for this spontaneously elicited dataset classifying laughter by emotional valence is not satisfactory. 
    more » « less
  7. We evaluated whether DHH individuals benefit from the addition of subtle visual effects to captioned educational videos to guide their gaze toward potentially informative content. 
    more » « less
  8. We report and review a visualization system capable of displaying gaze and speech data elicited from pairs of subjects interacting in a discussion. We elicit such conversation data in our first experiment, where two participants are given the task of reaching a consensus about questions involving images. We validate the system in a second experiment where the purpose is to see if a person could determine which question had elicited a certain visualization. The visualization system allows users to explore reasoning behavior and participation during multimodal dialogue interactions. 
    more » « less