skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Examining the Use of Nonverbal Communication in Virtual Agents
Virtual agents are systems that add a social dimension to computing, often featuring not only natural language input but also an embodiment or avatar. This allows them to take on a more social role and leverage the use of nonverbal communication (NVC). In humans, NVC is used for many purposes, including communicating intent, directing attention, and conveying emotion. As a result, researchers have developed agents that emulate these behaviors. However, challenges pervade the design and development of NVC in agents. Some articles reveal inconsistencies in the benefits of agent NVC; others show signs of difficulties in the process of analyzing and implementing behaviors. Thus, it is unclear what the specific outcomes and effects of incorporating NVC in agents and what outstanding challenges underlie development. This survey seeks to review the uses, outcomes, and development of NVC in virtual agents to identify challenges and themes to improve and motivate the design of future virtual agents.  more » « less
Award ID(s):
1750840
PAR ID:
10250007
Author(s) / Creator(s):
;
Date Published:
Journal Name:
International Journal of Human–Computer Interaction
ISSN:
1044-7318
Page Range / eLocation ID:
1 to 26
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Culturally informed design for virtual agents has been shown to positively impact health outcomes when tailored to target audiences. We present a participatory design methodology for culturally tailoring virtual agents. Investigators worked with key informants from our target population, members of predominantly Black church communities, to design culturally-relevant and sensitive virtual agent health promotion interventions. In the first participatory session, key informants designed agents to assist them with different aspects of their lives, providing input on agent appearance and agent functionality. In a second design session, participants re-wrote the content of a health conversation with an agent, to include personally-relevant content related to their community (e.g., religious and scriptural references). We report design principles for religious tailoring derived from these studies. We conducted a validation study to assess the effects of applying these principles to agents that promoted two health behaviors, finding that participants responded very positively to the tailored agents. 
    more » « less
  2. The authors present the design and implementation of an exploratory virtual learning environment that assists children with autism (ASD) in learning science, technology, engineering, and mathematics (STEM) skills along with improving social-emotional and communication skills. The primary contribution of this exploratory research is how educational research informs technological advances in triggering a virtual AI companion (AIC) for children in need of social-emotional and communication skills development. The AIC adapts to students’ varying levels of needed support. This project began by using puppetry control (human-in-the-loop) of the AIC, assisting students with ASD in learning basic coding, practicing their social skills with the AIC, and attaining emotional recognition and regulation skills for effective communication and learning. The student is given the challenge to program a robot, Dash™, to move in a square. Based on observed behaviors, the puppeteer controls the virtual agent’s actions to support the student in coding the robot. The virtual agent’s actions that inform the development of the AIC include speech, facial expressions, gestures, respiration, and heart color changes coded to indicate emotional state. The paper provides exploratory findings of the first 2 years of this 5-year scaling-up research study. The outcomes discussed align with a common approach of research design used for students with disabilities, called single case study research. This type of design does not involve random control trial research; instead, the student acts as her or his own control subject. Students with ASD have substantial individual differences in their social skill deficits, behaviors, communications, and learning needs, which vary greatly from the norm and from other individuals identified with this disability. Therefore, findings are reported as changes within subjects instead of across subjects. While these exploratory observations serve as a basis for longer term research on a larger population, this paper focuses less on student learning and more on evolving technology in AIC and supporting students with ASD in STEM environments. 
    more » « less
  3. A significant number and range of challenges besetting sustainability can be traced to the actions and interactions of multiple autonomous agents (people mostly) and the entities they create (e.g., institutions, policies, social network) in the corresponding social-environmental systems (SES). To address these challenges, we need to understand decisions made and actions taken by agents, the outcomes of their actions, including the feedbacks on the corresponding agents and environment. The science of Agent-based Complex Systems—ACS science—has a significant potential to handle such challenges. The advantages of ACS science for sustainability are addressed by way of identifying the key elements and challenges in sustainability science, the generic features of ACS, and the key advances and challenges in modeling ACS. Artificial intelligence and data science promise to improve understanding of agents’ behaviors, detect SES structures, and formulate SES mechanisms. 
    more » « less
  4. A significant number and range of challenges besetting sustainability can be traced to the actions and interactions of multiple autonomous agents (people mostly) and the entities they create (e.g., institutions, policies, social network) in the corresponding social-environmental systems (SES). To address these challenges, we need to understand decisions made and actions taken by agents, the outcomes of their actions, including the feedbacks on the corresponding agents and environment. The science of complex adaptive systems—CAS science—has a significant potential to handle such challenges. We address the advantages of CAS science for sustainability by identifying the key elements and challenges in sustainability science, the generic features of CAS, and the key advances and challenges in modeling CAS. Artificial intelligence and data science combined with agent-based modeling promise to improve understanding of agents’ behaviors, detect SES structures, and formulate SES mechanisms. 
    more » « less
  5. Researchers and educators have identified an urgent need for more rigorous teaching and learning about epidemiology topics and practices, such as engaging in behaviors that prevent the spread of viral disease such as COVID-19. Responding to this need, we designed a virtual epidemic as a special event hosted in a virtual world. In this paper we share the strategic, tactical, and detailed design of the SPIKEY-20 virtual epidemic and data that reflects back on the design in terms of player participation. Reflecting on the design, we ask: What kinds of players participated in the SPIKEY-20 virtual epidemic? How did players engage in designed activities (i.e., preventive measures, information seeking)? In what ways were players influenced by the concurrent real world pandemic of COVID-19? In the discussion we consider the potential connection points between real-life and virtual public health behaviors, new possibilities of classroom participation and teacher support for such a virtual event, and future design considerations for virtual epidemics. 
    more » « less