Search for: All records

The authors in this article provide a historical view (past) on the development of mixed reality (MR) simulation in teacher education as well as a brief history of simulation from other fields along with foundational knowledge on the evolution of simulation. The authors provide a systematic review of the current state (present) of the research in MR for teacher education within the past 5 years aligned with the research question “What are the uses, practices, and outcomes of MR simulation in teacher preparation?”. Three themes were identified, i.e., simulation to this point is designed by teacher educators, feedback matters in impacting outcomes, and practice is safe and reflective for those who prepare teachers in these environments. A summary is provided of these key articles and the findings. The authors conclude the article by sharing the potential evolution (future) of aspects of the model of MR, focusing on the use of AI agents and multi-modal data collection, including biometric signals, providing insights into simulation in teacher education. 

Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant’s emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives. 

Many studies have demonstrated the usefulness of virtual characters in educational settings; however, widespread adoption of such tools is limited by development costs and accessibility. This article describes a novel platform Web Automated Virtual Environment (WAVE) to deliver virtual experiences through the web. The system integrates data acquired from a variety of sources in a manner that allows the virtual characters to exhibit behaviors that are appropriate to the designer’s goals, such as providing support for users based on understanding their activities and their emotional states. Our WAVE platform overcomes the challenge of the scalability of the human-in-the-loop model by employing a web-based system and triggering automated character behavior. Therefore, we plan to make WAVE freely accessible (part of the Open Education Resources) and available anytime, anywhere. 

The authors present the design and implementation of an exploratory virtual learning environment that assists children with autism (ASD) in learning science, technology, engineering, and mathematics (STEM) skills along with improving social-emotional and communication skills. The primary contribution of this exploratory research is how educational research informs technological advances in triggering a virtual AI companion (AIC) for children in need of social-emotional and communication skills development. The AIC adapts to students’ varying levels of needed support. This project began by using puppetry control (human-in-the-loop) of the AIC, assisting students with ASD in learning basic coding, practicing their social skills with the AIC, and attaining emotional recognition and regulation skills for effective communication and learning. The student is given the challenge to program a robot, Dash™, to move in a square. Based on observed behaviors, the puppeteer controls the virtual agent’s actions to support the student in coding the robot. The virtual agent’s actions that inform the development of the AIC include speech, facial expressions, gestures, respiration, and heart color changes coded to indicate emotional state. The paper provides exploratory findings of the first 2 years of this 5-year scaling-up research study. The outcomes discussed align with a common approach of research design used for students with disabilities, called single case study research. This type of design does not involve random control trial research; instead, the student acts as her or his own control subject. Students with ASD have substantial individual differences in their social skill deficits, behaviors, communications, and learning needs, which vary greatly from the norm and from other individuals identified with this disability. Therefore, findings are reported as changes within subjects instead of across subjects. While these exploratory observations serve as a basis for longer term research on a larger population, this paper focuses less on student learning and more on evolving technology in AIC and supporting students with ASD in STEM environments. 

Law enforcement professionals require up to date training for interacting with individuals on the autism spectrum in a manner that facilitates positive citizen response. Although these officers interact with the public regularly, they may only have sporadic interactions with citizens who are not neurotypical. The timing of these interactions is not easy to predict; therefore, it is important to provide regular opportunities to practice contacts with special needs communities. However, in much the same way that it can be difficult to provide regular sessions with other protected groups of people, it is not practical to pull individuals on the autism spectrum to participate in law enforcement training. Role play with neurotypical individuals and classroom training presenting facts about autism do little to prepare these officers for their real-world encounters. Virtual interactions with people on the autism spectrum allow officers to practice techniques without compromising the health and safety of the communities they serve. This paper presents results of a study comparing police training through experiences in virtual reality (VR) with video training regarding police interactions with individuals on the autism spectrum. Police officers in a municipal police department who participated in the study were divided into three groups for continuing training purposes. One group received video training, one group received practice in VR, and one group received training through both video and VR. The differences in training method did not result in significant differences in training effectiveness. However, subjective data did support the efficacy of practice in a virtual setting. This project addressed three important challenges with training in VR. First, the team needed to define the specifics of behavior and language that the simulated individuals would exhibit. Second, the VR had to be tailored to be relevant to the officers participating. Third and finally, the schedule for training delivery had to minimize the time that officers were away from their assigned duties. Officer feedback on their training experiences indicated the approach to these challenges was well-received. The primary research question is whether training in VR is any more effective that watching a training video. 

In many scenarios, 1) data streams are generated in real time; 2) labeled data are expensive and only limited labels are available in the beginning; 3) real-world data is not always i.i.d. and data drift over time gradually; 4) the storage of historical streams is limited. This learning setting limits the applicability and availability of many Machine Learning (ML) algorithms. We generalize the learning task under such setting as a semi-supervised drifted stream learning with short lookback problem (SDSL). SDSL imposes two under-addressed challenges on existing methods in semi-supervised learning and continuous learning: 1) robust pseudo-labeling under gradual shifts and 2) anti-forgetting adaptation with short lookback. To tackle these challenges, we propose a principled and generic generation-replay framework to solve SDSL. To achieve robust pseudo-labeling, we develop a novel pseudo-label classification model to leverage supervised knowledge of previously labeled data, unsupervised knowledge of new data, and, structure knowledge of invariant label semantics. To achieve adaptive anti-forgetting model replay, we propose to view the anti-forgetting adaptation task as a flat region search problem. We propose a novel minimax game-based replay objective function to solve the flat region search problem and develop an effective optimization solver. Experimental results demonstrate the effectiveness of the proposed method.