This work describes the design of real-time dance-based interaction with a humanoid robot, where the robot seeks to promote physical activity in children by taking on multiple roles as a dance partner. It acts as a leader by initiating dances but can also act as a follower by mimicking a child’s dance movements. Dances in the leader role are produced by a sequence-to-sequence (S2S) Long Short-Term Memory (LSTM) network trained on children’s music videos taken from YouTube. On the other hand, a music orchestration platform is implemented to generate background music in the follower mode as the robot mimics the child’s poses. In doing so, we also incorporated the largely unexplored paradigm of learning-by-teaching by including multiple robot roles that allow the child to both learn from and teach to the robot. Our work is among the first to implement a largely autonomous, real-time full-body dance interaction with a bipedal humanoid robot that also explores the impact of the robot roles on child engagement. Importantly, we also incorporated in our design formal constructs taken from autism therapy, such as the least-to-most prompting hierarchy, reinforcements for positive behaviors, and a time delay to make behavioral observations. We implemented a multimodal child engagement model that encompasses both affective engagement (displayed through eye gaze focus and facial expressions) as well as task engagement (determined by the level of physical activity) to determine child engagement states. We then conducted a virtual exploratory user study to evaluate the impact of mixed robot roles on user engagement and found no statistically significant difference in the children’s engagement in single-role and multiple-role interactions. While the children were observed to respond positively to both robot behaviors, they preferred the music-driven leader role over the movement-driven follower role, a result that can partly be attributed to the virtual nature of the study. Our findings support the utility of such a platform in practicing physical activity but indicate that further research is necessary to fully explore the impact of each robot role.
more »
« less
A Model-free Affective Reinforcement Learning Approach to Personalization of an Autonomous Social Robot Companion for Early Literacy
Personalized education technologies capable of delivering
adaptive interventions could play an important role in addressing
the needs of diverse young learners at a critical time
of school readiness. We present an innovative personalized
social robot learning companion system that utilizes children’s
verbal and nonverbal affective cues to modulate their
engagement and maximize their long-term learning gains.
We propose an affective reinforcement learning approach to
train a personalized policy for each student during an educational
activity where a child and a robot tell stories to each
other. Using the personalized policy, the robot selects stories
that are optimized for each child’s engagement and linguistic
skill progression. We recruited 67 bilingual and English language
learners between the ages of 4–6 years old to participate
in a between-subjects study to evaluate our system. Over
a three-month deployment in schools, a unique storytelling
policy was trained to deliver a personalized story curriculum
for each child in the Personalized group. We compared their
engagement and learning outcomes to a Non-personalized
group with a fixed curriculum robot, and a baseline group that
had no robot intervention. In the Personalization condition,
our results show that the affective policy successfully personalized
to each child to boost their engagement and outcomes
with respect to learning and retaining more target words as
well as using more target syntax structures as compared to
children in the other groups.
more »
« less
- Award ID(s):
- 1523118
- NSF-PAR ID:
- 10108263
- Date Published:
- Journal Name:
- Proceedings of the ... AAAI Conference on Artificial Intelligence
- ISSN:
- 2159-5399
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Broadband infrastructure in urban parks may serve crucial functions including an amenity to boost overall park use and a bridge to propagate WiFi access into contiguous neighborhoods. This project: SCC:PG Park WiFi as a BRIDGE to Community Resilience has developed a new model —Build Resilience through the Internet and Digital Greenspace Exposure, leveraging off-the-shelf WiFi technology, novel algorithms, community assets, and local partnerships to lower greenspace WiFi costs. This interdisciplinary work leverages: computer science, information studies, landscape architecture, and public health. Collaboration methodologies and relational definitions across disciplines are still nascent —especially when paired with civic-engaged, applied research. Student researchers (UG/Grad) are excellent partners in bridging disciplinary barriers and constraints. Their capacity to assimilate multiple frameworks has produced refinements to the project’s theoretical lenses and suggested novel socio-technical methodology improvements. Further, they are excellent ambassadors to community partners and stakeholders. In BRIDGE, we tested two mechanisms to augment student research participation. In both, we leveraged a classic, curriculum-based model named the Partnership for Action Learning in Sustainability program (PALS). This campus-wide, community-engaged initiative pairs faculty and students with community partners. PALS curates economic, environmental, and social sustainability challenges and scopes projects to customize appropriate coursework that addresses identified challenges. Outcomes include: literature searches, wireframes, and design plans that target solutions to civic problems. Constraints include the short semester timeframe and curriculum-learning-outcome constraints. (1) On BRIDGE, Dr. Kweon executed a semester-based Landscape Architecture PALS 400-level-studio. 18 undergraduates conducted in-class and in-field work to assess community needs and proposed design solutions for future park-wide WiFi. Research topics included: community-park history, neighborhood demographics, case-study analysis, and land-cover characteristics. The students conducted an in-Park, community engagement session —via interactive posterboard surveys, to gain input on what park amenities might be redesigned or added to promote WiFi use. The students then produced seven re-design plans; one included a café/garden, with an eco-corridor that integrated technology with nature. (2) From the classic, curriculum-based PALS model we created a summer-intensive for our five research assistants, to stimulate interdisciplinary collaboration in their research tasks and co-analysis of project data products: experimental technical WiFi-setup, community survey results, and stakeholder needs-assessments. Students met weekly with each other and team leadership, exchanged journal articles, and attended joint research events. This model shows promise for integrating students more formally into an interdisciplinary research project. An end-of-intensive focus group highlighted, from the students’ perspective, the pro/cons of this model. Results: In contrasting the two mechanisms, our results include: Model 1 is tried-and-trued and produces standardized, reliable products. However, as work is group based, student independence is limited —to explore topics/themes of interest. Civic groups are typically thrilled with the diversity of action plans produced. Model 2 provides greater independence in student-learning outcomes, fosters interdisciplinary, “dictionary-building” that can be used by the full team, deepens methodological approaches, and allows for student stipend payments. Lessons learned: intensive time frame needed more research team support and ideally should be extended, when possible, over the full project-span. UMD-IRB#1785365-4; NSF-award: 2125526.more » « less
-
As K-12 engineering education becomes more ubiquitous in the U.S, increased attention has been paid to preparing the heterogeneous group of in-service teachers who have taken on the challenge of teaching engineering. Standards have emerged for professional development along with research on teacher learning in engineering that call for teachers to facilitate and support engineering learning environments. Given that many teachers may not have experienced engineering practice calls have been made to engage teaches K-12 teachers in the “doing” of engineering as part of their preparation. However, there is a need for research studying more specific nature of the “doing” and the instructional implications for engaging teachers in “doing” engineering. In general, to date, limited time and constrained resources necessitate that many professional development programs for K-12 teachers to engage participants in the same engineering activities they will enact with their students. While this approach supports teachers’ familiarity with curriculum and ability to anticipate students’ ideas, there is reason to believe that these experiences may not be authentic enough to support teachers in developing a rich understanding of the “doing” of engineering. K-12 teachers are often familiar with the materials and curricular solutions, given their experiences as adults, which means that engaging in the same tasks as their students may not be challenging enough to develop their understandings about engineering. This can then be consequential for their pedagogy: In our prior work, we found that teachers’ linear conceptions of the engineering design process can limit them from recognizing and supporting student engagement in productive design practices. Research on the development of engineering design practices with adults in undergraduate and professional engineering settings has shown significant differences in how adults approach and understand problems. Therefore, we conjectured that engaging teachers in more rigorous engineering challenges designed for adult engineering novices would more readily support their developing rich understandings of the ways in which professional engineers move through the design process. We term this approach meaningful engineering for teachers, and it is informed by work in science education that highlights the importance of learning environments creating a need for learners to develop and engage in disciplinary practices. We explored this approach to teachers’ professional learning experiences in doing engineering in an online graduate program for in-service teachers in engineering education at Tufts University entitled the Teacher Engineering Education Program (teep.tufts.edu). In this exploratory study, we asked: 1. How did teachers respond to engaging in meaningful engineering for teachers in the TEEP program? 2. What did teachers identify as important things they learned about engineering content and pedagogy? This paper focuses on one theme that emerged from teachers’ reflections. Our analysis found that teachers reported that meaningful engineering supported their development of epistemic empathy (“the act of understanding and appreciating someone's cognitive and emotional experience within an epistemic activity”) as a result of their own affective experiences in doing engineering that required significant iteration as well as using novel robotic materials. We consider how epistemic empathy may be an important aspect of teacher learning in K-12 engineering education and the potential implications for designing engineering teacher education.more » « less
-
This Innovate Practice full paper presents a cloud-based personalized learning lab platform. Personalized learning is gaining popularity in online computer science education due to its characteristics of pacing the learning progress and adapting the instructional approach to each individual learner from a diverse background. Among various instructional methods in computer science education, hands-on labs have unique requirements of understanding learner's behavior and assessing learner's performance for personalization. However, it is rarely addressed in existing research. In this paper, we propose a personalized learning platform called ThoTh Lab specifically designed for computer science hands-on labs in a cloud environment. ThoTh Lab can identify the learning style from student activities and adapt learning material accordingly. With the awareness of student learning styles, instructors are able to use techniques more suitable for the specific student, and hence, improve the speed and quality of the learning process. With that in mind, ThoTh Lab also provides student performance prediction, which allows the instructors to change the learning progress and take other measurements to help the students timely. For example, instructors may provide more detailed instructions to help slow starters, while assigning more challenging labs to those quick learners in the same class. To evaluate ThoTh Lab, we conducted an experiment and collected data from an upper-division cybersecurity class for undergraduate students at Arizona State University in the US. The results show that ThoTh Lab can identify learning style with reasonable accuracy. By leveraging the personalized lab platform for a senior level cybersecurity course, our lab-use study also shows that the presented solution improves students engagement with better understanding of lab assignments, spending more effort on hands-on projects, and thus greatly enhancing learning outcomes.more » « less
-
more » « less
Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant’s emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
