skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Multicraft: A Multimodal Interface for Supporting and Studying Learning in Minecraft
In this paper, we present work on bringing multimodal interaction to Minecraft. The platform, Multicraft, incorporates speech-based input, eye tracking, and natural language understanding to facilitate more equitable gameplay in Minecraft. We tested the platform with elementary, middle school students and college students through a collection of studies. Students found each of the provided modalities to be a compelling way to play Minecraft. Additionally, we discuss the ways that these different types of multimodal data can be used to identify the meaningful spatial reasoning practices that students demonstrate while playing Minecraft. Collectively, this paper emphasizes the opportunity to bridge a multimodal interface with a means for collecting rich data that can better support diverse learners in non-traditional learning environments.  more » « less
Award ID(s):
1822865
PAR ID:
10290974
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
HCI in Games: Serious and Immersive Games. HCII 2021. Lecture Notes in Computer Science
Volume:
1
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; (, Spatial Cognition XII. Spatial Cognition 2020. Lecture Notes in Computer Science)
    Šķilters, J.; Newcombe, N.; Uttal, D. (Ed.)
    As excitement for Minecraft continues to grow, we consider its potential to function as an engaging environment for practicing and studying spatial reasoning. To support this exposition, we describe a glimpse of our current analysis of spatial reasoning skills in Minecraft. Twenty university students participated in a laboratory study that asked them to recreate three existing buildings in Minecraft. Screen captures of user actions, together with eye tracking data, helped us identify ways that students utilize perspective taking, constructing mental representations, building and place-marking, and error checking. These findings provide an initial impetus for further studies of the types of spatial skills that students may exhibit while playing Minecraft. It also introduces questions about how the design of Minecraft activities may promote, or inhibit, the use of certain spatial skills. 
    more » « less
  2. ; (, Computersupported collaborative learning)
    Spatial reasoning is an important skillset that is malleable to training interventions. One possible context for intervention is the popular video game Minecraft. Minecraft encourages users to engage in spatial manipulation of 3D objects. However, few papers have chronicled any in-game practices that might evidence spatial reasoning, or how we might study its development through the game. In this paper, we report on 11 middle school students’ spatial reasoning practices while playing Minecraft. We use audio and video data of student gameplay to delineate five in-game practices that align with spatial reasoning. We expand on a student case study, to explicate these practices. The identified practices may be beneficial for studying spatial reasoning development in game-based environments and contribute to a growing body of research on ways games support development of important and transferable skills. 
    more » « less
  3. ; (, Springer)
    Wang, N; Rebolledo-Mendez, G; Dimitrova, V; Matsuda, N; Santos, O C (Ed.)
    Minecraft continues to be a popular digital game throughout the world, and the ways in which adolescents play can provide insight into their existing interests. Through informal summer camps using Minecraft to expose middle school students to concepts in astronomy and earth science, we collected self-reports of STEM and Minecraft interest, as well as behavioral log data through player in-game interactions. Finding relationships between in-game behaviors and individual interest can provide insight into how educational experiences in digital games might be designed to support learner interests and competencies in STEM. Bayesian model averaging of data across camps was implemented to address the relatively small sample size of the data. Results revealed the important role of existing interest and knowledge for developing and sustaining interest. 
    more » « less
  4. ; (, International Educational Data Mining Society)
    Benjamin, Paaßen; Carrie, Demmans Epp (Ed.)
    Extensive research underscores the importance of stimulating students' interest in learning, as it can improve key educational outcomes such as self-regulation, collaboration, problem-solving, and overall enjoyment. Yet, the mechanisms through which interest manifests and impacts learning remain less explored, particularly in open-ended game-based learning environments like Minecraft. The unstructured nature of gameplay data in such settings poses analytical challenges. This study employed advanced data mining techniques, including changepoint detection and clustering, to extract meaningful patterns from students' movement data. Changepoint detection allows us to pinpoint significant shifts in behavior and segment unstructured gameplay data into distinct phases characterized by unique movement patterns. This research goes beyond traditional session-level analysis, offering a dynamic view of the learning process as it captures changes in student behaviors while they navigate challenges and interact with the environment. Three distinct exploration patterns emerged: surface-level exploration, in-depth exploration, and dynamic exploration. Notably, we found a negative correlation between surface-level exploration and interest development, whereas dynamic exploration positively correlated with interest development, regardless of initial interest levels. In addition to providing insights into how interest can manifest in Minecraft gameplay behavior, this paper makes significant methodological contributions by showcasing innovative approaches for extracting meaningful patterns from unstructured behavioral data within game-based learning environments. The implications of our research extend beyond Minecraft, offering valuable insights into the applications of changepoint detection in educational research to investigate student behavior in open-ended and complex learning settings. 
    more » « less
  5. ; ; ; (, Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020),)
    This paper presents an expansion to the Abstract Meaning Representation (AMR) annotation schema that captures fine-grained semantically and pragmatically derived spatial information in grounded corpora. We describe a new lexical category conceptualization and set of spatial annotation tools built in the context of a multimodal corpus consisting of 185 3D structure-building dialogues between a human architect and human builder in Minecraft. Minecraft provides a particularly beneficial spatial relation-elicitation environment because it automatically tracks locations and orientations of objects and avatars in the space according to an absolute Cartesian coordinate system. Through a two-step process of sentence-level and document-level annotation designed to capture implicit information, we leverage these coordinates and bearings in the AMRs in combination with spatial framework annotation to ground the spatial language in the dialogues to absolute space. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email