Search for: All records

Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost. 

Given the incredible popularity of video games in contexts from entertainment to education, and the capacity of internet-connected games to record fine-grained telemetry data, there exists an unprecedented opportunity to investigate gameplay behaviors, outcomes, and their relationships to learning processes. However, with these opportunities come the need for technical infrastructures to manage the collection and analysis of massive amounts of game event data. In this work, we build upon existing literature to develop an architectural design for such infrastructure. We address issues of play data collection across many games; regular, repeatable extraction of gameplay features from raw data; and access to data for secondary analyses. In addition, we describe an implementation of this infrastructure and provide real-world examples of the implementation’s usage in prior large-scale analysis work. 

This research considers the impact of a digital science game that provides immersive experiences in which participants take on the role of a scientist and learn through active engagement with simulated science environments and tools. Wake: Tales from the Aqualab is an immersive web-based middle school science game designed to teach science practices of experimentation, modeling, and argumentation in aquatic ecosystems. This paper describes findings from a study of approximately 250 middle school students who used a beta version of the game over two weeks. A pre-post survey of affective measures found significant gains in student science identity, self-efficacy, and interest. Classroom observations and interviews with students and teachers supported these findings, suggesting that the immersive qualities of the game helped students think of themselves as scientists and engage in authentic science practices, contributing to shifts in students’ attitudes and beliefs about science. 

Students in open-ended educational games have a number of different pathways that they can select to work productively through a learning activity. Educators and system designers may want to know which of these pathways are most effective for engagement, learning, or other desirable outcomes. In this paper, we investigate which prior jobs and factors are associated with higher rates of student quitting behavior in an educational science exploration game. We use a series of Chi squared analyses to identify the jobs with the highest rates of quitting overall, and we calculate logistic regressions within specific jobs to determine the potential factors that lead to students quitting those jobs. Our analysis revealed that for 23 of the 40 jobs examined, having experience in at least one previous job significantly decreased the chances of students quitting the subsequent job, and that completing specific prior jobs reduces quit rates on specific later jobs. In our discussion, we describe the challenges associated with modeling quitting behavior, and how these analyses could be used to better optimize students’ pathways through the game environment. Specially, guiding students through specific sequences of preliminary jobs before tackling more challenging jobs can improve their engagement and reduce dropout rates, thus optimizing their learning pathways. 

In this paper we describe the need for a framework to support collaborative educational research with game data, then demonstrate a promising solution. We review existing efforts, explore a collection of use cases and requirements, then propose a new data architecture with related data standards. The approach provides modularity to the various stages of game data generation and analysis, exposing intermediate transformations and work products. Foregrounding flexibility, each stage of the pipeline generates datasets for use in other tools and workflows. A series of interconnected standards allow for the development of reusable analysis and visualization tools across games, while remaining responsive to the diversity of potential game designs. Finally, we demonstrate the feasibility of the approach through an existing implementation that uses this architecture to process and analyze data from a wide range of games developed by multiple institutions, at scale, supporting a variety of research projects. 

Assessing scientific thinking and inquiry skills can be challenging because of the complexity and divergence in student behaviors. Scholars have advocated the use of more open-ended problems and choice for the assessment of scientific inquiry. In this paper, we interrogate an experimentation mechanic in an educational science game to examine challenges that choice introduces to game-based assessment of science inquiry practices. Descriptive analysis of gameplay elucidates the difference between choices to explore and iterating on choices as a sign of struggling to progress. 

This short paper presents a description and analysis of personal and social immersion features in a life science educational videogame for middle school students. The paper defines personal immersion in relation to the player taking on a virtual role and engaging with the game as if they are themselves a part of in-game events. Social immersion comes from situating the player within a virtual community, in which the player and other characters in the world interact based on their virtual roles. The research study involved interviews with 24 students who used the game over a two-week classroom implementation. The study explored student perceptions about immersive elements of the game, and their impact on student self-efficacy, interest, and identity in science. Student responses indicated that they experienced personal immersion through engaging with the game narrative as a virtual scientist and doing science tasks, and social immersion through role-based interactions with other virtual scientists. Students described how their self-efficacy, interest, and identity in science were impacted by personal and social immersive elements of the game. The paper contributes to the literature through an analysis of these specific immersive game mechanics and their impacts on student attitudes. 

In this paper we describe a technical infrastructure, entitled Open Game Data, for conducting educational game research using open science, educational data mining and learning engineering approaches. We describe a modular data pipeline which begins with telemetry events from gameplay and ends with real time APIs and automated archival exports that support research. We demonstrate the usefulness of this infrastructure by summarizing several game research projects that have utilized and contributed back to Open Game Data. We then conclude with current efforts to expand the infrastructure. 

In this paper we describe a technical infrastructure, entitled Open Game Data, for conducting educational game research using open science, educational data mining and learning engineering approaches. We describe a modular data pipeline which begins with telemetry events from gameplay and ends with real time APIs and automated archival exports that support research. We demonstrate the usefulness of this infrastructure by summarizing several game research projects that have utilized and contributed back to Open Game Data. We then conclude with current efforts to expand the infrastructure.