Search for: All records

Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost. 

Given the incredible popularity of video games in contexts from entertainment to education, and the capacity of internet-connected games to record fine-grained telemetry data, there exists an unprecedented opportunity to investigate gameplay behaviors, outcomes, and their relationships to learning processes. However, with these opportunities come the need for technical infrastructures to manage the collection and analysis of massive amounts of game event data. In this work, we build upon existing literature to develop an architectural design for such infrastructure. We address issues of play data collection across many games; regular, repeatable extraction of gameplay features from raw data; and access to data for secondary analyses. In addition, we describe an implementation of this infrastructure and provide real-world examples of the implementation’s usage in prior large-scale analysis work. 

In this paper we describe the need for a framework to support collaborative educational research with game data, then demonstrate a promising solution. We review existing efforts, explore a collection of use cases and requirements, then propose a new data architecture with related data standards. The approach provides modularity to the various stages of game data generation and analysis, exposing intermediate transformations and work products. Foregrounding flexibility, each stage of the pipeline generates datasets for use in other tools and workflows. A series of interconnected standards allow for the development of reusable analysis and visualization tools across games, while remaining responsive to the diversity of potential game designs. Finally, we demonstrate the feasibility of the approach through an existing implementation that uses this architecture to process and analyze data from a wide range of games developed by multiple institutions, at scale, supporting a variety of research projects. 

Assessing scientific thinking and inquiry skills can be challenging because of the complexity and divergence in student behaviors. Scholars have advocated the use of more open-ended problems and choice for the assessment of scientific inquiry. In this paper, we interrogate an experimentation mechanic in an educational science game to examine challenges that choice introduces to game-based assessment of science inquiry practices. Descriptive analysis of gameplay elucidates the difference between choices to explore and iterating on choices as a sign of struggling to progress. 

In this paper we describe a technical infrastructure, entitled Open Game Data, for conducting educational game research using open science, educational data mining and learning engineering approaches. We describe a modular data pipeline which begins with telemetry events from gameplay and ends with real time APIs and automated archival exports that support research. We demonstrate the usefulness of this infrastructure by summarizing several game research projects that have utilized and contributed back to Open Game Data. We then conclude with current efforts to expand the infrastructure. 

In this paper we describe a technical infrastructure, entitled Open Game Data, for conducting educational game research using open science, educational data mining and learning engineering approaches. We describe a modular data pipeline which begins with telemetry events from gameplay and ends with real time APIs and automated archival exports that support research. We demonstrate the usefulness of this infrastructure by summarizing several game research projects that have utilized and contributed back to Open Game Data. We then conclude with current efforts to expand the infrastructure.