This content will become publicly available on June 14, 2026
Title: Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games
Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost. more »« less
Herrick, I.R.; Sinatra, G.; Kennedy, A.; Nye, B.; Swartout, W.; Lindsey, E.
(, National Association for Research in Science Teaching (NARST) International Conference)
A key mission for museums is to engage a large and diverse public audience in science learning (Macdonald, 1997). To that end, science museums attempt to use immersive technologies in entertaining, socially oriented, and innovative ways. An example is the use of augmented reality (AR) to overlay virtual objects onto the real-world (Azuma, Baillot, Behringer, Feiner, Julier, & MacIntyre, 2001).We used a Design Based Research (DBR) approach to develop and test four features of an AR experience to promote place-based science learning in an museum setting. While quantitative differences were not found among conditions in knowledge gained, significant learning gains were seen from pre to post, illustrating the potential for place-based informal science learning. Incorporating AR technology into museum exhibits can update them with 21st tools to support visitor engagement in the learning experience. This research contributes to understanding of usability and logistical issues for different AR designs for a public, outdoor informal settings.
Most of today’s studies investigating the driver-vehicle interaction of automated driving systems are conducted in simulated environments like driving simulators or virtual reality. While this simulation-based experimental research can produce valuable and valid results, it is at the same time limited by the inherent lack of realism. Important insights into real-world driving experiences and repeated system usage are rarely collected due to the constraints imposed by time and financial resources. In a two-step research approach, we aim to connect the AutoUI research with real-world users. In the first step, we conducted qualitative interviews with 10 experienced, tech-savvy users of current automated driving systems (Waymo, Cruise, Tesla) and clustered the results into the most important topics from a human factor perspective. On this basis, the workshop now aims to bring these insights into the AutoUI research community to identify the most relevant and urgent issues that should be addressed in the coming years.
Letavish, Sean; Meliksetyan, Ani; Ravel, Victoria; Ok, Hurriyet. A.; Milman, Natalie B.; Adam, Gina C.
(, 2023 American Society for Engineering Education (ASEE) Annual Conference & Exposition)
Given the strategic importance of the semiconductor manufacturing sector and the CHIPS Act impact on microelectronics, it is more imperative than ever to train the next generation of scientists and engineers in the field. However, this is a challenging feat since nanofabrication education uses hands-on cleanroom facilities. Since cleanrooms are expensive, have access constraints due to safety concerns, and offer limited instructional space, class sizes and outreach events are limited. To complement instruction in nanotechnology education, there is some open- or educational-access software, which is computer-based and focuses only on training for individual equipment, not on the typical workflow for device fabrication. The objective of this work was to develop an accessible virtual reality ecosystem that provides an immersive education and outreach on device nanofabrication that is user-friendly for a broad range of audiences. At the George Washington University (GWU), a virtual reality cleanroom prototype has been developed. It consists of a 45-minute gameplay module that covers the process flow for the fabrication of micro-scale resistors, from sample preparation to electrical characterization. We also performed a mixed methods study to investigate how 5 students in a nanoelectronics course utilized this virtual reality cleanroom prototype and what changes they recommend to improve its user interface and learner experience. The study population for this work-in-progress consisted of students enrolled in a nanoelectronics course at GWU during the 2022-2023 school year. Students taking this course can be undergraduate (junior or senior) or graduate (masters or PhD). The research questions for this study were 1) what is the user experience with the virtual reality cleanroom prototype, 2) what challenges, if any, did students experience, and 3) what changes did students recommend to improve the virtual reality cleanroom prototype learner experience? Preliminary results indicate that the students found the virtual reality cleanroom simulator helpful in repeatedly exploring the cleanroom space and the nanofabrication process flow in a safe way, thus developing more confidence in utilizing the actual cleanroom facility. The results of this study will provide insight on the design of future modules with more complicated levels and device process flows. Moreover, the study could inform the development of other virtual reality simulators for other lab activities. The improved usability of the proposed software could provide students in large classes or attending online programs in electrical and computer engineering, as well as K-12 students participating in nanotechnology-related outreach events, the opportunity to conduct realistic process workflows, learn first-hand about nanofabrication, and practice using a nanofabrication lab via trial and error in a safe virtual environment.
Aqlan, Faisal; Zhao, Richard; Yang, Hui; Ramakrishnan, Sreekanth
(, Proceedings of the Winter Simulation Conference)
null
(Ed.)
Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience.
Aqlan, Faisal; Zhao, Richard; Yang, Hui; Ramakrishnan, Sreekanth
(, Proceedings of the Winter Simulation Conference)
null
(Ed.)
Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience.
Gagnon, D, Ponto, K, Swanson, L, and Tredinnick, R. Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games. Retrieved from https://par.nsf.gov/biblio/10609204.
Gagnon, D, Ponto, K, Swanson, L, & Tredinnick, R. Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games. Retrieved from https://par.nsf.gov/biblio/10609204.
Gagnon, D, Ponto, K, Swanson, L, and Tredinnick, R.
"Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games". Country unknown/Code not available: 2025 11th International Conference of the Immersive Learning Research Network (iLRN) Proceedings - Selected Academic Contributions. https://par.nsf.gov/biblio/10609204.
@article{osti_10609204,
place = {Country unknown/Code not available},
title = {Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games},
url = {https://par.nsf.gov/biblio/10609204},
abstractNote = {Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost.},
journal = {},
publisher = {2025 11th International Conference of the Immersive Learning Research Network (iLRN) Proceedings - Selected Academic Contributions},
author = {Gagnon, D and Ponto, K and Swanson, L and Tredinnick, R},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
An official website of the United States government
