This content will become publicly available on June 14, 2026
Title: Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games
Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost. more »« less
Herrick, I.R.; Sinatra, G.; Kennedy, A.; Nye, B.; Swartout, W.; Lindsey, E.
(, National Association for Research in Science Teaching (NARST) International Conference)
A key mission for museums is to engage a large and diverse public audience in science learning (Macdonald, 1997). To that end, science museums attempt to use immersive technologies in entertaining, socially oriented, and innovative ways. An example is the use of augmented reality (AR) to overlay virtual objects onto the real-world (Azuma, Baillot, Behringer, Feiner, Julier, & MacIntyre, 2001).We used a Design Based Research (DBR) approach to develop and test four features of an AR experience to promote place-based science learning in an museum setting. While quantitative differences were not found among conditions in knowledge gained, significant learning gains were seen from pre to post, illustrating the potential for place-based informal science learning. Incorporating AR technology into museum exhibits can update them with 21st tools to support visitor engagement in the learning experience. This research contributes to understanding of usability and logistical issues for different AR designs for a public, outdoor informal settings.
Letavish, Sean; Meliksetyan, Ani; Ravel, Victoria; Ok, Hurriyet. A.; Milman, Natalie B.; Adam, Gina C.
(, 2023 American Society for Engineering Education (ASEE) Annual Conference & Exposition)
Given the strategic importance of the semiconductor manufacturing sector and the CHIPS Act impact on microelectronics, it is more imperative than ever to train the next generation of scientists and engineers in the field. However, this is a challenging feat since nanofabrication education uses hands-on cleanroom facilities. Since cleanrooms are expensive, have access constraints due to safety concerns, and offer limited instructional space, class sizes and outreach events are limited. To complement instruction in nanotechnology education, there is some open- or educational-access software, which is computer-based and focuses only on training for individual equipment, not on the typical workflow for device fabrication. The objective of this work was to develop an accessible virtual reality ecosystem that provides an immersive education and outreach on device nanofabrication that is user-friendly for a broad range of audiences. At the George Washington University (GWU), a virtual reality cleanroom prototype has been developed. It consists of a 45-minute gameplay module that covers the process flow for the fabrication of micro-scale resistors, from sample preparation to electrical characterization. We also performed a mixed methods study to investigate how 5 students in a nanoelectronics course utilized this virtual reality cleanroom prototype and what changes they recommend to improve its user interface and learner experience. The study population for this work-in-progress consisted of students enrolled in a nanoelectronics course at GWU during the 2022-2023 school year. Students taking this course can be undergraduate (junior or senior) or graduate (masters or PhD). The research questions for this study were 1) what is the user experience with the virtual reality cleanroom prototype, 2) what challenges, if any, did students experience, and 3) what changes did students recommend to improve the virtual reality cleanroom prototype learner experience? Preliminary results indicate that the students found the virtual reality cleanroom simulator helpful in repeatedly exploring the cleanroom space and the nanofabrication process flow in a safe way, thus developing more confidence in utilizing the actual cleanroom facility. The results of this study will provide insight on the design of future modules with more complicated levels and device process flows. Moreover, the study could inform the development of other virtual reality simulators for other lab activities. The improved usability of the proposed software could provide students in large classes or attending online programs in electrical and computer engineering, as well as K-12 students participating in nanotechnology-related outreach events, the opportunity to conduct realistic process workflows, learn first-hand about nanofabrication, and practice using a nanofabrication lab via trial and error in a safe virtual environment.
Aqlan, Faisal; Zhao, Richard; Yang, Hui; Ramakrishnan, Sreekanth
(, Proceedings of the Winter Simulation Conference)
null
(Ed.)
Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience.
Aqlan, Faisal; Zhao, Richard; Yang, Hui; Ramakrishnan, Sreekanth
(, Proceedings of the Winter Simulation Conference)
null
(Ed.)
Virtual reality (VR) technology allows for the creation of fully immersive environments that enable personalized manufacturing learning. This case study discusses the development of a virtual learning factory that integrates manual and automated manufacturing processes such as welding, fastening, 3D printing, painting, and automated assembly. Two versions of the virtual factory are developed: (1) a multiplayer VR environment for the design and assembly of car toys; which allows for the collaboration of multiple users in the same VR environment, and (2) a virtual plant that utilizes heavy machinery and automated assembly lines for car manufacturing. The virtual factory also includes an intelligent avatar that can interact with the users and guide them to the different sections of the plant. The virtual factory enhances the learning of advanced manufacturing concepts by combining virtual objects with hands-on activities and providing students with an engaging learning experience.
Virtual Access to STEM Careers (VASC) is a technology-rich, inquiry and problem-based curriculum designed to expose and stimulate student interest in marine, environmental, computer, and geological sciences. Intended for 3rd through 5th grade students, VASC builds academic momentum at the intermediate level to prepare students for STEM opportunities later in middle school and high school. Our program is aligned with “Next Generation Science Standards” and “Common Core State Standards” and immerses students in rigorous, high-interest learning modules where students are introduced to and take on the roles of different STEM occupations. We are specifically developing and testing virtual reality-based modules that place students in a coastal environment where they learn about the sea turtle life-cycle. Students also practice the types of measurements and conservation tasks that park rangers and marine scientists regularly perform. The investigations focused on the design of a user interface that meets the needs of students and their teachers. We collected feedback on user interface design and knowledge gained by the users from the simulation. Additionally, we compared two different virtual reality head-mounted displays; i) HTC Vive and ii) Oculus Quest 2, to identify the pros and cons of each technology in future classroom settings. Our investigations yielded valuable information about how instructions should be presented to users, how the interface should provide immediate feedback for user error, how surveys should be administered, what equipment is most efficient for transporting and setting up large scale experiments in schools, and what types of interactions students and teachers want to experience in VASC.
Gagnon, D, Ponto, K, Swanson, L, and Tredinnick, R. Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games. Retrieved from https://par.nsf.gov/biblio/10609204.
Gagnon, D, Ponto, K, Swanson, L, & Tredinnick, R. Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games. Retrieved from https://par.nsf.gov/biblio/10609204.
Gagnon, D, Ponto, K, Swanson, L, and Tredinnick, R.
"Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games". Country unknown/Code not available: 2025 11th International Conference of the Immersive Learning Research Network (iLRN) Proceedings - Selected Academic Contributions. https://par.nsf.gov/biblio/10609204.
@article{osti_10609204,
place = {Country unknown/Code not available},
title = {Demonstrating Replay for Highly Scalable and Cost-Effective User Research of Virtual Reality Learning Games},
url = {https://par.nsf.gov/biblio/10609204},
abstractNote = {Observing users interacting with a learning game, often referred to as playtesting, is a critical component of usability testing. Unfortunately, this practice is expensive, requiring users and researchers to be in the same place at the same time as the participants. With Virtual Reality, this difficulty is amplified due to the experience being hidden from researcher view by default, and the extra complexity of setting up casting to an external monitor, especially with groups. In this paper we develop and utilize a replay-based approach to usability testing that relieves these concerns. The approach uses a low-bandwidth stream of telemetry signals that are generated by the original play session. These signals are then reconstructed into a full representation of the original experience at a different time or place and leveraged to identify usability issues. Once the issues have been discovered, automated processes are developed to computationally identify their presence and severity in arbitrarily large public audiences. This work contributes a demonstrated use of replay for Virtual Reality usability research, and a novel use of replay combined with educational data mining to develop an automated process for studying large audiences at low cost.},
journal = {},
publisher = {2025 11th International Conference of the Immersive Learning Research Network (iLRN) Proceedings - Selected Academic Contributions},
author = {Gagnon, D and Ponto, K and Swanson, L and Tredinnick, R},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
An official website of the United States government
