skip to main content


Title: HoloMentor: Enabling Remote Instruction through Augmented Reality Mobile Views
The introduction of Augmented Reality (AR) Head Mounted Displays (HMDs) in collaboration between remote and local workers, introduces new challenges given that camera views are now mobile. We introduce HoloMentor, an AR HMD-based collaborative system designed for remote instruction over live mobile views during physical tasks. Through Actionport, we provide a world-stabilized area where remote helpers can dynamically place a pointer and annotations on the physical environment. Through Actionpad, we provide an indirect input mechanism with an absolute position to the Actionport. We show how these innovations worked for participants engaged in a remote instructional task and how they supported effective and efficient communication. Finally, we provide the next steps for addressing AR on mobile views for remote instruction.  more » « less
Award ID(s):
2026510
NSF-PAR ID:
10389601
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Proceedings of the ACM on Human-Computer Interaction
Volume:
7
Issue:
GROUP
ISSN:
2573-0142
Page Range / eLocation ID:
1 to 29
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    Spatial reasoning skills contribute to performance in many STEM fields. For example, drawing sectional views of three-dimensional objects is an essential skill for engineering students. There is considerable variation in the spatial reasoning skills of prospective engineering students, putting some at risk for compromised performance in their classes. This study takes place in a first-year engineering Spatial Visualization course to integrate recent practices in engineering design education with cognitive psychology research on the nature of spatial learning. We employed three main pedagogical strategies in the course - 1) in class instruction on sketching; 2) spatial visualization training; and 3) manipulation of physical objects (CAD/3D print creations). This course endeavors to use current technology, online accessibility, and implementation of the three pedagogical strategies to bring about student growth in spatial reasoning. This study is designed to determine the effect of adding two different spatial reasoning training apps to this environment. Over 230 students (three sections) participated in our study. In two of the three sections, students received interactive spatial visualization training using either a spatial visualization mobile touchscreen app in one section or an Augmented Reality (AR) app in the other section. Research suggests that there are benefits to using the Spatial Vis Classroom mobile app for college students.The app has been shown to increase student persistence resulting in large learning gains as measured by the Purdue assessment of spatial visualization (PSVT-R), especially for students starting with poor spatial visualization skills. The Spatial Vis Classroom app can be used in the classroom or assigned as homework. The AR app is designed to help users develop their mental rotation abilities. It is designed to support a holistic understanding of 3-dimensional objects, and research has shown that, in combination with a traditional curriculum, it increases students’ abilities also measured by the PSVT-R. Of particular interest, the data suggest that the app overcomes the advantage found by males over females in a traditional class alone focused on spatial reasoning. Both of the course sections were required to use the apps for approximately the same time in class and outside of class. Students in the control section were required to do hand sketching activities in class and outside of class, with roughly the same completion time as for the sections with the apps. Students grades were not affected by using the three different approaches as grading was based on completion only. Based on current literature, we hypothesize that overall benefits (PSVT-R gains) will be comparable across the 3 treatments but there will be different effects on attitude and engagement (confidence,enjoyment, and self-efficacy). Lastly, we hypothesize that the treatments will have different effects on male/female and ethnic categories of the study participants. The final paper will include an analysis of results and a report of the findings. 
    more » « less
  2. null (Ed.)
    Immersive technologies such as Virtual Reality (VR) and Augmented Reality (AR) have become the worldwide huge technological innovations impacting human life significantly. While the VR is an enclosed environment separated completely from the real world, AR allows users to merge the digital and physical worlds and enable the interaction between them. The wide usage of AR has led researchers to investigate its potential capability in several areas including STEM-related fields. Previous research shows that AR assisted courses tend to enhance students’ learning, spatial cognition, increase the students’ motivation and engagement in the learning process. In this study, the researchers have developed an AR application to assist students with spatial cognition and remote course engagement independently. The ARCADE tool enables students to not only visualize the isometric product from its orthogonal views, but it also provides short tutorial clips on how a specific feature was developed and what tools were used. The students can perform basic modifications on the 3D part in the ARCADE such as section views, details views, scale, rotate and explode the assembly views. Although this project is a work in progress, the preliminary pretest and posttest results show there is a significant improvement in students’ spatial cognition when the proposed tool is used to assist the course. 
    more » « less
  3. null (Ed.)
    Due to the additive light model employed by most optical see-through head-mounted displays (OST-HMDs), they provide the best augmented reality (AR) views in dark environments, where the added AR light does not have to compete against existing real-world lighting. AR imagery displayed on such devices loses a significant amount of contrast in well-lit environments such as outdoors in direct sunlight. To compensate for this, OST-HMDs often use a tinted visor to reduce the amount of environment light that reaches the user’s eyes, which in turn results in a loss of contrast in the user’s physical environment. While these effects are well known and grounded in existing literature, formal measurements of the illuminance and contrast of modern OST-HMDs are currently missing. In this paper, we provide illuminance measurements for both the Microsoft HoloLens 1 and its successor the HoloLens 2 under varying environment lighting conditions ranging from 0 to 20,000 lux. We evaluate how environment lighting impacts the user by calculating contrast ratios between rendered black (transparent) and white imagery displayed under these conditions, and evaluate how the intensity of environment lighting is impacted by donning and using the HMD. Our results indicate the further need for refinement in the design of future OST-HMDs to optimize contrast in environments with illuminance values greater than or equal to those found in indoor working environments. 
    more » « less
  4. In the following position paper we introduce the use of AR HMD for remote instruction in healthcare and present the challenges our team has faced in achieving this application in two contexts: surgical telementoring and paramedic teleconsulting. After the presentation of how these challenges come to be and indications on how to address them, we argue that those who wish to pursue this area of research must be grounded in best practices from the field of CSCW integrated with technical innovations in AR interaction development. This is a truly interdisciplinary research and development area that has many challenging topics to tackle through collaborative efforts. 
    more » « less
  5. null (Ed.)
    COVID-19 has altered the landscape of teaching and learning. For those in in-service teacher education, workshops have been suspended causing programs to adapt their professional development to a virtual space to avoid indefinite postponement or cancellation. This paradigm shift in the way we conduct learning experiences creates several logistical and pedagogical challenges but also presents an important opportunity to conduct research about how learning happens in these new environments. This paper describes the approach we took to conduct research in a series of virtual workshops aimed at teaching rural elementary teachers about engineering practices and how to teach a unit from an engineering curriculum. Our work explores how engineering concepts and practices are socially constructed through interactions with teachers, students, and artifacts. This approach, called interactional ethnography has been used by the authors and others to learn about engineering teaching and learning in precollege classrooms. The approach relies on collecting data during instruction, such as video and audio recordings, interviews, and artifacts such as journal entries and photos of physical designs. Findings are triangulated by analyzing these data sources. This methodology was going to be applied in an in-person engineering education workshop for rural elementary teachers, however the pandemic forced us to conduct the workshops remotely. Teachers, working in pairs, were sent workshop supplies, and worked together during the training series that took place over Zoom over four days for four hours each session. The paper describes how we collected video and audio of teachers and the facilitators both in whole group and in breakout rooms. Class materials and submissions of photos and evaluations were managed using Google Classroom. Teachers took photos of their work and scanned written materials and submitted them all by email. Slide decks were shared by the users and their group responses were collected in real time. Workshop evaluations were collected after each meeting using Google Forms. Evaluation data suggest that the teachers were engaged by the experience, learned significantly about engineering concepts and the knowledge-producing practices of engineers, and feel confident about applying engineering activities in their classrooms. This methodology should be of interest to the membership for three distinct reasons. First, remote instruction is a reality in the near-term but will likely persist in some form. Although many of us prefer to teach in person, remote learning allows us to reach many more participants, including those living in remote and rural areas who cannot easily attend in-person sessions with engineering educators, so it benefits the field to learn how to teach effectively in this way. Second, it describes an emerging approach to engineering education research. Interactional ethnography has been applied in precollege classrooms, but this paper demonstrates how it can also be used in teacher professional development contexts. Third, based on our application of interactional ethnography to an education setting, readers will learn specifically about how to use online collaborative software and how to collect and organize data sources for research purposes. 
    more » « less