Siyu Huang * Voicu Popescu †
Figure 1: HyperXRC designs illustrated with actual frames from the instructor's X R headset. The remote students (faces blurred for privacy) are integrated into the field of view of the instructor using either virtual billboards (left) or empty classroom seats (right).
This paper investigates HyperXRC, a hybrid classroom design that accommodates both local and remote students. The instructor wears an extended reality (XR) headset that shows the local classroom and the local students, as well as remote students modeled with video sprites. The remote students are displayed either on virtual banners hanging off the classroom ceiling, or on virtual billboards placed in empty classroom seats. Thereby, the remote students are integrated into the field of view of the instructor, who remains aware of the remote students while teaching. A controlled user study with two experiments evaluated the HyperXRC design from the instructor and from the local students perspective. In the first experiment (N = 15) participants served as instructors to a hybrid classroom of 14 local and 15 remote students. Participants were more likely to detect hand-raising and head-on-desk remote student actions in the HyperXRC conditions (59%) than in a conventional videoconferencing condition (36%). This advantage did not come at the cost of decreasing the detection rate of local student actions. Furthermore, instructor participants preferred the HyperXRC to the videoconferencing approach. In the second experiment (N = 16) participants served as local students. The participants preferred the lecture when the instructor used videoconferencing to the one when the instructor used HyperXRC, wearing the X R headset.
Index Terms: Human-centered computing-Human computer interaction (HCI)-Interaction paradigms-Mixed / augm. reality.
Distance education allows students to partake in educational activities remotely, without having to be located on the educational institution campus. Distance education thereby provides an important service to society, giving access to education to students who would otherwise find it difficult to enroll in on campus programs.
In asynchronous distance education the remote learner accesses pre-recorded educational materials, which brings the benefit of scheduling flexibility, but that also the shortcomings of lack of interaction between remote students and instructor, and between remote students and fellow local (on campus) students. Remote students engaged in asynchronous education often feel isolated which reduces motivation and the effectiveness of the learning experience [10].
In synchronous distance education the remote students participate in on-campus learning activities in real time. This promises to alleviate the isolation of remote students, albeit at the cost of reduced scheduling flexibility. Synchronous distance education typically focuses on remote lecture attendance. Current remote lecture attendance systems rely on videoconferencing [16]. Such systems fail to engage remote students adequately [7]. Furthermore, the local instructor and the local students also do not feel that the remote students are part of the classroom, which exacerbates the isolation of the remote students [18]. Another challenge faced by education institutions is the high cost of current distance education offerings, which require courses with special materials, with special rooms, and with instructors who have received special training.
Extended reality (XR) technology can bridge geographic distances for seamless real-time communication between sites and hence holds the promise of effective synchronous education solutions. Instead of specialized distance learning lecture halls with specialized equipment, X R has the potential to extend conventional on-campus education to seamlessly integrate remote students. The goal is to alleviate the remote student isolation and to increase their engagement that is cost effective and that does not interfere with the educational experience of local students. What is needed is a hybrid classroom, that can accommodate both local and remote students, in a variable ratio. For this to become reality, an important question is how to extend the local classroom to host remote students that minimizes the disruption for the instructor and the local students.
In this application/design study paper we investigate the design of a classroom that accommodates both local and remote students, and we evaluate it in a user study. We have dubbed the design HyperXRC, from HYbrid in-PErson and Remote X R Classroom. Each remote student is modeled with a background subtracted video sprite. The instructor wears an extended reality (XR) headset that shows the local classroom and the local students, as well as the remote student screen user interface of video conferencing imposes challenging focus shifts on the instructor, who cannot remain aware of even highly salient remote student non-verbal communication elements, while at the same time lecturing and maintaining awareness of the local students. HyperXRC leverages the power of extended reality technology for connecting disparate locations seamlessly, affording situational awareness uniformly across the hybrid class of students. video sprites. In one variant, i.e., HyperXRC Banners, the sprites are rendered hanging off the ceiling, and in a second variant, i.e., HyperXRC Seats, the sprites are placed in the empty seats of the classroom (Fig. 1). Thereby, the remote students are integrated into the field of view of the instructor, which has the potential to improve the instructor's ability to monitor the remote students body language. We also refer the reader to the video accompanying our submission.
We have conducted a controlled, within-subject user study to evaluate our HyperXRC design from the instructor perspective (Experiment 1) and from the local students perspective (Experiment 2). In Experiment 1 (N = 15) participants served as instructors and gave a mock-lecture on quadratic equations to a hybrid classroom with 14 local and 15 remote students. The remote students were modeled with prerecorded video sprites. The participant had to recognize when a student raised their hand or put their head on the desk (Fig. 2). The participant gave the lecture three times, in a videoconferencing like control condition, in a HyperXRC Banners first experimental condition, and in a HyperXRC Seats second experimental condition. The participant was more likely to notice the remote student actions in the HyperXRC conditions (detection rate of 59%) compared to the videoconferencing condition (detection rate of 36%). The participant detected local student actions at a rate of 61% for HyperXRC and 57% for the videoconferencing, which shows that the better monitoring of remote students did not come at the expense of a worse monitoring of local students. A subjective experience questionnaire showed that participants preferred the Hy-perXRC approaches to videoconferencing. In Experiment 2 (N = 16) participants served as local students and watched the same lecture given by the same instructor once using videoconferencing and once using HyperXRC. The experiment shows that the participants preferred the lecture when the instructor used videoconferencing to the one when the instructor used HyperXRC, wearing the headset. Overall, the two experiments indicate that the X R technology strengthens the connection between the instructor and the remote students, but at the expense of local students feeling disconnected.
An essential requirement for a successful hybrid classroom is that hosting the remote student does not come at the cost of reducing the effectiveness of the on-campus experience. As such, this first study examines the proposed HyperXRC design from the instructor and local student perspectives. The experience of remote students does not change significantly from conventional video conferencing. The HyperXRC design innovates at the local classroom level. Our first study uses prerecorded remote student video sprites to maintain control in experiments on instructor and local student perspectives, reducing confounding factors and strengthening the investigative power of the study. Our controlled study compares HyperXRC to conventional video conferencing, which proves to be inadequate in a hybrid synchronous distance education scenario. The separate small We first discuss the requirements for synchronous distance education and the shortcomings of current approaches as reflected by education research (Sec. 2.1), then we discuss prior implementations of hybrid classrooms that host both local and remote students (Sec. 2.2), and we conclude with a discussion of prior implementations of hybrid classrooms with the help of X R technology (Sec. 2.3).
Distance education is a fundamental societal need, and, as such, it has been discussed for a long time. Education research has been investigating the theoretical development, the implementation, and the pedagogy of distance learning, pointing both its needs and the shortcomings of available solutions [16,22,25]. Instructors rely on immediate access to the verbal and non-verbal actions of their students. Such actions are available to the instructor in conventional on-campus classroom settings, and they allow instructors to adapt their teaching methods in response to real-time student feedback. When the distance learning setting does not provide immediate access to the actions of their students, instructors find it challenging to simultaneously observe, interpret, and apply student feedback to adjust instruction. Consequently, instructors have to resort to pausing their teaching frequently to elicit explicit feedback from the students, interruptions that fragment the lesson [14,21].
In recent years, videoconferencing has become a popular approach for remote lecture attendance. Each participant is acquired by their webcam and everyone can see everyone else's video feed on their screen, arranged in a matrix of windows. Whereas cost effective, this method fails to connect participants meaningfully, especially when there are more than a handful of participants. As a consequence, remote students can experience a sense of isolation. This sentiment often stems from the perception that distance education fails to recreate the immersive and interactive ambiance of a real classroom setting [19]. The danger of isolation is even higher for students who may not naturally possess an inherent proactive engagement with their learning objectives [24]. The origin of this isolation has long been traced to limited opportunities for interaction [26].
In a traditional classroom setting, students have the opportunity to engage with their peers and teachers directly. This interaction is important for distance learning systems to replicate [7,24]. However, studies have found that in distance education interactions between students and instructors decreases [7,10,18].
Instructors cannot sustain adequate awareness of their students, who feel disconnected from the instructor and from their peers, the students are aware of the instructor's lack of class awareness, and, consequently, students disengage, reducing the effectiveness of the remote lecture. The problem is exacerbated when videoconferencing is used to allow remote students to attend an on-campus lecture. The remote students displayed on the instructor's laptop are all but ignored by the instructor and the local students, with the exception of erratic probing for "questions from the remote audience".
Synchronous hybrid classrooms enable remote students and local students to study together simultaneously, and it should allow the local instructor to interact with both local and remote students [15].
A recent systematic literature review discusses the state of the art, benefits, and challenges of hybrid classrooms, providing a set of guidelines [23]. The review finds that hybrid classrooms can have both pedagogical and organizational benefits that encompass: Increased institutional recruitment, flexible learning environments, reduced instructor workloads, diverse external educators, and equitable opportunities for underrepresented students are benefits observed [12,28,29]. The review also finds several challenges brought by hybrid classrooms. Teachers must adapt their approach to maintain a consistent instructional standard for both local and remote students, juggling tasks such as attending to both groups and managing actions like switching class materials in videoconferencing software, resulting in added mental strain for instructors [8].
One prior hybrid classroom design employs videoconferencing software to deliver the lecture to remote students, while simultaneously conducting a traditional lecture for local students [27]. The design also entails displaying the video feed of remote students on screens within the classroom. The researchers analyzed videoconferencing platforms and designed a customized audio, visual, and camera system tailored to the capabilities of cloud-based platforms. The screens were placed in the field of view of the instructor who could see the remote students as the instructor looked at the class. The approach incurs a high equipment and logistics cost.
Another design focused not on remote students, but rather on allowing the instructor to teach remotely [30]. In this setup instructors can deliver lectures using their laptops from outside the classroom. Meanwhile, local students gather in a classroom where they can view lecture slides projected on a screen, along with other media. They can also see the instructor and remote students on their laptops. However, this approach demands multitasking from both instructors and students, as they must simultaneously engage with class materials, their instructor, and their classmates.
Our focus is on enabling remote participation for students, with the instructor physically present in the classroom. The remote students are hosted through an extension of the local classroom. Our goal is to control equipment and logistics costs by virtualizing displays to integrate remote students into the instructor's field of view.
While numerous studies exist regarding distance learning and extended reality applications in education, there is a scarcity of research that specifically concentrates on integrating remote learners into the classroom environment through the utilization of extended reality.
One early investigation advocates creating a Shared Reality Environment [13] by displaying video streams of remote participants within the local classroom. The report outlines an initial implementation and argues that the approach could foster a sense of co-presence among participants. Early actual implementations extend the classroom by projecting virtual rows of seats on the back wall of the classroom [20]. The remote students are modeled with backgroundsubtracted video sprites, and the sprites are integrated into the virtual extension of the classroom. The hole-in-a-wall design was reused in a hybrid classroom where remote students are modeled with avatars [11]. The design integrates the remote students into the field of view of the instructor and shows the benefit of better instructor awareness of the remote students. The design has the moderate equipment cost of an additional ceiling projector and an additional screen. One shortcoming is that remote students are relegated to rows beyond the last rows of the classroom, even if there are empty classroom seats closer to the instructor, where the instructor would be able to sustain more easily awareness of the remote students.
Similarly, [9] executed the development of a 3D virtual environment to facilitate remote student attendance during lectures. Within this environment, remote students were able to observe both local students and lecture materials displayed on screens within the virtual world. In parallel, local students were granted the ability to view their remote classmates through a projected screen within the physical classroom. The study encompassed a qualitative analysis of participant evaluations. As per the feedback received from participants, the study identified a convergence of technological and pedagogical factors that contributed to desired outcomes such as effective communication and a sense of co-presence. However, the study also acknowledged that the intricate technical setup rendered the concept impractical for actual lecture settings.
Whereas earlier implementations of the X R hybrid classroom had to rely primarily on projectors to insert remote students in the local classroom, X R headsets have become a promising alternative with a much lower equipment and logistics cost. Indeed, we are in the middle of an X R headset revolution. All-in-one X R headsets now feature on-board inside-looking-out tracking, rendering, and power, offering users a fully untethered X R experience [2][3][4][5]. The X R headset revolution is poised to continue, with now three trilliondollar companies competing to advance the technology. Meta's Quest 3, released in fall 2023, promises a better pass-through than the Quest 2 at a lower price than the Quest Pro [4]. Apple's Vision Pro entry in the X R space, expected in early 2024, promises to unmask the user by displaying their face on the headset [1].
We set out to design a hybrid classroom that accommodates both local and remote students. We first analyze the hybrid classroom design space (Sec. 3.1) and we then describe our HyperXRC design and its implementation (Sec. 3.2). We limit the discussion of the hybrid classroom design to features relevant to the instructor and local students, who are the focus of our paper.
Designing a hybrid classroom requires solving several problems.
Remote student acquisition. There are a variety of distance education scenarios. In one scenario, several remote students are collocated at the same remote location, e.g., a group of high school students attending a college lecture from their high school classroom. We are concerned with a common distance education scenario where each remote student is at a different location, e.g., in their home office. In order to integrate a remote student into the local classroom, the remote student has to be acquired. Individual remote students seated in front their laptop can be conveniently acquired using a webcam, which is typically built into the laptop. Background subtraction now works reasonably well providing a real-time video sprite of the remote student that can be transmitted to the local classroom.
Remote student display for local instructor. The remote student visualization has to be inserted into the field of view of the instructor. The goal is to allow the instructor to see the remote students as the instructor lectures normally, without asking the instructor to change their behavior to account for the remote students. Conventional videoconferencing does not meet this requirement, as the instructor has to look more frequently at their lectern screen (e.g., laptop on instructor desk) in order to see the remote students. Furthermore, accommodating many remote students on a small screen reduces the visualization of each remote student to a thumbnail-sized video, which the instructor cannot monitor adequately as they lecture.
Extended reality technology can integrate remote students seamlessly into the field of view of the instructor. The instructor wears an X R headset that allows the instructor to see both the local classroom with the local students and the remote students. One option is an X R headset with a video pass-through mode [2][3][4], which offers a large active field of view and true opacity, providing rendering visibility even over the bright parts of the real world scene; the shortcomings are that the user does not see the real world scene directly, but rather a live video of it, which has resolution, dynamic range, distortion, and field of view issues. Another option is an X R headset with an optical pass-through [5], which lets the user see the real world directly, with their own eyes, but has the shortcomings of a small active field of view, limited brightness, and lack of true opacity.
The remote student video sprites are overlaid on top of the instructor's view of the local classroom. An important question is the video sprite placement. One option is to use the part of the instructor's field of view that is not occupied by the local students, e.g., at the top part of the instructor's field of view, to place the remote student sprites as virtual banners hanging off the classroom ceiling. Another option is to place the video sprites in the empty seats of the classroom. The first option has the advantage of not using any of the local classroom seats, increasing the classroom's capacity; the disadvantage is that the remote students are integrated into the local classroom at an unnatural location. The second option strives to integrate the remote students seamlessly into the local classroom, minimizing the difference between local and remote students.
Placing the remote students in the empty seats of the local classroom brings several challenges. One is that the total number of students is limited by the classroom capacity. Another is that depth compositing the video sprites with the local classroom is more difficult; with the ceiling banner placement the sprite is always in front of local classroom geometry so it can be drawn over the pass-through video feed; with the seat placement, the remote and local students have to be interleaved, which requires solving visibility at a finer grain. A third challenge is finding empty classroom seats where to display sprites. Solutions include reserving predetermined seats for remote students, relying on the instructor to assign seats to remote students, or computer vision automated detection of empty seats. Another potential problem is the insufficient dynamic range and resolution for the pass-through video for the instructor to easily see the lecture slides, which has to be solved by "virtualizing" the classroom projection screen and/or the instructor laptop screen.
An alternative to asking the instructor to wear an X R headset is to rely on a classroom projector facing backwards to "beam" the remote student sprites into the local classroom, either onto physical screens hanging like banners from the ceiling of the classroom, or directly onto empty classroom seats. The important advantage is that the instructor does not have to wear a headset. The disadvantages include having to equip the classroom with an additional projector and possibly additional screens, as well as causing local students discomfort from the bright projector aimed at them.
Remote student display for local students. Ideally, a hybrid classroom would also allow the local students to see the remote students. When the instructor wears an X R headset, the remote student sprites could be displayed on the classroom projection screen, at the cost of reducing the screen real estate allocated to the lecture slides. When the instructor does not wear a headset and the remote student sprites are projected into seats or onto dedicated ceiling banner screens, the local students can see them directly. A compromise solution is to show a remote student on the classroom screen only when the remote student is engaged in dialogue with the instructor.
Based on the design space analysis above, we have opted for the following design for our HyperXRC approach. We target the scenario of remote students each individually located at their own remote location. Each remote student is modeled with a real-time video sprite. The instructor wears an X R headset to see the remote student sprites as well as the local classroom. We have opted for a video pass-through X R headset, but much of our design and implementation would readily work with an optical pass-through headset. We provide two options for placing the remote student video sprites in the field of view of the instructor: HyperXRC Banners, with remote student sprites hanging off the ceiling like virtual banners, and Hy-perXRC Seats, with remote student sprites aligned with the empty seats of the local classroom (Fig. 1). For both options, the instructor sees the lecture slides hanging off the ceiling, on their own virtual banner. Virtualizing the lecture slides for the instructor is both (1) a need, as the passthrough mode of X R headsets might not provide the quality needed for laptop or classroom screen legibility, as well as (2) an opportunity, as it avoids that the instructor shifts focus away from the students to look at their laptop, or turn their back to the students to look at the classroom screen.
We modeled the student desk tabletops in the classroom manually, as a preprocess (yellow in Fig. 3). The classroom seats are then defined automatically as vertical rectangles (red and blue) aligned with the far edges of the tabletops. The empty seat rectangles (red) were used in the HyperXRC Seats configuration to display the remote student sprites. In the HyperXRC Banners configuration the sprites are displayed on rectangles hanging off the ceiling (green). For each session, the virtual and real worlds of the X R instructor headset were aligned using the four corners of the front desk.
Resolving visibility correctly between the virtual and real worlds is straightforward in the Banners configuration as the remote student sprites can be simply drawn over the video frame of the real world. In the Seats configuration, resolving visibility is more challenging as parts of the real world could occlude parts of the virtual world. Specifically, local (real) students and tabletops that are in front of remote (virtual) students should not be covered by the remote students' sprite. We resolve visibility by first rendering a preliminary z-buffer from the tabletop rectangles and the local student rectangles (blue in Fig. 3). We then render the remote student sprites on top of this preliminary z-buffer. This way, a local student or a tabletop is protected from being overwritten by a remote student seated behind.
The focus of the present work is designing the hybrid classroom in a way that integrates remote students seamlessly into the field of view of the instructor. The goal is for the instructor to maintain awareness of remote students, without this to come at the cost of a reduced awareness of the local students. The HyperXRC design assumes that the remote students rely on a conventional video conferencing interface that shows the instructor in a live video stream and shows the lecture material through a pre-downloaded deck of slides.
Our HyperXRC implementation uses Unity 3D version 2022.3 [6] and a Quest Pro headset [4] with frame rate 90Hz. The rendering load is negligible, as the system only has to render a few sprites.
We have evaluated the hybrid classroom design in a controlled user study, with the approval of the ethics commission of our institution. A first experiment compares HyperXRC Banners and HyperXRC Seats to videoconferencing from the perspective of the instructor, for whom the placement of the remote students is highly relevant (Section 4.1). A second experiment compares HyperXRC Seats to videoconferencing from the local student perspective (Section 4.2).
We describe the methods and then present and discuss the results.
Participants. We evaluated the HyperXRC design from the instructor's point of view with N = 15 participants, recruited from the graduate student population of our university. One participant had an age between 18 and 22, 12 participants had an age between 23 and 29, and two participants had an age over 30. 14 participants self-identified as male and 1 as female. Two participants had served before as undergraduate teaching assistants, 13 had served before as graduate teaching assistants, and 2 had served before as course instructors. 14 participants had given presentations via videoconferencing to remote audiences, and 8 had given a presentation to a local audience extended with remote participants via videoconferencing. Regarding V R experience, 8 participants indicated that they had never used V R before, 3 had used V R once, 1 a few times, and 3 frequently. Regarding A R experience, 6 participants indicated they had none, 3 had used A R once before, 5 had used A R a few times before, and 1 had used A R frequently.
Task. The participant served in the role of an instructor with a hybrid audience of local and remote students. The participant had to give a three-slide three-minute presentation on quadratic equations. The participant received the lecture slides beforehand, i.e., when they agreed to serve as participant, and when they were told that they will have to give a three-minute presentation using the slides provided. The participant had to monitor both the local and the remote students, and interrupt their lecture when they noticed a student raising their hand or placing their head on the desk. After acknowledging the student's action the participant continued the lecture.
Conditions. We used a within-subject controlled-study design, with each participant performing the task in each of three conditions. In a first, control condition (CCZ), the instructor used a laptop to see the video sprites of the remote students as well as the lecture slides, simulating a videoconferencing setup (e.g., Zoom). In a first experimental condition (ECB), the instructor wore the X R headset that provided the HyperXRC Banners visualization of the hybrid classroom. In a second experimental condition (ECS), the instructor wore the X R headset that provided the HyperXRC Seats visualization.
Experimental procedure. The experiment was held in a lecture room with 14 local student actors and with 15 remote students rendered with video sprites. The actors were not participants, i.e., they only impersonated local students and performed the hand raising and placing head on desk actions, to allow testing the instructor participants. The local student actors were given scripts that indicated when to perform what action, based on the slide of the presentation, e.g., "raise your hand when you see slide 2". The video sprites of the remote students were prerecorded which provided identical conditions for all instructor participants. For each three minute lecture, there were four hand raising and two head on desk actions, performed by different students: two hand raising and one head on desk by remote students, and two hand raising and one head on desk by local student actors. Both types of actions lasted for five seconds.
A consenting participant first filled out a pen-and-paper demographics questionnaire. Then it was asked to put on the X R headset and was given verbal instructions by the experimenter regarding the task. The participant was instructed to use the virtual laser pointer and to click the trigger button to indicate the remote student that raised their hand or put their head on the desk. The participant was instructed to indicate verbally when they noticed a local student actor raising their hand or placing their head on the desk, and the experimenter recorded the event. The participant gave the lecture three times, once in each of the three conditions, in counter-balanced order. After each lecture, the participant was asked to remove the headset and to answer a pen-and-paper subjective experience questionnaire.
A participant was involved in the experiment for a total of 20 minutes. Each participant was remunerated with a $15 gift card. The local student actors were remunerated with gift cards at the rate of $15 per hour. The experiment was conducted in three sessions, each on a different day, for a total of five hours combined. Data collection. We have collected data according to objective and subjective metrics. We have collected the number of actions noticed by a participant serving in an instructor role, with an integer value between 0 and 6. We have also recorded the type of action, i.e., hand raising or placing head on desk, and whether the acting student was local or remote. We have also collected the instructor participant's opinion on each of the three modalities of delivering the hybrid lecture, through a subjective experience questionnaire with five questions, each scored on a five-point Likert scale (i.e., "strongly disagree", "disagree", "neither agree nor disagree", "agree", "strongly agree"). The five questions are: Q1: "The remote students were hard to see"; Q2: "The local students were hard to see"; Q3: "The PowerPoint slides were easy to see while at the same time keeping track of the students"; Q4: "This was a good way of teaching a hybrid lecture for both local and remote students"; Q5: "Wearing the headset was uncomfortable". Some questions are positive and some are negative to avoid mechanical answers.
Data analysis. For the data analysis, the numerical score of negative questions was flipped, i.e., 6-x instead of x, such that a higher score always means better. We analyzed the data with box plots and investigated the statistical significance of differences between conditions. Our within-subjects design with three conditions produced three related samples of data. The data violated the distribution normality assumption, so we compared the three conditions using the Friedman non-parametric test, with a statistical significance threshold of 0.05. When the three way comparison indicated significant differences, we performed a posthoc pairwise comparison with a Bonferroni significance level adjustment of x3 to account for the number of comparisons (i.e., three unsorted pairs of conditions).
We present and discuss results for objective metrics first, followed by subjective metrics.
We have investigated the rates at which instructor participants notice student actions, all together, as well as separately for hand raising and head down, and for local and remote students.
All students together, each action separately. We examined the average rates at which the instructor participants detected student actions, for local and remote students counted together (Fig. 4). In the experimental conditions, participants noticed on average 77% (ECB) and 70% (ECS) of the hand-raising actions, and only 55% in the control condition (CCZ). The Friedman test indicates that there was a difference between the three conditions, χ 2 (2) = 6.780, p = 0.034. The pairwise comparison (Tab. 1) shows that no pairwise difference is significant at the Bonferroni conservatively adjusted level, which is due to insufficient statistical power of our sample size. Indeed, the Cohen's d effect size is d = 0.794, which, for our significance level of α = 0.05, implies a power of 0.60. In terms of the headdown action, the average detection rates are 30% for C C Z , 23% for ECB, and 43% for ECS. This indicates that the head-down action is harder to detect than hand raising, which is more salient by design, i.e., hand raising is a gesture designed to attract the attention of the instructor. The differences between conditions are not significant (Friedman χ 2 (2) = 3.161, p = 0.206), but the averages indicate that displaying remote students on banners might make it particularly hard for the instructor to notice when a student places their head on the desk, becoming disengaged.
Local and remote students separately, all actions together. Fig. 5 shows that when considering both types of actions together, the experimental conditions do not worsen the detection rate of local actions compared to the control condition (57% C C Z , 60% E C B , and 62% ECS). Furthermore, the experimental conditions have an advantage over the control condition (36% C C Z , 58% E C B , and 60% ECS ) in terms of detecting remote actions. The advantage is significant (Friedman χ 2 (2) = 9.000, p = 0.011), although the effect size is small, i.e., d = 0.672. The pairwise comparisons are given in Tab. 2.
Local and remote students separately, each action separately. We examined the instructor participant's monitoring of local and re- mote students separately, for each type of action (Fig. 6). The local students are seen the same way by instructor participants in all conditions, but having to monitor the remote students might interfere with the instructor's ability to monitor the local students. Instructor participants noticed the local students raising their hands at 83% for both experimental conditions, and only at 73% for the control condition. The differences are not significant (Friedman χ 2 (2) = 0.700, p = 0.705), and, either way, the means are higher for the experimental conditions, so we conclude that wearing the X R headset does not reduce the instructor's ability to monitor the local students. Regarding remote students, instructor participants noticed on average only 37% of their hand raises in the video conferencing condition (CCZ), and 70% and 57% for the experimental conditions E C B and ECS. As expected, monitoring the remote students is more difficult with video conferencing (CCZ), since the remote students are not included in the default field of view of the instructor. E C B has an advantage over E CS , which we attribute to the unoccluded visualization of the remote students provided by the banners. The differences are significant (Friedman χ 2 (2) = 8.486, p = 0.014), and the posthoc analysis (Tab. 3) shows that ECB's advantage over C C Z approaches significance (p = 0.067, d = 0.820).
Regarding the head-down student action there were no significant differences between conditions for the local students (Friedman χ 2 (2) = 1.000, p = 0.607), and the large p value indicates that the conditions are equivalent, i.e., wearing the headset does not lower the instructor's ability to notice when local students place their head Table 3: Posthoc pairwise comparisons of hand-raising detection rates between the three conditions, for remote students only.
Table 5: Posthoc pairwise comparisons between the three conditions for Q3: "slides easy to see".
Figure 7: Box plots of the answers for the instructor preference questionnaire, for each of the three conditions. The scores of negative questions are flipped for higher to always mean better: Q1-remote students (not) hard to see, Q2-local students (not) hard to see, Q3slides easy to see, Q4-good way to teach hybrid lecture, Q5-wearing the headset (not) uncomfortable. on the desk. For remote students, E C S showed twice the headdown detection rate compared to E C B and CCZ , with the difference between the three conditions approaching significance (Friedman χ 2 (2) = 5.556, p = 0.062). We expected the better performance of E C S compared to C C Z , as the instructor cannot easily monitor remote students on their laptop. Surprisingly, the detection rate for E C S was significantly higher than for E C B . We explain this based on the fact that E C B places the remote students at the periphery of the field of view of the instructor, where head-down actions are harder to detect, whereas E C S places the remote students at a more central location in the instructor's field of view. The instructor pans their view direction left-right but is less likely to tilt it up-down to look specifically at the remote students displayed on banners.
We are relying on remote student actions that are visually salient. The results show that videoconferencing struggles even with these salient actions, so it is reasonable to assume that it will struggle even more with more subtle actions. Of course, although HyperXRC performs well on these salient actions, extrapolating this conclusion to more subtle actions requires additional studies. Furthermore, hand raising is a common action of students not only to interject, but also, for example, to answer a multiple-choice question posed by the instructor during lecture. Such quick polls have been shown to be beneficial for engaging conventional classrooms, and they are incompatible with videoconferencing [17]. We note that our study indicates the inadequacy of videoconferencing when it comes to supporting a hybrid classroom, and not a completely online lecture. Indeed, videoconferencing has a series of mechanisms that allow remote students to draw the instructor's attention, such as hand raising icons displayed in the corner of the video feed of the interjecting remote student. However, in a hybrid classroom scenario the instructor cannot pause to examine the matrix of video feeds in detail as this comes at the expense of disconnecting from the local students.
Subjective metrics: instructor preferences Q2. Instructor participants found that the local students are easy to see, with no significant differences between the three conditions (Friedman χ 2 (2) = 0.341, p = 0.843). The small average difference is in favor of the experimental conditions. This shows that the instructor participants did not find that seeing local students is harder when wearing the X R headset, than when not.
Q3. Instructor participants found it easier to see the lecture slides in the experimental conditions than in the control condition (4.13 for E C B and 4.00 for E CS , vs. 3.06 for CCZ). The difference is significant (Friedman χ 2 (2) = 7.400, p = 0.025, d = 0.951), but not in terms of pairwise comparisons (Tab. 5).
Q4. Instructor participants disliked the videoconferencing option for teaching a hybrid class (2.33 for C C Z , which is less than the neutral score of 2.5). Instructor participants rated favorably both experimental conditions (3.87 for E C B and 4.13 for ECS). The differences between the three conditions were significant (Friedman χ 2 (2) = 18.653, p < 0.001), and each of E C B and E C S had a significant advantage over C C Z (Tab. 6).The effect sizes for E C B vs. C C Z and for E C S vs. C C Z are d = 1.621 and d = 2.306, respectively, i.e. "very large"/"huge" and "huge", so the 15 instructor participants are sufficient for excellent statistical power of 0.95.
Q5. Instructor participants were neutral to mildly positive regarding the comfort of wearing the X R headset (2.93 for E C B and 2.87 for E CS ) during the short three-minute lectures of our study. Although the form factor of X R headsets has improved, wearing the headset for extended periods of time, which our study did not test, might remain a problem.
We describe the methods and then present and discuss the results.
Participants. We evaluated the HyperXRC design from the local students' point of view with N = 16 participants, recruited from the undergraduate and graduate student population of our university. Table 6: Posthoc pairwise comparisons between conditions for Q4: "good way to teach hybrid lecture". Both HyperXRC conditions E C B and E C S had a significant advantage over the videoconferencing condition C C Z . negative questions was flipped, i.e., 6-x instead of x. We analyzed the data with box plots and we investigated the statistical significance of differences between conditions. The data of our within-subjects design with two conditions generated two samples. The data violated the distribution normality assumption, so we compared the two conditions using the Friedman non-parametric test, with a statistical significance threshold of 0.05.
Figure 8: Box plots of the answers for the local student preference questionnaire. The scores of negative questions are flipped for higher to always mean better: Q1-The instructor (did not) seem to look at the ceiling all the time, Q2-The instructor (did not) consistently direct their gaze straight ahead and did look at the students, Q3-The instructor made good eye contact, Q4-It was (not) hard to pay attention to the instructor, Q5-It was (not) frustrating to not see the remote students addressed by the instructor, Q6-I wouldn't mind attending a class taught this way. 6 participants had an age between 18 and 22, 9 between 23 and 29, and 1 had an age of over 30. 11 participants self-identified as male and 5 as female. 14 participants had video conferences constantly, and 2 had less experience. Regarding V R experience, 2 participants indicated that they had never used V R before, 3 had used V R once, 9 a few times, and 2 frequently. Regarding A R experience, 4 participants indicated they had none, 6 had used A R once before, 5 had used A R a few times before, and 1 had used A R frequently.
Task. Participants served as local students, watched an instructor's presentation on quadratic equations, after which they answered a questionnaire on the presentation they had watched.
Conditions. We used a within-subject controlled-study design, with participants watching the same quadratic equation presentation given by an instructor twice, once in the videoconferencing control condition C C Z , and once in the HyperXRC Seats condition, ECS.
Experimental procedure. The experiment was held in a lecture room where all 16 local student participants were seated together. Like for experiment 1, there were 15 remote students, rendered with pre-captured video sprites. The instructor could see the remote students either on their laptop (CCZ) or using the X R headset (ECS). The local student participants could not see the remote students. The instructor answered two questions from remote students. A consenting participant first filled out a pen-and-paper demographics questionnaire. After each lecture, the participant was asked to answer a pen-and-paper subjective experience questionnaire. A participant was involved in the experiment for a total of 15 minutes. Each participant was remunerated with a $15 gift card. The experiment was conducted in a single session, with all 16 participants together.
Data collection. We collected the student participant's opinion on each of the two modalities of delivering the hybrid lecture, through a subjective experience questionnaire with six questions, each scored on a five-point Likert scale. The six questions are: Q1: "The instructor seemed to look at the ceiling all the time"; Q2: "The instructor appeared to consistently direct their gaze straight ahead and not at the students"; Q3: "The instructor made good eye contact"; Q4: "It was hard to pay attention to the instructor"; Q5: "It was frustrating to not see the remote students addressed by the instructor"; Q6: "I wouldn't mind attending a class taught this way". Some questions are positive and some are negative to avoid mechanical answers.
Data analysis. For the data analysis, the numerical score of Fig. 8 shows the average scores for the six questions of the student experience questionnaire. Q1. The first question investigates whether the display of the lecture slides as a virtual banner hanging off the ceiling interfered with the instructor / student eye contact. Student participants found that the instructor seemed to look at the ceiling more than in the control condition, and the difference was significant (Friedman χ 2 (2) = 12.000, p < 0.001).
Q2. Student participants found that the instructor seemed more likely to direct their gaze straight ahead and not at the students when delivering the lecture with the headset. The difference between the two conditions is significant (Friedman χ 2 (2) = 14.000, p < 0.001), in favor of C C Z over E CS . This indicates that wearing the headset interferes with the instructor's ability to monitor local students.
Q3. The third question asks local student participants directly about the quality of the eye contact made by the instructor. The difference between the two conditions is significant (Friedman χ 2 (2) = 13.000, p < 0.001), confirming the eye contact loss when wearing the headset.
Q4. Students found that it was harder to pay attention to the instructor when the instructor wore the headset. The advantage of C C Z over E C S is significant (Friedman χ 2 (2) = 10.286, p = 0.001).
Q5. Surprisingly, the local student participants did not mind too much not seeing the remote students (average scores of 3.44 and 3.46 out of 5). There were no significant differences between the two conditions (Friedman χ 2 (2) = 0.500, p = 0.480), which is not surprising as neither condition shows the remote students.
Q6. Student participants prefer the videoconferencing option for attending a hybrid class with both local and remote students (4.25 for C C Z vs 2.69 for ECS). The difference between the two conditions is significant (Friedman χ 2 (2) = 10.286, p = 0.001), which indicates that the X R headset is placing a barrier between the local students and the instructor.
Overall, based on the two experiments that examine the X R hybrid lecture design from the perspective of the instructor and of the local student participants, we conclude that the X R technology does strengthen the connection between the instructor and the remote students, but this comes at the cost of weakening the connection between the instructor and the local students.
We have investigated the design of a hybrid classroom that accommodates both local and remote students with the help of an X R headset worn by the instructor. We have investigated two designs, one in which the remote students are shown in the empty seats of the classroom (HyperXRC Seats), and one in which they are shown on virtual ceiling banners (HyperXRC Banners). We have conducted a controlled user study with two experiments that investigate the Hy-perXRC approach from the instructor perspective, as well as from the local students perspective, comparing it to conventional videoconferencing. Objective metrics show that the HyperXRC approaches help the instructor maintain awareness of the remote students better than the videoconferencing approach. Furthermore, the objective metrics show that the HyperXRC approaches afford the instructor similar if not better awareness of the local students, compared to videoconferencing, which indicates that the X R headset does not preclude the instructor from adequately monitoring the local students. Subjective metrics indicate that instructors prefer the HyperXRC approaches over conventional videoconferencing. This is unsurprising, given the well-known difficulty of keeping track of remote students displayed on a laptop while teaching. Our study found no differences between the two HyperXRC approaches, with the notable exception HyperXRC Seats affording the instructor higher detection rates of when remote students placed their head on their desk. This indicates that whereas in the HyperXRC Banners approach instructors can notice student actions that are purposefully designed to be conspicuous, such as hand raising, instructors might find it difficult to notice more subtle student actions and attitudes. This points to a potentially important advantage of the HyperXRC Seats approach, which might better allow instructors to maintain awareness of the level of engagement of remote students compared to HyperXRC Banners, advantage that should be further investigated in future work.
Our work has several limitations. As indicated by the effect sizes, our study did not always have sufficient statistical power. For example, the 15 instructor participants translate to 0.60 power for the differences in the rates of detection of remote student actions. Future studies can use the effect size of d = 0.794 determined by our study to run 34 participants to achieve a power of 0.90.
We have opted for a within-subjects design, which has the benefit of requiring fewer participants. Of course, a within-subjects design also brings the shortcoming of before and after effects and of participant fatigue. We kept the lectures to a minimum, i.e., three minutes, to mitigate fatigue concerns. Furthermore, participants were already familiar with the lecture topic and had received the slides in advance, which mitigates learning effects. The data of any within-subjects study can be analyzed as a between-subjects study by considering only the data for a participant's first condition. However, this comes at the cost of reducing the number of participants per condition by a factor equal to the number of conditions. This would leave only five participants per condition in our case, too low of a number for a conclusive analysis. Future work could conduct between-subjects studies to rule out fatigue and before and after effects.
Our study directly measured the instructor awareness of students by asking the instructor to indicate when they became aware of specific student actions. Future studies can also employ indirect metrics to quantify the instructor's awareness of the remote students such as those derived from eye and head tracking data. Furthermore, future studies could explore the instructor experience through task load, user experience, and even extended reality presence questionnaires.
Another limitation of our work is the assumption that the geometry of the local classroom is fixed. The assumption holds true for classrooms with desks and seats bolted to the floor, but the Hy-perXRC implementation will have to be enhanced with real-time classroom configuration acquisition for generality. Our current implementation uses pre-assigned remote students seats, but local students should be allowed to take any classroom seat, with HyperXRC detecting the remaining seats for the remote students. Our current implementation resolves visibility conservatively, by giving each local student a generous bounding rectangle that cannot be overwritten by remote students seated behind. Fine-grain real-time classroom acquisition to identify local students at pixel-level would allow for a finer interleaving of real and remote students, avoiding any unnecessary clipping of remote student sprites. Our study investigates local and remote students seated in the first four rows of a large lecture hall. Future work should compare local and remote students seated at the back of the classroom. As these students are hard to monitor even "in-person", X R technology could provide a solution for increasing the instructor's awareness of these local students, an issue orthogonal to distance education. Another limitation is that our study relied on 3-minute lectures; future evaluations should occur in authentic lecture settings with full length sessions, and even in multiple sessions through semester-long longitudinal studies.
Our study reveals that local students noticed the lack of eye contact with the instructor wearing the X R headset. Future work should investigate restoring the connection to the instructor felt by the local students, as accommodating remote students cannot compromise the educational experience of those on-site. One approach that we have already discussed in Sec. 3.1 is to use an audience facing projector to integrate the remote students into the local classroom, so the instructor can see the remote students without wearing a headset. Another approach is to display the instructor's first-person view of the hybrid classroom, stabilized, on a classroom screen for local students. A third option is to rely on headsets that display the user's face for real-world interlocutors to see, like the "snorkeling goggles effect" of Apple's upcoming Vision Pro headset [1]. The Vision Pro price tag of $3,499, which will continue to motivate research to find cost effective solutions for rendering the user' eyes.
Our work relies on remote student actions with high visual salience, and shows that videoconferencing struggles even with such actions. The remote student placing their head on their desk is an extreme form of disengagement, and future studies should exam-ine more subtle forms of disengagement, such as looking at phones or laptops, or breaking eye contact with the instructor. Such future studies will only be possible when the remote student acquisition and rendering will be able to convey eye contact, and these studies should assess the disengagement rate for local students, providing a baseline for accurate evaluation of the distance learning setup.
We have begun exploring the use of extended reality technology to enable hybrid classrooms. A fundamental benefit of X R is breaking free from the constraint of limited and fixed display surfaces. We have investigated two locations for displaying remote students, one where X R mimics the real world, and one where X R goes beyond what can be done in the real world. Future work should examine other possible locations, for example to the sides of the classroom, to identify the most promising presently underutilized region of the instructor's field of view. Another possible configuration is to place remote students in between the instructor and the first row of students, upgrading them remote students to the best classroom seats, without the logistical concerns of a real world classroom with no space between the instructor and the students. Finally, future work could also experiment with alternative ways of displaying the instructor slides, which compete with the remote students in the HyperXRC Banners implementation.
Our work assumes that remote students use conventional videoconferencing, and focuses on investigating the hybrid classroom design from the perspectives of the instructor and of the local students. Future work should investigate and evaluate X R distance education designs for the remote student side. Important questions are how to convey to the remote students that they are part of the classroom and are visible to the instructor, and how to allow remote students to interact with local as well as other remote students. Our work investigates the distance education scenario of individual remote students. Other scenarios should be investigated as well, such as the scenario of groups of co-located remote students, the scenario of a remote instructor, and scenarios corresponding to other oncampus learning activities, beyond remote lecture attendance, such as remote study group attendance. The goal is to fill the great and growing societal need for an effective, robust, flexible, and equitable distance education system.
This material is based upon work supported by the National Science Foundation under Grants No. 2212200 and 2318657.
Authorized licensed use limited to: Purdue University. Downloaded on May 15,2024 at 14:32:09 UTC from IEEE Xplore. Restrictions apply.