Immersive technologies ranging from mobile technology to massive online gaming are gaining a foothold as learning tools for a variety of subjects in teacher education and teacher professional development. For instance, virtual reality (including 360 videos) has been shown to improve situational awareness, understanding (Roche & Gal-Petitfaux, 2017) and critical thinking (Cross, Wolfenden, Adinolfi, 2018) across various professions (Harrington et al., 2017;Skupin et al., 2017).
The increasing availability of 360 videos and stand-alone headset displays has specifically led to the recent adoption of such technology in preservice teacher education (Kosko et al., 2019;Walshe & Driver, 2019). Standard video of classroom instruction has been widely adopted in teacher education for improving reflection and professional vision among future educators. However, 360 video may allow for improved reflection on professional practice due to an increased perceptual capacity of the medium (Kosko et al., 2019;Walshe & Driver, 2019). One means of assessing this capacity is through teachers' perceived sense of presence in the virtual environment. Unfortunately, most surveys assessing presence focus on created, not recorded, virtual environments. To our knowledge, no instrument has been developed to evaluate sense of presence in 360 videos, much less for the purpose of teacher education.
This paper addresses this gap by presenting the results of cognitive interviews (n = 15) assessing a presence scale adapted from the Multimodal Presence Scale for virtual reality environments (Makransky, Lilleholt, & Aaby, 2017). Subjects recruited were undergraduate preservice teachers from a Midwestern U.S. university who watched a 360 video of a math lesson. The findings point to important revisions to the original list of items, resulting in a better instrument for evaluating instructional presence in immersive videos for teacher education.
Virtual Reality (VR) can be defined as a technology mediated environment that tries to involve the user within an alternative setting (Ferdig, Gandolfi, & Immel;2018). We have recently seen an evolutionfrom common VR experienced through a screen (i.e., Desktop VR or DVR) to Immersive Virtual Reality (IVR), which tries to go beyond hearing and 2D sight. As observed by Lorenzo, Pomares and Lledó (2013), "compared to conventional VR environments, in IVR systems the student is completely surrounded by the environment and constantly receives stimuli and has the possibility to interact with it" (p. 89). Oculus Rift, HTC Vive, PlayStation VR and even Google Cardboard can serve as leading examples of such a turning point, which aims to engage users by addressing embodied interaction processes. IVR applications may cover the whole sight rotation (e.g., HMD systems) and reproduce vibrations (haptic media) and environmental sounds (3D sound).
This technology has been shown to facilitate emotional arousal, presence, and engagement in comparison with DVR (Kim et al., 2014;North and North 2016;Hupont et al., 2015). Among these factors, presence is particularly important because it addresses the core aspect of VR (i.e., the ability to grab users and make them feel immersed; Ferdig, Gandolfi, Immel, 2018). Indeed, presence refers to a sense of being there or what Mestre (2005) described as a "psychological state experienced as a consequence of focusing one's energy and attention on a coherent set of stimuli" (p. 2). From an educational perspective, IVR has been increasingly adopted in K-12 settings, higher education, and teacher professional development (Freina & Ott, 2015;Rodriguez, 2016;Randall et al., 2016). A pertinent example is the use of instructional videos for guiding preservice teachers. Videos have been widely used for training this population, ranging from helping reflection about personal and peer teaching performance (e.g., Blomberg et all, 2013: Lee & Wu, 2006) to introducing technology integration and expert lead (Ottenbreit-Leftwich et al., 2018;Weber et al., 2018); potential benefits have been observed for a variety of different subject areas (Blomberg et all, 2011).
The few studies examining 360 videos in teacher education have found that users do report a higher sense of perceived presence (Roche et al., 2017;Walshe & Driver, 2019). However, recent evidence suggests that existing presence scales may not distinguish sense of presence well between 360 video viewed on a flat screen and 360 video viewed on a VR headset (Ferdig & Kosko, in review). One potential reason for existing presence scales' lack of sensitivity regarding standard and 360 video viewing is that these scales are often designed with realism of graphics in mind. Another potential reason is that context of use may influence how presence is perceived by the user. Therefore, we sought to build from existing literature on studying virtual presence to develop an appropriate scale for viewing 360 video in the context of preservice teacher education.
We interviewed preservice teachers (sophomore) from different backgrounds enrolled in educational technology courses (n=15). The courses broadly addressed introductory learning theories and related technologies (e.g., learning management systems, educational video games, etc.); IVR and 360 videos were not among the course topics.
The Multimodal Presence Scale for virtual reality environments (MPS) served as an initial stimulus due to its broad scope toward presence in digital settings. The MPS was created and validated by Makransky, Lilleholt, & Aaby (2017) for measuring the definition of presence as physical, social, and self-presence suggested by Lee (2004) in virtual environments. The MPS psychometric properties were validated with a confirmatory factor analysis and item response theory (Makransky, Lilleholt, & Aaby, 2017).
From the MPS initial list of 21 items, we developed an alternative roster of 29 items (see Table 1 for examples of items); the Extended Reality Presence Scale (XRPS). Considering the constructs articulated by Makransky et al. (2017), new items were written considering the non-interactive nature of 360 videos, the specific learning goal (attending to pedagogy in a classroom lesson), and the audience targeted (preservice teachers). For example, in 360 videos, interaction is typically limited to selfmovement. Issues of realism are less meaningfully assessed than for an artificial virtual environment because the user is observing actual events. Therefore, the three presence sub-constructs were kept, while adjusting certain facets appropriately. For the physical presence, "the unawareness of physical mediation" and "physical realism" were removed because specifically tied to artificial environments; the social presence components were maintained due to the presence of individuals in the 360 video watched, while for the self-presence the "sense of self being in the virtual environment" was not considered due to its focus on virtual settings (Makransky, Lilleholt, & Aaby, 2017). Social presence I felt I was in a realistic educational setting. During the experience there were times where the computer interface seemed to disappear, and I felt like I was immersed in the 360-video environment I was able to look around and observe without perceiving the computer interface.
When something happened around my viewpoint, it felt like it was happening to my real body During the 360 video, I felt like my viewpoint and my real body became one and the same I felt like my viewpoint was an extension of my real body within the 360 video.
Cognitive interviewing is a scale development technique for preventing potential misinterpretations between participants and researchers (AERA et al., 2014;Dumas et al., 2008;Hughes, 2004). It relies on a twofold process where the subject answers each item prompt and then explains their choice to researchers, who can revise the items accordingly in case of misunderstanding and mismatch (Karabenick et al., 2007). For the current study, basic verbal probes (Peterson, Peterson & Powell, 2017) were preferred (i.e., what do you think the item means? What is your answer to the question and why?).
Cognitive interviews were directed in Fall 2019 for assessing the first version of XRPS. The participants recruited were undergraduate pre-service teachers from different disciplinary backgrounds (n=15). Participants used an Oculus Go to watch a five-minute 360 video of third grade students completing a math activity. Next, participants engaged in a one-on-one cognitive interview. Each interview lasted approximately 15-20 minutes. The research setting and procedure mimicked the one in which the final scale will be disseminated (i.e., after watching a training 360 video in a university/professional development environment), which is desirable for cognitive interview-based inquiries (Beatty & Willis, 2007).
Overall, findings from the cognitive interviews supported the item selection and revision. All but one of the items were interpreted as intended by the authors. The failed item stated 'I felt I was in a meaningful educational setting'; it was meant to assess social presence. However, most participants interpreted this as asking whether the lesson being taught in the video was meaningful. The item was, therefore, discarded and replaced with the negatively worded item The educational setting felt fake."
The remaining items on the scale were interpreted as intended by nearly all participants. However, participants pointed to certain terms or wordings that were unclear. It is important to note that participants still interpreted the item to convey the meaning that we conjectured them to have. Still, we elected to incorporate revisions where participants were explicit about confusing terms or phrasings. In sum, there were two terms to revise (i.e., mixed reality embodiment, computer interface), one item to reword ('I felt I was in a meaningful educational setting' was replaced with 'The educational setting felt fake'), and one item to remove due to its similarity with others (i.e., 'I felt to be emotionally attached to the persons and events during the 360 video'"). These modifications were also intended to facilitate the use of the scale for evaluating different IVR experiences that go beyond the current use of evaluating o 360 video.
It can be argued that this initial round of cognitive interviews supported the overall structure and items' words and selection of XRPS for use in teacher education. At the same time, participants' suggestions were important for refining the instrument, replacing unclear terms and rephrasing dense statements. Moreover, this study provided the authors an opportunity to reflect on the scale range of application, highlighting the need of broader concepts like technology. Nevertheless, cognitive interviews are just one of the first steps toward the validation of a new instrument. This method provides insightful information about wording and association with the topics and metrics that researchers intend to investigate. Next steps involve psychometrically examining the XRPS and its revised items with a larger sample.
In order to address this further step, a scale assessment is currently under development with the revised version of the instrument, and additional studies targeting alternative IVR settings and experiences are planned for testing and problematizing the XRPS range of application, from addressing specific content areas to targeting expert teachers and practitioners. In addition, this scale is going to be used with different types of 360 videos for professional training involving classroom settings, including different technological features, class environments, and teaching styles and conditions. Indeed, the objective of this article is to have provided a flexible instrument that can be used in a variety of mediated environments for learning for pre-service teachers and future educators at large.