skip to main content


Title: Virtual Reality as a Context for Adaptation
The COVID-19 pandemic has accelerated interest in virtual reality (VR) for education, entertainment, telerehabilitation, and skills training. As the frequency and duration of VR engagement increases—the number of people in the United States using VR at least once per month is forecasted to exceed 95 million—it is critical to understand how VR engagement influences brain and behavior. Here, we evaluate neurophysiological effects of sensory conflicts induced by VR engagement and posit an intriguing hypothesis: the brain processes VR as a unique “context” leading to the formation and maintenance of independent sensorimotor representations. We discuss known VR-induced sensorimotor adaptations to illustrate how VR might manifest as a context for learning and how technological and human factors might mediate the context-dependency of sensorimotor representations learned in VR.  more » « less
Award ID(s):
1804550 1935337
NSF-PAR ID:
10357673
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Frontiers in Virtual Reality
Volume:
2
ISSN:
2673-4192
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ABSTRACT

    Research from brain science and the learning sciences support the notion that human cognition is grounded in our sensorimotor engagement with the physical world and that processes of learning can be shaped by our movements and actions. Increasing recognition that effective educational interventions can be seeded with embodied actions is paralleled by recent and rapid advances in sensing and motion capture technologies. These advances allow for a new wave of cyberlearning environments that permit learners to use their bodies to create and manipulate digital representations of core ideas in a variety of learning domains. Drawing on the research literature on embodied cognition and design principles for creating effective embodied educational simulations, we present the design of a cyberlearning platform called ELASTIC3S. We describe the design rationale and preliminary data from a small empirical study of the ELASTIC3S platform that address the potential for enhancing science, technology, engineering, and mathematics (STEM) learning and engagement through embodied interaction.

     
    more » « less
  2. Speech processing is highly incremental. It is widely accepted that human listeners continuously use the linguistic context to anticipate upcoming concepts, words, and phonemes. However, previous evidence supports two seemingly contradictory models of how a predictive context is integrated with the bottom-up sensory input: Classic psycholinguistic paradigms suggest a two-stage process, in which acoustic input initially leads to local, context-independent representations, which are then quickly integrated with contextual constraints. This contrasts with the view that the brain constructs a single coherent, unified interpretation of the input, which fully integrates available information across representational hierarchies, and thus uses contextual constraints to modulate even the earliest sensory representations. To distinguish these hypotheses, we tested magnetoencephalography responses to continuous narrative speech for signatures of local and unified predictive models. Results provide evidence that listeners employ both types of models in parallel. Two local context models uniquely predict some part of early neural responses, one based on sublexical phoneme sequences, and one based on the phonemes in the current word alone; at the same time, even early responses to phonemes also reflect a unified model that incorporates sentence-level constraints to predict upcoming phonemes. Neural source localization places the anatomical origins of the different predictive models in nonidentical parts of the superior temporal lobes bilaterally, with the right hemisphere showing a relative preference for more local models. These results suggest that speech processing recruits both local and unified predictive models in parallel, reconciling previous disparate findings. Parallel models might make the perceptual system more robust, facilitate processing of unexpected inputs, and serve a function in language acquisition. MEG Data MEG data is in FIFF format and can be opened with MNE-Python. Data has been directly converted from the acquisition device native format without any preprocessing. Events contained in the data indicate the stimuli in numerical order. Subjects R2650 and R2652 heard stimulus 11b instead of 11. Predictor Variables The original audio files are copyrighted and cannot be shared, but the make_audio folder contains make_clips.py which can be used to extract the exact clips from the commercially available audiobook (ISBN 978-1480555280). The predictors directory contains all the predictors used in the original study as pickled eelbrain objects. They can be loaded in Python with the eelbrain.load.unpickle function. The TextGrids directory contains the TextGrids aligned to the audio files. Source Localization The localization.zip file contains files needed for source localization. Structural brain models used in the published analysis are reconstructed by scaling the FreeSurfer fsaverage brain (distributed with FreeSurfer) based on each subject's `MRI scaling parameters.cfg` file. This can be done using the `mne.scale_mri` function. Each subject's MEG folder contains a `subject-trans.fif` file which contains the coregistration between MEG sensor space and (scaled) MRI space, which is used to compute the forward solution. 
    more » « less
  3. null (Ed.)
    Combining functional magnetic resonance imaging (fMRI) with models of neuromotor adaptation is useful for identifying the function of different neuromotor control centers in the brain. Current models of neuromotor adaptation to force perturbations have been studied primarily in whole-arm reaching tasks that are ill-suited for MRI. We have previously developed the MR-SoftWrist, an fMRI-compatible wrist robot, to study motor control during wrist adaptation. Because the wrist joint has intrinsic dynamics dominated by stiffness, it is unclear if these models will apply to the wrist. Here, we characterize adaptation of the wrist to lateral forces to determine if established adaptation models are valid for wrist pointing. We recruited thirteen subjects to perform our task using the MR-SoftWrist. Our task included a clockwise (CW) - counterclockwise (CCW) - error clamp schedule and an alternating CW-CCW force field schedule. To determine applicability of previous models, we fit three candidate models - a single-state, two-state, and context dependent multi-state model - to behavioral data. Our results indicate that features of sensorimotor adaptation reported in the literature are present in the wrist, including spontaneous recovery, and anterograde and retrograde interference between the learning of two oppositely directed force fields. A two-state model best fit our behavioral data. Under this model, adaptation was dominated by a fast learning state with minor engagement of a slow learning state. Finally, all adaptation models tested showed a consistent over-estimation of performance error, suggesting that the control of the wrist relies not only on internal models but likely other mechanisms, like impedance control, to reject perturbations. 
    more » « less
  4. Speech processing is highly incremental. It is widely accepted that human listeners continuously use the linguistic context to anticipate upcoming concepts, words, and phonemes. However, previous evidence supports two seemingly contradictory models of how a predictive context is integrated with the bottom-up sensory input: Classic psycholinguistic paradigms suggest a two-stage process, in which acoustic input initially leads to local, context-independent representations, which are then quickly integrated with contextual constraints. This contrasts with the view that the brain constructs a single coherent, unified interpretation of the input, which fully integrates available information across representational hierarchies, and thus uses contextual constraints to modulate even the earliest sensory representations. To distinguish these hypotheses, we tested magnetoencephalography responses to continuous narrative speech for signatures of local and unified predictive models. Results provide evidence that listeners employ both types of models in parallel. Two local context models uniquely predict some part of early neural responses, one based on sublexical phoneme sequences, and one based on the phonemes in the current word alone; at the same time, even early responses to phonemes also reflect a unified model that incorporates sentence-level constraints to predict upcoming phonemes. Neural source localization places the anatomical origins of the different predictive models in nonidentical parts of the superior temporal lobes bilaterally, with the right hemisphere showing a relative preference for more local models. These results suggest that speech processing recruits both local and unified predictive models in parallel, reconciling previous disparate findings. Parallel models might make the perceptual system more robust, facilitate processing of unexpected inputs, and serve a function in language acquisition. 
    more » « less
  5. Abstract Background

    This paper contributes to current discussions about supporting prospective teachers (PSTs) in developing skills of noticing students’ mathematical thinking. We draw attention to PSTs’ initial noticing skills (prior to instruction focused on supporting noticing) as PSTs engage in analyzing written artifacts of student work and video-records. We examined and compared PSTs’ noticing skills as they analyzed how students reason about, generalize, and justify generalizations of figural patterns given student written work and video records. We identified aspects of student thinking about generalizations and justifications, which PSTs addressed and interpreted. We also examined how PSTs respond to students as they analyze student thinking given written artifacts of student work or video-records of small group discussions, and we identified the foci of PSTs’ responding practice.

    Results

    Our data revealed that PSTs’ initial noticing skills of student generalizations and justifications differed while accounting for ways in which student thinking was externalized (written work or video-records). PSTs’ attending-and-interpreting and their responding practices were focused on mathematically significant aspects of student thinking to a greater extent in the context of analyzing written artifacts compared to video records. While analyzing students’ written work, PSTs demonstrated greater attention to ways in which students analyzed patterns, students’ generalization strategies, and justifications linked to an understanding of the pattern structure, compared to analyzing student thinking captured via videos.

    Conclusion

    Our results document that without providing any intentional support for PSTs’ noticing skills, PSTs are more deliberate to focus on mathematically significant aspects of student thinking while analyzing written artifacts of student work compared to video-records. We believe that the analysis of student written work might demand from PSTs to be more analytical. While examining written representations, PSTs have to reconstruct students’ reasoning. Unlike the videos where the students tell or use gestures to express their thinking, written work provides fewer clues about student thinking. Thus, written work demands a deeper level of engagement from PSTs as they strive to understand student reasoning. Our study extends research on PSTs’ noticing skills by documenting differences in PSTs’ noticing in relation to the nature of artifacts of student work that PSTs analyze. Our work also adds to prior research on PSTs’ noticing by characterizing specific aspects of students’ thinking about pattern generalizations and justifications that PSTs address as they analyze student thinking and respond to students.

     
    more » « less