Title: Linking global top-down views to first-person views in the brain
Humans and other animals have a remarkable capacity to translate their position from one spatial frame of reference to another. The ability to seamlessly move between top-down and first-person views is important for navigation, memory formation, and other cognitive tasks. Evidence suggests that the medial temporal lobe and other cortical regions contribute to this function. To understand how a neural system might carry out these computations, we used variational autoencoders (VAEs) to reconstruct the first-person view from the top-down view of a robot simulation, and vice versa. Many latent variables in the VAEs had similar responses to those seen in neuron recordings, including location-specific activity, head direction tuning, and encoding of distance to local objects. Place-specific responses were prominent when reconstructing a first-person view from a top-down view, but head direction–specific responses were prominent when reconstructing a top-down view from a first-person view. In both cases, the model could recover from perturbations without retraining, but rather through remapping. These results could advance our understanding of how brain regions support viewpoint linkages and transformations.  more » « less
Award ID(s):
2024633
PAR ID:
10465604
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Proceedings of the National Academy of Sciences
Volume:
119
Issue:
45
ISSN:
0027-8424
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Many attachments to a scanning electron microscope (SEM), such as energy dispersive x‐ray spectroscopy, extend its function significantly. Typically, the application of such attachments requires that the specimen has a planar surface at a specific orientation. It is a challenge to make the plane of a microscale specimen satisfy the orientation requirement since they are visible only in an SEM. An in‐situ procedure is needed to adjust specimen orientation by using stage rotation and tilting functions, in the process of which the key is to determine the initial orientation. This study proposed and tested top‐down and side‐view approaches to determine the orientation of a planar surface inside an SEM. In the top‐down one, the projected area is monitored on SEM images as stage rotation and tilt angles are adjusted. When the surface normal is along the electron beam direction, the area has a maximum value. In the side‐view approach, the stage is adjusted so that the projection appears to be a straight horizontal line on the SEM image. Once the orientation of the specimen for top‐down or side‐view observation is determined, the original can be calculated, and a desired orientation can be realized by manipulating the stage. The procedures have been tested by analyzing planar surfaces of spherical particles in Al‐Cu‐Fe alloy in the form of facets. The measured angles between two surfaces are consistent with those expected from crystallographic consideration within 2.7° and 1.7° for the top‐down and side‐view approaches, respectively. Research HighlightsTop‐down and side‐view approaches have been proposed and tested for in‐situ determination of specimen planar surface orientation in a Scanning Electron Microscope.The measured angles between two surfaces are consistent with those expected from crystallographic consideration within 2.7° and 1.7° for the top‐down and side‐view approaches, respectively. 
    more » « less
  2. A common approach to interpreting spiking activity is based on identifying the firing fields---regions in physical or configuration spaces that elicit responses of neurons. Common examples include hippocampal place cells that fire at preferred locations in the navigated environment, head direction cells that fire at preferred orientations of the animal's head, view cells that respond to preferred spots in the visual field, etc. In all these cases, firing fields were discovered empirically, by trial and error. We argue that the existence and a number of properties of the firing fields can be established theoretically, through topological analyses of the neuronal spiking activity. In particular, we use Leray criterion powered by persistent homology theory, Eckhoff conditions and Region Connection Calculus to verify consistency of neuronal responses with a single coherent representation of space. 
    more » « less
  3. null (Ed.)
    Third-person is a popular perspective for video games, but virtual reality (VR) seems to be primarily experienced from a first-person point of view (POV). While a first-person POV generally offers the highest presence; a third-person POV allows users to see their avatar; which allows for a better bond, and the higher vantage point generally increases spatial awareness and navigation. Third-person locomotion is generally implemented using a controller or keyboard, with users often sitting down; an approach that is considered to offer a low presence and embodiment. We present a novel thirdperson locomotion method that enables a high avatar embodiment by integrating skeletal tracking with head-tilt based input to enable omnidirectional navigation beyond the confines of available tracking space. By interpreting movement relative to an avatar, the user will always keep facing the camera which optimizes skeletal tracking and keeps required instrumentation minimal (1 depth camera). A user study compares the performance, usability, VR sickness incidence and avatar embodiment of our method to using a controller for a navigation task that involves interacting with objects. Though a controller offers a higher performance and usability, our locomotion method offered a significantly higher avatar embodiment. 
    more » « less
  4. Our daily interactions with the world are shaped by sensory expectations informed by context and prior experiences, which in turn influence how we allocate our attention. Prominent predictive coding models suggest that sensory expectancy and attention interact but disagree on the precise mechanisms. One possibility is that the ventral attention network (VAN) may play a role by facilitating attentional reorienting when expectancy is violated. To test this in humans (23 males and 43 females), we employed an auditory-visual trial-by-trial cueing paradigm in three experiments integrating EEG and fMRI to investigate the VAN’s role in violations of cross-modal expectancy. Behavioral results showed faster responses to expected targets, confirming the efficacy of cue-induced expectations in orienting attention to the expected target modality. EEG analyses revealed differences in early (∼100 ms latency) event-related potentials (ERPs) to both auditory and visual stimuli when expectations were violated. Unexpected stimuli elicited significantly larger early-latency negative ERPs, across both modalities. Source localization of these ERPs and subsequent fMRI evidence revealed activation in the right VAN. Functional connectivity analyses showed greater coupling between VAN regions and sensory cortices, with modality-specific pathways involving superior temporal gyrus for auditory and fusiform gyrus for visual targets. These findings demonstrate that expectancy violations recruit the VAN to reorient attention and resolve sensory conflict. By coordinating top-down control and bottom-up sensory input, the VAN supports adaptive responses to unexpected stimuli. This work advances our understanding of predictive processing in multisensory perception and highlights the VAN’s central role in flexible cognitive control. 
    more » « less
  5. Researchers have adopted remote methods, such as online surveys and video conferencing, to overcome challenges in conducting in-person usability testing, such as participation, user representation, and safety. However, remote user evaluation on hardware testbeds is limited, especially for blind participants, as such methods restrict access to observations of user interactions. We employ smart glasses in usability testing with blind people and share our lessons from a case study conducted in blind participants’ homes (N=12), where the experimenter can access participants’ activities via dual video conferencing: a third-person view via a laptop camera and a first-person view via smart glasses worn by the participant. We show that smart glasses hold potential for observing participants’ interactions with smartphone testbeds remotely; on average 58.7% of the interactions were fully captured via the first-person view compared to 3.7% via the third-person. However, this gain is not uniform across participants as it is susceptible to head movements orienting the ear towards a sound source, which highlights the need for a more inclusive camera form factor. We also share our lessons learned when it comes to dealing with lack of screen readers, a rapidly draining battery, and Internet connectivity in remote studies with blind participants. 
    more » « less