More Like this

1. Situating presence within extended reality for teacher training: Validation of the eXtended Reality Presence Scale (XRPS) in preservice teacher use of immersive 360 video

   Gandolfi, E. (January 2021, British journal of educational technology)

   null (Ed.)

   The use of video is commonplace for professional preparation in education and other fields. Research has provided evidence that the use of video in these contexts can lead to increased noticing and reflection. However, educators now have access to evolving forms of video such as 360 video. The purpose of this study was to adapt and validate an instrument for assessing immersive 360 video use in an undergraduate preservice teacher university training program. Data provided evidence of the validity of the Extended Reality Presence Scale (XRPS) for 360 video research in preservice teacher professional development. Moreover, evidence from the study suggests that those with higher feelings of presence are less likely to jump around (or twitch) while watching 360 videos. The main implications are that: a) the XRPS is a validated and reliable instrument and b) more research is needed to examine the presence and practices for in-service and preservice teachers while watching 360 video. 

   more » « less
2. Integrating 360 media in teaching and teacher education

   Kosko, K. W.; Roche, L.; Ferdig, R E; Gandolfi, E; Kratcoski, A (January 2021, What teacher educators should have learned from 2020)

   null (Ed.)

   One of the most disruptive aspects of 2020 for teacher education, mainly due to COVID, was the loss of field placements for future teachers. Teacher educators attempted to respond to this gap with videos of exemplary practice—something used commonly in teacher education to supplement such field experiences. Teacher educators, however, should have learned about the potential and promise for the use of 360 video for teaching and teacher education. This chapter highlights the research behind the use of 360, also showcasing how it has been used successfully in mathematics teacher education and physical education teacher education. The chapter includes evidence supporting the use of 360 as a dissemination technique and a technology skill needed to be taught to current and future teachers. Finally, evidence is provided to suggest that the use of 360 should be continued even when field placements return fully face-to-face. 

   more » « less
3. Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

   Chen, Changan; Peng, Puyuan; Baid, Ami; Xue, Zihui; Hsu, Wei-Ning; Harwath, David; Grauman, Kristen (July 2024, https://doi.org/10.48550/arXiv.2406.09272)

   Generating realistic audio for human actions is critical for applications such as film sound effects and virtual reality games. Existing methods assume complete correspondence between video and audio during training, but in real-world settings, many sounds occur off-screen or weakly correspond to visuals, leading to uncontrolled ambient sounds or hallucinations at test time. This paper introduces AV-LDM, a novel ambient-aware audio generation model that disentangles foreground action sounds from ambient background noise in in-the-wild training videos. The approach leverages a retrieval-augmented generation framework to synthesize audio that aligns both semantically and temporally with the visual input. Trained and evaluated on Ego4D and EPIC-KITCHENS datasets, along with the newly introduced Ego4D-Sounds dataset (1.2M curated clips with action-audio correspondence), the model outperforms prior methods, enables controllable ambient sound generation, and shows promise for generalization to synthetic video game clips. This work is the first to emphasize faithful video-to-audio generation focused on observed visual content despite noisy, uncurated training data. 

   more » « less
4. Learning from Silence and Noise for Visual Sound Source Localization

   Juanola, Xavier; Morais, Giovanal; Fuentes, Magdalena; Haro, Gloria (November 2025, BMVA - the British Machine Vision Association.)

   Visual sound source localization is a fundamental perception task that aims to detect the location of sounding sources in a video given its audio. Despite recent progress, we identify two shortcomings in current methods: 1) most approaches perform poorly in cases with low audio-visual semantic correspondence such as silence, noise, and offscreen sounds, i.e. in the presence of negative audio; and 2) most prior evaluations are limited to positive cases, where both datasets and metrics convey scenarios with a single visible sound source in the scene. To address this, we introduce three key contributions. First, we propose a new training strategy that incorporates silence and noise, which improves performance in positive cases, while being more robust against negative sounds. Our resulting self-supervised model, SSL-SaN, achieves state-of-the-art performance compared to other self-supervised models, both in sound localization and cross-modal retrieval. Second, we propose a new metric that quantifies the trade-off between alignment and separability of auditory and visual features across positive and negative audio-visual pairs. Third, we present IS3+, an extended and improved version of the IS3 synthetic dataset with negative audio. Our data, metrics and code are available on the Project page. 

   more » « less
5. Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey

   https://doi.org/10.1109/ACCESS.2024.3391817  

   Shahabaz, Ahmed; Sarkar, Sudeep (January 2024, IEEE Access)

   The joint analysis of audio and video is a powerful tool that can be applied to various contexts, including action, speech, and sound recognition, audio-visual video parsing, emotion recognition in affective computing, and self-supervised training of deep learning models. Solving these problems often involves tackling core audio-visual tasks, such as audio-visual source localization, audio-visual correspondence, and audio-visual source separation, which can be combined in various ways to achieve the desired results. This paper provides a review of the literature in this area, discussing the advancements, history, and datasets of audio-visual learning methods for various application domains. It also presents an overview of the reported performances on standard datasets and suggests promising directions for future research. 

   more » « less