skip to main content


Title: Effect of Image Captioning with Description on the Working Memory
Working memory plays an important role in human activities across academic, professional, and social settings. Working memory is defined as the memory extensively involved in goal-directed behaviors in which information must be retained and manipulated to ensure successful task execution. The aim of this research is to understand the effect of image captioning with image description on an individual’s working memory. A study was conducted with eight neutral images comprising situations relatable to daily life such that each image could have a positive or negative description associated with the outcome of the situation in the image. The study consisted of three rounds where the first and second round involved two parts and the third round consisted of one part. The image was captioned a total of five times across the entire study. The findings highlighted that only 25% of participants were able to recall the captions which they captioned for an image after a span of 9–15 days; when comparing the recall rate of the captions, 50% of participants were able to recall the image caption from the previous round in the present round; and out of the positive and negative description associated with the image, 65% of participants recalled the former description rather than the latter.  more » « less
Award ID(s):
1828010
NSF-PAR ID:
10344385
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
International Conference on Human-Computer Interaction (HCII) 2022. Lecture Notes in Computer Science
Volume:
13096
Page Range / eLocation ID:
107–120
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Few VR applications and games implement captioning of speech and audio cues, which either inhibits or prevents access of their application by deaf or hard of hearing (DHH) users, new language learners, and other caption users. Additionally, little to no guidelines exist on how to implement live captioning on VR headsets and how it may differ from traditional television captioning. To help fill the void of information behind user preferences of different VR captioning styles, we conducted a study with eight DHH participants to test three caption movement behaviors (head-locked, lag, and appear- locked) while watching live-captioned, single-speaker presentations in VR. Participants answered a series of Likert scale and open-ended questions about their experience. Participants’ preferences were split, but most participants reported feeling comfortable with using live captions in VR and enjoyed the experience. When participants ranked the caption behaviors, there was almost an equal divide between the three types tested. IPQ results indicated each behavior had similar immersion ratings, however participants found head-locked and lag captions more user-friendly than appear-locked captions. We suggest that participants may vary in caption preference depending on how they use captions, and that providing opportunities for caption customization is best. 
    more » « less
  2. Few VR applications and games implement captioning of speech and audio cues, which either inhibits or prevents access of their application by deaf or hard of hearing (DHH) users, new language learners, and other caption users. Additionally, little to no guidelines exist on how to implement live captioning on VR headsets and how it may differ from traditional television captioning. To help fill the void of information behind user preferences of different VR captioning styles, we conducted a study with eight DHH participants to test three caption movement behaviors (head-locked, lag, and appearlocked) while watching live-captioned, single-speaker presentations in VR. Participants answered a series of Likert scale and open-ended questions about their experience. Participants’ preferences were split, but most participants reported feeling comfortable with using live captions in VR and enjoyed the experience. When participants ranked the caption behaviors, there was almost an equal divide between the three types tested. IPQ results indicated each behavior had similar immersion ratings, however participants found head-locked and lag captions more user-friendly than appear-locked captions. We suggest that participants may vary in caption preference depending on how they use captions, and that providing opportunities for caption customization is best 
    more » « less
  3. This research paper is a study of the support needs of nontraditional students in engineering (NTSE). Nontraditional students in engineering are one segment of the student body that has traditionally not been a part of the conversation in engineering education– those students who do not go through a typical four-year college degree largely at a residential campus. It is only by better understanding the range of issues that NTSE face that we will be able to design interventions and support systems that can assist them. Recent work in engineering education particularly argues that co-curricular support is a critical factor in student success as it effects curricular progress but there has been no work looking specifically at co-curricular support for NTSE and their retention and persistence. The population of NTSE is increasing across campuses as more students take on jobs to support their education and as those in the workforce return to complete their education. It is imperative that higher educational systems understand how to serve the needs of these students better. Although there are a range of ways in which nontraditional students (NTS) are defined, the NCES has proposed a comprehensive definition that includes enrollment criteria, financial and family status, and high school graduation status. Overall, the seven characteristics specifically associated with NTS are: (1) Delayed enrollment by a year or more after high school, (2) attended part-time, (3) having dependents, (4) being a single parent, (5) working full time while enrolled, (6) being financially independent from parents, and (7) did not receive a standard high school diploma. We ground our research in the Model of Co-Curricular Support (MCCS) which suggests it is the role of the institution to provide the necessary support for integration. If students are aware and have access to resources, which lead to their success, then they will integrate into the university environment at higher rates than those students who are not aware and have access to those resources. This research study focuses on answering one research question: How do NTSE engage with co-curricular supports as they progress through their degree programs? To answer this question, we recruited 11 NTSE with a range of nontraditional characteristics to complete prompted reflective journaling assignments five times throughout the Fall 2021 semester. Qualitative results showcase the nuanced lives of NTSE as they pursue their engineering degrees. In particular, results indicate students interact with faculty, classmates, and friends/peers the most, and only interact with advising when required. Students rarely reach out to larger student support for help or are involved with campus or other events happening. Classmate and friend/peer interactions are the most positive, while interactions with faculty had the largest negative outcomes. 
    more » « less
  4. Abstract

    Nearly 200 governments rely on the Intergovernmental Panel on Climate Change (IPCC) for scientific assessments of climate change. IPCC figures are important for conveying key findings, but can be difficult for policymakers and practitioners to understand. Best practices in graph design, summarized in the IPCC’s visual style guide, recommend conducting interviews with members of the target audience before finalizing figures. Therefore, we interviewed 20 policy makers and practitioners from different countries about three figures drafted for the second order draft of the summary for policymakers associated with IPCC’s Working Group III Sixth Assessment Report. Half were frequent users and half were occasional users of climate science, but similar comments emerged from both groups. The figures received a median rating of 3, on a scale from 1 (= not easy at all to understand) to 5 (= very easy to understand). Showing the caption did not always improve these ratings. Overall, two types of recommendations emerged. First, participants suggested focusing each figure on one key message for policymakers, and removing irrelevant details. For IPCC authors, this involves making hard choices about what to show in the figure and what to leave for the text. Additionally, participants suggested straightforward fixes such as using clear titles, labels, and captions that support the key message. Based on our findings, we present recommendations for the design of climate change figures, and examples of revised figures. These recommendations should be useful for the next round of IPCC reports, and for other organizations that communicate about climate science with policymakers and practitioners.

     
    more » « less
  5. To fluidly engage with the world, our brains must simultaneously represent both the scene in front of us and our memory of the immediate surrounding environment (i.e., local visuospatial context). How does the brain's functional architecture enable sensory and mnemonic representations to closely interface while also avoiding sensory-mnemonic interference? Here, we asked this question using first-person, head-mounted virtual reality and fMRI. Using virtual reality, human participants of both sexes learned a set of immersive, real-world visuospatial environments in which we systematically manipulated the extent of visuospatial context associated with a scene image in memory across three learning conditions, spanning from a single FOV to a city street. We used individualized, within-subject fMRI to determine which brain areas support memory of the visuospatial context associated with a scene during recall (Experiment 1) and recognition (Experiment 2). Across the whole brain, activity in three patches of cortex was modulated by the amount of known visuospatial context, each located immediately anterior to one of the three scene perception areas of high-level visual cortex. Individual subject analyses revealed that these anterior patches corresponded to three functionally defined place memory areas, which selectively respond when visually recalling personally familiar places. In addition to showing activity levels that were modulated by the amount of visuospatial context, multivariate analyses showed that these anterior areas represented the identity of the specific environment being recalled. Together, these results suggest a convergence zone for scene perception and memory of the local visuospatial context at the anterior edge of high-level visual cortex.

    SIGNIFICANCE STATEMENTAs we move through the world, the visual scene around us is integrated with our memory of the wider visuospatial context. Here, we sought to understand how the functional architecture of the brain enables coexisting representations of the current visual scene and memory of the surrounding environment. Using a combination of immersive virtual reality and fMRI, we show that memory of visuospatial context outside the current FOV is represented in a distinct set of brain areas immediately anterior and adjacent to the perceptually oriented scene-selective areas of high-level visual cortex. This functional architecture would allow efficient interaction between immediately adjacent mnemonic and perceptual areas while also minimizing interference between mnemonic and perceptual representations.

     
    more » « less