Search for: All records

Human scene understanding dynamically evolves over the course of sequential viewing fixations from a gistlevel understanding to a more detailed comprehension of the scene. Each fixation provides rich visual information about objects and their spatial relationships. To model this incremental process, we introduce Seen2Scene, a framework for modeling human scene understanding by controlling the inputs used to generate a visual hypothesis of the scene. Seen2Scene uses a self-supervised encoder to extract features from fixated scene regions, which guide a pre-trained text-to-image latent diffusion modelthroughamodularadapterframework. Asfixations accumulate, the model iteratively refines its visual hypotheses, filling in unseen areas with contextually plausible content. We evaluated Seen2Scene on COCOFreeView using two experimental conditions: fixationonly conditioning to isolate the contribution of foveal information, and fixation+gist conditioning to examine how non-fixated scene information integrates with foveal details. Results show that initial fixations drive the greatest gains in semantic and perceptual fidelity and that the fixation+gist condition reached high-fidelity scene understanding with the fewest fixations, thus demonstrating the importance of integrating peripheral gist information with visual details collected foveally. Keywords: generative modeling, human scene understanding 

Human efficiency in finding a target in an image has attracted the attention of machine learning researchers, but what about when no target is there? Knowing how people search in the absence of a target, and when they stop, is important for Human-computer-interaction systems attempting to predict human gaze behavior in the wild. Here we report a rigorous evaluation of target-absent search behavior using the COCO-Search18 dataset to train stateof- the-art models. We focus on two specific aims. First, we characterize the presence of a target guidance signal in target-absent search behavior by comparing it to targetpresent guidance and free viewing. We do this by comparing how well a model trained on one type of fixation behavior (target-present, target-absent, free viewing) can predict behavior in either the same or different task. To compare target-absent search to free viewing behavior we created COCO-FreeView, a dataset of free-viewing fixations for the same images used in COCO-Search18. These comparisons revealed the existence of a target guidance signal in targetabsent search, albeit one much less dominant compared to when a target actually appeared in an image, and that the target-absent guidance signal was similar to free viewing in that saliency and center bias were both weighted more than guidance from target features. Our second aim focused on the stopping criteria, a question intrinsic to target-absent search. Here we propose to train a foveated target detector whose target detection representation is sensitive to the relationship between distance from the fovea. Then combining the predicted target detection representation with other information such as fixation history and subject ID, our model outperforms the baselines in predicting when a person stops moving his attention during target-absent search.