Search for: All records

Abstract When and how do children come to understand various kinds of visual media (e.g., pictures, videos, scale models), and how does early experience contribute to variation in the development of visual media comprehension across global contexts? In this selective review, we show that while researchers have investigated how children from Western convenience samples understand visual media, less is known about how this comprehension varies across children in global contexts. Indeed, prior work investigating picture comprehension suggests that children in different contexts may understand pictures at different developmental time points, potentially due to variation in their early picture experiences. These findings demonstrate the need for more research investigating children's comprehension of additional kinds of visual media across contexts. The experience‐dependence of visual media comprehension could provide important insight into these abilities’ origins, as well as the appropriateness of cross‐cultural use of visual media in early childhood measurement. 

Abstract The development of large datasets of natural images has galvanized progress in psychology, neuroscience, and computer science. Notably, the THINGS database constitutes a collective effort towards understanding of human visual knowledge by accumulating rich data on a shared set of visual object concepts across several studies. In this paper, we introduce (DoT), a novel dataset of 28,627 human drawings of 1854 diverse object concepts, sampled systematically from concrete picturable and nameable nouns in the American English language, mirroring the structure of the THINGS image database. In addition to data on drawings’ stroke history, we further collected fine-grained recognition data for each drawing, along with metadata on participant demographics, drawing ability, and mental imagery. We characterize people’s ability to communicate and recognize semantic information encoded in drawings and compare this ability to their ability to recognize real-world images of the same visual objects. We also explore the relationship between drawing understanding and the memorability and typicality of the objects contained in THINGS. In sum, we envisionDoTas a powerful tool that builds on the THINGS database to advance understanding of how humans express knowledge about visual concepts. 

Abstract Data visualizations play a crucial role in communicating patterns in quantitative data, making data visualization literacy a key target of STEM education. However, it is currently unclear to what degree different assessments of data visualization literacy measure the same underlying constructs. Here, we administered two widely used graph comprehension assessments (Galesic and Garcia-Retamero in Med Dec Mak 31:444–457, 2011; Lee et al. in IEEE Trans Vis Comput Graph 235:51–560, 2016) to both a university-based convenience sample and a demographically representative sample of adult participants in the USA (N=1,113). Our analysis of individual variability in test performance suggests that overall scores are correlated between assessments and associated with the amount of prior coursework in mathematics. However, further exploration of individual error patterns suggests that these assessments probe somewhat distinct components of data visualization literacy, and we do not find evidence that these components correspond to the categories that guided the design of either test (e.g., questions that require retrieving values rather than making comparisons). Together, these findings suggest opportunities for development of more comprehensive assessments of data visualization literacy that are organized by components that better account for detailed behavioral patterns. 

Human behavior is fundamentally generative: People create pictures, write stories, compose music, and engage in conversation. Traditional approaches in psychology and cognitive science have not focused on this open-endedness, instead favoring more constrained task settings that admit a limited set of outcomes. Although those approaches have been fruitful, new approaches might be needed to develop a unified understanding of the generative, open-ended behaviors that are so emblematic of human cognition. This article demonstrates the value of generative behaviors as targets for cognitive modeling by providing rich behavioral data that reveal how multiple cognitive processes coordinate. Drawing production serves as a case study illustrating this approach, showing how perception, memory, social inference, and motor control coordinate flexibly on the basis of communicative context. Recent advances in generative artificial intelligence offer both new tools for modeling open-ended human behavior and new comparative targets for understanding similarities and differences between human and machine intelligence. However, applying these tools effectively might require new experimental paradigms, larger data sets, and careful consideration of what mechanistic correspondence between models and human cognition is necessary for scientific progress. Embracing the open-ended nature of human thought and behavior poses methodological challenges but offers a promising path toward understanding the most distinctive aspects of human intelligence. 

Sketching serves as a versatile tool for externalizing ideas, enabling rapid exploration and visual communication that spans various disciplines. While artificial systems have driven substantial advances in content creation and human-computer interaction, capturing the dynamic and abstract nature of human sketching remains challenging. In this work, we introduce SketchAgent, a language-driven, sequential sketch generation method that enables users to create, modify, and refine sketches through dynamic, conversational interactions. Our approach requires no training or fine-tuning. Instead, we leverage the sequential nature and rich prior knowledge of off-the-shelf multimodal large language models (LLMs). We present an intuitive sketching language, introduced to the model through in-context examples, enabling it to “draw” using string-based actions. These are processed into vector graphics and then rendered to create a sketch on a pixel canvas, which can be accessed again for further tasks. By drawing stroke by stroke, our agent captures the evolving, dynamic qualities intrinsic to sketching. We demonstrate that SketchAgent can generate sketches from diverse prompts, engage in dialogue-driven drawing, and collaborate meaningfully with human users. 

We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches We introduce a sketch encoder that results in semantically- aware feature space, which we evaluate by testing its performance on a semantic sketch seg- mentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require any pixel-level annotations. To obtain generalization to a large set of sketches and categories, we build on a vision transformer encoder pretrained with the CLIP model. We freeze the text encoder and perform visual-prompt tuning of the visual encoder branch while introducing a set of critical modifications. Firstly, we augment the classical key-query (k-q) self-attention blocks with value-value (v-v) self-attention blocks. Central to our model is a two-level hierarchical network design that enables efficient semantic disentanglement: The first level ensures holistic scene sketch encoding, and the second level focuses on individual categories. We, then, in the second level of the hierarchy, introduce a cross-attention between textual and visual branches. Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of 85.5% on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us to identify further improvements needed over our method to reconcile machine and human understanding of scene sketches.