An official website of the United States government
Abstract Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback.
more »
« less
This content will become publicly available on August 1, 2026
Bridging the gap in engineering creativity evaluations: exploring novice eye-gaze behavior across design modalities
ABSTRACT: The Consensual Assessment Technique (CAT) is one of the most effective and commonly used design evaluation methods. However, it fails to capture implicit cognitive processes and has mainly been studied in a homogenous design modality. To bridge this gap, the present study investigates the impact of design ideas represented in different modalities (i.e., text-only, sketch-only, text + sketch) on design evaluations for creativity, novelty, and usefulness, and examine human gaze patterns during the evaluation process. Our findings showed that novice raters exhibit higher interrater reliability and greater convergence in visual attention when rating ideas containing sketches compared to text-only design modality, highlighting the value of visual elements in design evaluations.
more »
« less
- Award ID(s):
- 2231254
- PAR ID:
- 10635109
- Publisher / Repository:
- Cambridge University Press
- Date Published:
- Journal Name:
- Proceedings of the Design Society
- Volume:
- 5
- ISSN:
- 2732-527X
- Page Range / eLocation ID:
- 781 to 790
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Engineering design is undergoing a transformative shift with the advent of AI, marking a new era in how we approach product, system, and service planning. Large language models have demonstrated impressive capabilities in enabling this shift. Yet, with text as their only input modality, they cannot leverage the large body of visual artifacts that engineers have used for centuries and are accustomed to. This gap is addressed with the release of multimodal vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more types of tasks. Our work presents a comprehensive evaluation of VLMs across a spectrum of engineering design tasks, categorized into four main areas: Conceptual Design, System-Level and Detailed Design, Manufacturing and Inspection, and Engineering Education Tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems. Through this structured evaluation, we not only explore VLMs’ proficiency in handling complex design challenges but also identify their limitations in complex engineering design applications. Our research establishes a foundation for future assessments of vision language models. It also contributes a set of benchmark testing datasets, with more than 1000 queries, for ongoing advancements and applications in this field.more » « less
-
Abstract Engineering design involves intensive visual-spatial reasoning, and engineers depend upon external representation to develop concepts during idea generation. Previous research has not explored how our visual representation skills influence our idea generation effectiveness. A designer’s deficit in sketching skills could create a need for increased focus on the task of visual representation reducing cognitive resources available for the task at hand — generating concept. Further, this effect could be compounded if designers believed that their sketching skill would be evaluated or judged by their peers. This evaluation apprehension could cause additional mental workload distracting from the production of idea generation. The goal of this study is to investigate and better understand the relationship between designers’ sketching skills and idea generation abilities. In this paper, we present preliminary results of the relationship between independent measures of sketching skill and idea generation ability from an entry-level engineering design and graphics course. During data collection, task instructions were given in two ways to independent groups: one group was instructed upfront that sketching would be evaluated, while the second group was kept blind to the sketch evaluation. In this paper, we also examine the potential priming effects of sketch quality evaluation apprehension on idea generation productivity. The results show that sketching quality and idea quantity are largely independent, and that the priming effects of sketch evaluation instructions are small to negligible on idea generation productivity.more » « less
-
We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches We introduce a sketch encoder that results in semantically- aware feature space, which we evaluate by testing its performance on a semantic sketch seg- mentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require any pixel-level annotations. To obtain generalization to a large set of sketches and categories, we build on a vision transformer encoder pretrained with the CLIP model. We freeze the text encoder and perform visual-prompt tuning of the visual encoder branch while introducing a set of critical modifications. Firstly, we augment the classical key-query (k-q) self-attention blocks with value-value (v-v) self-attention blocks. Central to our model is a two-level hierarchical network design that enables efficient semantic disentanglement: The first level ensures holistic scene sketch encoding, and the second level focuses on individual categories. We, then, in the second level of the hierarchy, introduce a cross-attention between textual and visual branches. Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of 85.5% on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us to identify further improvements needed over our method to reconcile machine and human understanding of scene sketches.more » « less
-
Conceptual design is the foundational stage of a design process that translates ill-defined design problems into low-fidelity design concepts and prototypes through design search, creation, and integration. In this stage, product shape design is one of the most paramount aspects. When applying deep learning-based methods to product shape design, two major challenges exist: (1) design data exhibit in multiple modalities and (2) an increasing demand for creativity. With recent advances in deep learning of cross-modal tasks (DLCMTs), which can transfer one design modality to another, we see opportunities to develop artificial intelligence (AI) to assist the design of product shapes in a new paradigm. In this paper, we conduct a systematic review of the retrieval, generation, and manipulation methods for DLCMT that involve three cross-modal types: text-to-3D shape, text-to-sketch, and sketch-to-3D shape. The review identifies 50 articles from a pool of 1341 papers in the fields of computer graphics, computer vision, and engineering design. We review (1) state-of-the-art DLCMT methods that can be applied to product shape design and (2) identify the key challenges, such as lack of consideration of engineering performance in the early design phase that need to be addressed when applying DLCMT methods. In the end, we discuss the potential solutions to these challenges and propose a list of research questions that point to future directions of data-driven conceptual design.more » « less
