More Like this

1. How Should We Measure Creativity in Engineering Design? A Comparison Between Social Science and Engineering Approaches

   https://doi.org/10.1115/1.4049061  

   Miller, Scarlett R.; Hunter, Samuel T.; Starkey, Elizabeth; Ramachandran, Sharath; Ahmed, Faez; Fuge, Mark (March 2021, Journal of Mechanical Design)

   null (Ed.)

   Abstract Design researchers have long sought to understand the mechanisms that support creative idea development. However, one of the key challenges faced by the design community is how to effectively measure the nebulous construct of creativity. The social science and engineering communities have adopted two vastly different approaches to solving this problem, both of which have been deployed throughout engineering design research. The goal of this paper was to compare and contrast these two approaches using design ratings of nearly 1000 engineering design ideas. The results of this study identify that while these two methods provide similar ratings of idea quality, there was a statistically significant negative relationship between these methods for ratings of idea novelty. In addition, the results show discrepancies in the reliability and consistency of global ratings of creativity. The results of this study guide the deployment of idea ratings in engineering design research and evidence. 

   more » « less
2. Calibrating Novelty Metrics Using Human Raters: A Case Study

   Ramachandran, S.K.; Ahmed, F.; Fuge, M.; Hunter, S.; Miller, S.R. (January 2019, ASME 2018 International Design Engineering Technical Conferences & Design Conference)

   One of the key challenges facing the engineering design community is how to effectively measure the nebulous construct of design novelty. The community has adopted two vastly different approaches to solving this problem; a more quantitative approach that relies on feature-trees and a more subjective approach that uses human raters. The goal of this study was to identify a method for using human raters as a means of calibrating feature-tree based novelty metrics in engineering design. This was accomplished through a study where four raters were asked to follow a think-out-loud protocol while they physically created idea maps for 10 design concepts based on the similarity of these concepts. Content analysis was used to identify the relative importance of idea properties that informed judgements of concept similarity. This analysis was then compared to the weights used in traditional feature-tree based novelty methods. These results of this study can be used to calibrate existing metrics against expert ratings to provide justification for the categorizes used in the creation of a feature tree in engineering design research and also justify the weights used in the computation of design novelty. 

   more » « less
3. Automating creativity assessment with SemDis: An open platform for computing semantic distance

   https://doi.org/10.3758/s13428-020-01453-w  

   Beaty, Roger E.; Johnson, Dan R. (April 2021, Behavior Research Methods)

   null (Ed.)

   Abstract Creativity research requires assessing the quality of ideas and products. In practice, conducting creativity research often involves asking several human raters to judge participants’ responses to creativity tasks, such as judging the novelty of ideas from the alternate uses task (AUT). Although such subjective scoring methods have proved useful, they have two inherent limitations—labor cost (raters typically code thousands of responses) and subjectivity (raters vary on their perceptions and preferences)—raising classic psychometric threats to reliability and validity. We sought to address the limitations of subjective scoring by capitalizing on recent developments in automated scoring of verbal creativity via semantic distance, a computational method that uses natural language processing to quantify the semantic relatedness of texts. In five studies, we compare the top performing semantic models (e.g., GloVe, continuous bag of words) previously shown to have the highest correspondence to human relatedness judgements. We assessed these semantic models in relation to human creativity ratings from a canonical verbal creativity task (AUT; Studies 1–3) and novelty/creativity ratings from two word association tasks (Studies 4–5). We find that a latent semantic distance factor—comprised of the common variance from five semantic models—reliably and strongly predicts human creativity and novelty ratings across a range of creativity tasks. We also replicate an established experimental effect in the creativity literature (i.e., the serial order effect) and show that semantic distance correlates with other creativity measures, demonstrating convergent validity. We provide an open platform to efficiently compute semantic distance, including tutorials and documentation ( https://osf.io/gz4fc/ ). 

   more » « less
4. A MAD method to assess idea novelty: Improving validity of automatic scoring using maximum associative distance (MAD)

   https://doi.org/10.1037/aca0000573  

   Yu, Yuhua; Beaty, Roger E; Forthmann, Boris; Beeman, Mark; Cruz, John Henry; Johnson, Dan (April 2023, Psychology of Aesthetics, Creativity, and the Arts)

   Creativity research often relies on human raters to judge the novelty of participants’ responses on open-ended tasks, such as the Alternate Uses Task (AUT). Albeit useful, manual ratings are subjective and labor intensive. To address these limitations, researchers increasingly use automatic scoring methods based on a natural language processing technique for quantifying the semantic distance between words. However, many methodological choices remain open on how to obtain semantic distance scores for ideas, which can significantly impact reliability and validity. In this project, we propose a new semantic distance-based method, maximum associative distance (MAD), for assessing response novelty in AUT. Within a response, MAD uses the semantic distance of the word that is maximally remote from the prompt word to reflect response novelty. We compare the results from MAD with other competing semantic distance-based methods, including element-wise-multiplication—a commonly used compositional model—across three published datasets including a total of 447 participants. We found MAD to be more strongly correlated with human creativity ratings than the competing methods. In addition, MAD scores reliably predict external measures such as openness to experience. We further explored how idea elaboration affects the performance of various scoring methods and found that MAD is closely aligned with human raters in processing multi-word responses. The MAD method thus improves the psychometrics of semantic distance for automatic creativity assessment, and it provides clues about what human raters find creative about ideas. 

   more » « less
5. Unpacking subjective creativity ratings: Using embeddings to explain and measure idea novelty

   Ahmed, Faez; Fuge, Mark; Hunter, Sam; Miller, Scarlett (August 2018, ASME Design Engineering Technical Conferences)

   Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets and scoring ideas on their uniqueness. However, their decisions on novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two dimensional idea maps. We derive these maps by asking humans for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the domain. We also propose that the novelty of ideas can be estimated by measuring how far items are on these maps. We demonstrate our methodology through the experimental evaluations on two datasets of colored polygons (known answer) and milk frothers (unknown answer) sketches. We show that these maps shed light on factors considered by raters in judging idea similarity. We also show how maps change when less data is available or false/noisy ratings are provided. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods. 

   more » « less