Search for: All records

Design variety metrics measure how much a design space is explored. We propose that a generalized class of entropy measures based on Sharma-Mittal entropy offers advantages over existing methods to measure design variety. We show that an exemplar metric from Sharma-Mittal entropy, which we call the Herfindahl–Hirschman Index for Design (HHID) has the following desirable advantages over existing metrics: (a) More Accuracy: It better aligns with human ratings compared to existing and commonly used tree-based metrics for two new datasets; (b) Higher Sensitivity: It has higher sensitivity compared to existing methods when distinguishing between the variety of sets; (c) Allows Efficient Optimization: It is a submodular function, which enables us to optimize design variety using a polynomial-time greedy algorithm; and (d) Generalizes to Multiple Measures: The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. The paper also contributes a procedure for comparing metrics used to measure variety via constructing ground truth datasets from pairwise comparisons. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities. 

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets to score ideas on their uniqueness. However, their decisions about novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two-dimensional (2D) idea maps. We derive these maps by asking participants for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the design domain. We also propose that novel ideas can be identified by finding outliers on these idea maps. To demonstrate our method, we conduct experimental evaluations on two datasets—colored polygons (known answer) and milk frother sketches (unknown answer). We show that idea maps shed light on factors considered by participants in judging idea similarity and the maps are robust to noisy ratings. We also compare physical maps made by participants on a white-board to their computationally generated idea maps to compare how people think about spatial arrangement of design items. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods. 

In this paper, we propose a new design variety metric based on the Herfindahl index. We also propose a practical procedure for comparing variety metrics via the construction of ground truth datasets from pairwise comparisons by experts. Using two new datasets, we show that this new variety measure aligns with human ratings more than some existing and commonly used tree-based metrics. This metric also has three main advantages over existing metrics: a) It is a super-modular function, which enables us to optimize design variety using a polynomial time greedy algorithm. b) The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. c) It has higher sensitivity in distinguishing between variety of sets of randomly selected designs than existing methods. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities. 

One of the key challenges facing the engineering design community is how to effectively measure the nebulous construct of design novelty. The community has adopted two vastly different approaches to solving this problem; a more quantitative approach that relies on feature-trees and a more subjective approach that uses human raters. The goal of this study was to identify a method for using human raters as a means of calibrating feature-tree based novelty metrics in engineering design. This was accomplished through a study where four raters were asked to follow a think-out-loud protocol while they physically created idea maps for 10 design concepts based on the similarity of these concepts. Content analysis was used to identify the relative importance of idea properties that informed judgements of concept similarity. This analysis was then compared to the weights used in traditional feature-tree based novelty methods. These results of this study can be used to calibrate existing metrics against expert ratings to provide justification for the categorizes used in the creation of a feature tree in engineering design research and also justify the weights used in the computation of design novelty.