More Like this

1. Measuring and Optimizing Design Variety Using Herfindahl Index

   https://doi.org/10.1115/DETC2019-97778  

   Ahmed, Faez; Ramachandran, Sharath Kumar; Fuge, Mark; Hunter, Sam; Miller, Scarlett (August 2019, Proceedings of the ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference IDETC/CIE2019)

   null (Ed.)

   In this paper, we propose a new design variety metric based on the Herfindahl index. We also propose a practical procedure for comparing variety metrics via the construction of ground truth datasets from pairwise comparisons by experts. Using two new datasets, we show that this new variety measure aligns with human ratings more than some existing and commonly used tree-based metrics. This metric also has three main advantages over existing metrics: a) It is a super-modular function, which enables us to optimize design variety using a polynomial time greedy algorithm. b) The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. c) It has higher sensitivity in distinguishing between variety of sets of randomly selected designs than existing methods. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities. 

   more » « less
2. Measuring and Optimizing Design Variety using Herfindahl Index

   Ahmed, Faez; Ramachandran, Sharath Kumar; Fuge, Mark; Hunter, Sam; Miller, Scarlett (August 2019, ASME Design Engineering Technical Conferences)

   In this paper, we propose a new design variety metric based on the Herfindahl index. We also propose a practical procedure for comparing variety metrics via the construction of ground truth datasets from pairwise comparisons by experts. Using two new datasets, we show that this new variety measure aligns with human ratings more than some existing and commonly used tree-based metrics. This metric also has three main advantages over existing metrics: a) It is a super-modular function, which enables us to optimize design variety using a polynomial time greedy algorithm. b) The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. c) It has higher sensitivity in distinguishing between variety of sets of randomly selected designs than existing methods. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities. 

   more » « less
3. Design Variety Measurement using Sharma-Mittal Entropy

   https://doi.org/10.1115/1.4048743  

   Ahmed, Faez; Ramachandran, Sharath Kumar; Fuge, Mark; Hunter, Samuel; Miller, Scarlett (January 2020, Journal of Mechanical Design)

   null (Ed.)

   Design variety metrics measure how much a design space is explored. We propose that a generalized class of entropy measures based on Sharma-Mittal entropy offers advantages over existing methods to measure design variety. We show that an exemplar metric from Sharma-Mittal entropy, which we call the Herfindahl–Hirschman Index for Design (HHID) has the following desirable advantages over existing metrics: (a) More Accuracy: It better aligns with human ratings compared to existing and commonly used tree-based metrics for two new datasets; (b) Higher Sensitivity: It has higher sensitivity compared to existing methods when distinguishing between the variety of sets; (c) Allows Efficient Optimization: It is a submodular function, which enables us to optimize design variety using a polynomial-time greedy algorithm; and (d) Generalizes to Multiple Measures: The parametric nature of this metric allows us to fit the metric to better represent variety for new domains. The paper also contributes a procedure for comparing metrics used to measure variety via constructing ground truth datasets from pairwise comparisons. Overall, our results shed light on some qualities that good design variety metrics should possess and the non-trivial challenges associated with collecting the data needed to measure those qualities. 

   more » « less
4. An Evaluative Measure of Clustering Methods Incorporating Hyperparameter Sensitivity

   https://doi.org/10.1609/aaai.v36i7.20747  

   Mishra, Siddhartha; Monath, Nicholas; Boratko, Michael; Kobren, Ariel; McCallum, Andrew (June 2022, Proceedings of the AAAI Conference on Artificial Intelligence)

   Clustering algorithms are often evaluated using metrics which compare with ground-truth cluster assignments, such as Rand index and NMI. Algorithm performance may vary widely for different hyperparameters, however, and thus model selection based on optimal performance for these metrics is discordant with how these algorithms are applied in practice, where labels are unavailable and tuning is often more art than science. It is therefore desirable to compare clustering algorithms not only on their optimally tuned performance, but also some notion of how realistic it would be to obtain this performance in practice. We propose an evaluation of clustering methods capturing this ease-of-tuning by modeling the expected best clustering score under a given computation budget. To encourage the adoption of the proposed metric alongside classic clustering evaluations, we provide an extensible benchmarking framework. We perform an extensive empirical evaluation of our proposed metric on popular clustering algorithms over a large collection of datasets from different domains, and observe that our new metric leads to several noteworthy observations. 

   more » « less
5. Group-based Robustness: A General Framework for Customized Robustness in the Real World

   https://doi.org/10.14722/ndss.2024.24084  

   Lin, Weiran; Lucas, Keane; Eyal, Neo; Bauer, Lujo; Reiter, Michael K.; Sharif, Mahmood (February 2024, Network and Distributed System Security Symposium)

   Machine-learning models are known to be vulnerable to evasion attacks, which perturb model inputs to induce misclassifications. In this work, we identify real-world scenarios where the threat cannot be assessed accurately by existing attacks. Specifically, we find that conventional metrics measuring targeted and untargeted robustness do not appropriately reflect a model’s ability to withstand attacks from one set of source classes to another set of target classes. To address the shortcomings of existing methods, we formally define a new metric, termed group-based robustness, that complements existing metrics and is better suited for evaluating model performance in certain attack scenarios. We show empirically that group-based robustness allows us to distinguish between machine-learning models’ vulnerability against specific threat models in situations where traditional robustness metrics do not apply. Moreover, to measure group-based robustness efficiently and accurately, we 1) propose two loss functions and 2) identify three new attack strategies. We show empirically that, with comparable success rates, finding evasive samples using our new loss functions saves computation by a factor as large as the number of targeted classes, and that finding evasive samples, using our new attack strategies, saves time by up to 99% compared to brute-force search methods. Finally, we propose a defense method that increases group-based robustness by up to 3.52 times. 

   more » « less