More Like this

1. Evaluating Heuristics in Engineering Design: A Reinforcement Learning Approach

   https://doi.org/10.1115/detc2021-70425  

   ElSayed, Karim A. ; Bilionis, Ilias ; Panchal, Jitesh H. ( August 2021 , ASME IDETC)

   Heuristics are essential for addressing the complexities of engineering design processes. The goodness of heuristics is context-dependent. Appropriately tailored heuristics can enable designers to find good solutions efficiently, and inappropriate heuristics can result in cognitive biases and inferior design outcomes. While there have been several efforts at understanding which heuristics are used by designers, there is a lack of normative understanding about when different heuristics are suitable. Towards addressing this gap, this paper presents a reinforcement learning-based approach to evaluate the goodness of heuristics for three sub-problems commonly faced by designers: (1) learning the map between the design space and the performance space, (2) acquiring sequential information, and (3) stopping the information acquisition process. Using a multi-armed bandit formulation and simulation studies, we learn the suitable heuristics for these individual sub-problems under different resource constraints and problem complexities. Additionally, we learn the optimal heuristics for the combined problem (i.e., the one composing all three sub-problems), and we compare them to ones learned at the sub-problem level. The results of our simulation study indicate that the proposed reinforcement learning-based approach can be effective for determining the quality of heuristics for different problems, and how the effectiveness of the heuristics changes as a function of the designer’s preference (e.g., performance versus cost), the complexity of the problem, and the resources available.

    

   more » « less
2. Evaluating Heuristics in Engineering Design: A Reinforcement Learning Approach

   ElSayed, K.A. ; Bilionis, I. ; Panchal, J.H. ( August 2021 , ASME IDETC)

   null (Ed.)

   Heuristics are essential for addressing the complexities of engineering design processes. The goodness of heuristics is context-dependent. Appropriately tailored heuristics can enable designers to find good solutions efficiently, and inappropriate heuristics can result in cognitive biases and inferior design outcomes. While there have been several efforts at understanding which heuristics are used by designers, there is a lack of normative understanding about when different heuristics are suitable. Towards addressing this gap, this paper presents a reinforcement learning-based approach to evaluate the goodness of heuristics for three sub-problems commonly faced by designers while carrying out design under resource constraints: (i) learning the mapping between the design space and the performance space, (ii) sequential information acquisition in design, and (iii) decision to stop information acquisition. Using a multi-armed bandit formulation and simulation studies, we learn the heuristics that are suitable for these sub-problems under different resource constraints and problem complexities. The results of our simulation study indicate that the proposed reinforcement learning-based approach can be effective for determining the quality of heuristics for different sub-problems, and how the effectiveness of the heuristics changes as a function of the designer's preference (e.g., performance versus cost), the complexity of the problem, and the resources available. 

   more » « less
3. Beyond Trial & Error: Iteration-to-learn using computational papercrafts in a STEAM camp for girls

   Dixon, Colin ; Schimpf, Corey T. ; Hsi, Sherry H. ( January 2019 , 2019 ASEE Annual Conference & Exposition)

   When asked about how they deal with unforeseen problems, novice learners often describe a process of “trial and error.” This process might fairly be described as iteration, a critical step in the design process, but falls short of the practices that engineering education needs to develop. In the face of novel and multifaceted problems, future engineers must be comfortable and competent not just trying again, but identifying failure points, troubleshooting, and running systematic tests with relevant data. To examine the abilities of novice designers to test and effectively refine ideas and prototypes, we conducted qualitative analysis of structured interviews, audio, video, and designs of 11 girls, ages 9 -11, working on computational papercrafts as part of a museum-based STEAM summer camp. The projects involved design and construction of expressive paper and cardboard sculptures with gears and linkages powered by servomotors. Over the course of one day, the girls generated designs inspired by a camp theme, then had to work with mechanics, electronics and craft to create working versions that would be displayed as part of a public exhibit. Computational papercraft was selected because it lowers cost and intimidation. Our design conjecture was that by making materials familiar and abundant, learners would have more relevant knowledge, could easily modify and replicate components, and would therefore be better able to recognize potential faults and more likely to engage in testing and refinement. We also supported design and troubleshooting with a customized circuit board and an online gear simulator. In the first stage of this study, we looked at what engineering practices emerged, given these conditions. We asked: What opportunities for testing and refinement did computational papercrafts open up? What resources and tools do young learners employ when testing and refining designs? Analysis showed that technical supports for testing and refinement were successful in supporting valued testing and refinement practices as youth pursued personal goals. Use of the simulator and customized microcontroller allowed for consideration of multiple alternatives and for “trial before error.” Learners were able to conduct focused tests on subsystems of their paper machines, and to make “small bets,” keeping initial ideas and designs fluid. Inexpensive materials also allowed them to test and refine at late project stages, without feeling that they were wasting time or materials. The analysis sheds light on young students practices of testing and refinement, and how to best support young people as they begin learning trajectories in engineering. The approach is especially relevant within making-oriented engineering education and other settings working to broaden participation in engineering. 

   more » « less
4. Quantifying Individuals’ Theory-Based Knowledge Using Probabilistic Causal Graphs: A Bayesian Hierarchical Approach

   https://doi.org/10.1115/DETC2020-22613  

   Hans, Atharva ; Chaudhari, Ashish M. ; Bilionis, Ilias ; Panchal, Jitesh H. ( August 2020 , ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference)

   null (Ed.)

   Extracting an individual’s knowledge structure is a challenging task as it requires formalization of many concepts and their interrelationships. While there has been significant research on how to represent knowledge to support computational design tasks, there is limited understanding of the knowledge structures of human designers. This understanding is necessary for comprehension of cognitive tasks such as decision making and reasoning, and for improving educational programs. In this paper, we focus on quantifying theory-based causal knowledge, which is a specific type of knowledge held by human designers. We develop a probabilistic graph-based model for representing individuals’ concept-specific causal knowledge for a given theory. We propose a methodology based on probabilistic directed acyclic graphs (DAGs) that uses logistic likelihood function for calculating the probability of a correct response. The approach involves a set of questions for gathering responses from 205 engineering students, and a hierarchical Bayesian approach for inferring individuals’ DAGs from the observed responses. We compare the proposed model to a baseline three-parameter logistic (3PL) model from the item response theory. The results suggest that the graph-based logistic model can estimate individual students’ knowledge graphs. Comparisons with the 3PL model indicate that knowledge assessment is more accurate when quantifying knowledge at the level of causal relations than quantifying it using a scalar ability parameter. The proposed model allows identification of parts of the curriculum that a student struggles with and parts they have already mastered which is essential for remediation.

    

   more » « less
5. Quantifying Individuals' Theory-based Knowledge using Probabilistic Causal Graphs: A Bayesian Hierarchical Approach

   Hans, Atharva ; Chaudhari, Ashish M. ; Bilionis, Ilias ; Panchal, Jitesh H. ( January 2020 , ASME 2020 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference (IDETC/CIE 2020))

   Extracting an individual’s knowledge structure is a challenging task as it requires formalization of many concepts and their interrelationships. While there has been significant research on how to represent knowledge to support computational design tasks, there is limited understanding of the knowledge structures of human designers. This understanding is necessary for comprehension of cognitive tasks such as decision making and reasoning, and for improving educational programs. In this paper, we focus on quantifying theory-based causal knowledge, which is a specific type of knowledge held by human designers. We develop a probabilistic graph-based model for representing individuals’ concept-specific causal knowledge for a given theory. We propose a methodology based on probabilistic directed acyclic graphs (DAGs) that uses logistic likelihood function for calculating the probability of a correct response. The approach involves a set of questions for gathering responses from 205 engineering students, and a hierarchical Bayesian approach for inferring individuals’ DAGs from the observed responses. We compare the proposed model to a baseline three-parameter logistic (3PL) model from the item response theory. The results suggest that the graph-based logistic model can estimate individual students’ knowledge graphs. Comparisons with the 3PL model indicate that knowledge assessment is more accurate when quantifying knowledge at the level of causal relations than quantifying it using a scalar ability parameter. The proposed model allows identification of parts of the curriculum that a student struggles with and parts they have already mastered which is essential for remediation. 

   more » « less