skip to main content

Attention:

The NSF Public Access Repository (PAR) system and access will be unavailable from 11:00 PM ET on Thursday, January 16 until 2:00 AM ET on Friday, January 17 due to maintenance. We apologize for the inconvenience.


Title: Automatic Assessment of Students’ Engineering Design Performance Using a Bayesian Network Model

Integrating engineering design into K-12 curricula is increasingly important as engineering has been incorporated into many STEM education standards. However, the ill-structured and open-ended nature of engineering design makes it difficult for an instructor to keep track of the design processes of all students simultaneously and provide personalized feedback on a timely basis. This study proposes a Bayesian network model to dynamically and automatically assess students’ engagement with engineering design tasks and to support formative feedback. Specifically, we applied a Bayesian network to 111 ninth-grade students’ process data logged by a computer-aided design software program that students used to solve an engineering design challenge. Evidence was extracted from the log files and fed into the Bayesian network to perform inferential reasoning and provide a barometer of their performance in the form of posterior probabilities. Results showed that the Bayesian network model was competent at predicting a student’s task performance. It performed well in both identifying students of a particular group (recall) and ensuring identified students were correctly labeled (precision). This study also suggests that Bayesian networks can be used to pinpoint a student’s strengths and weaknesses for applying relevant science knowledge to engineering design tasks. Future work of implementing this tool within the computer-aided design software will provide instructors a powerful tool to facilitate engineering design through automatically generating personalized feedback to students in real time.

 
more » « less
Award ID(s):
1503196
PAR ID:
10546662
Author(s) / Creator(s):
 ;  ;  ;  ;  ;  ;  
Publisher / Repository:
SAGE Publications
Date Published:
Journal Name:
Journal of Educational Computing Research
Volume:
59
Issue:
2
ISSN:
0735-6331
Format(s):
Medium: X Size: p. 230-256
Size(s):
p. 230-256
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract

    Computer-aided Design for Manufacturing (DFM) systems play an essential role in reducing the time taken for product development by providing manufacturability feedback to the designer before the manufacturing phase. Traditionally, DFM rules are hand-crafted and used to accelerate the engineering product design process by integrating manufacturability analysis during design. Recently, the feasibility of using a machine learning-based DFM tool in intelligently applying the DFM rules have been studied. These tools use a voxelized representation of the design and then use a 3D-Convolutional Neural Network (3D-CNN), to provide manufacturability feedback. Although these frameworks work effectively, there are some limitations to the voxelized representation of the design. In this paper, we introduce a new representation of the computer-aided design (CAD) model using orthogonal distance fields (ODF). We provide a GPU-accelerated algorithm to convert standard boundary representation (B-rep) CAD models into ODF representation. Using the ODF representation, we build a machine learning framework, similar to earlier approaches, to create a machine learning-based DFM system to provide manufacturability feedback. As proof of concept, we apply this framework to assess the manufacturability of drilled holes. The framework has an accuracy of more than 84% correctly classifying the manufacturable and non-manufacturable models using the new representation.

     
    more » « less
  2. Engineering projects, such as designing a solar farm that converts solar radiation shined on the Earth into electricity, engage students in addressing real-world challenges by learning and applying geoscience knowledge. To improve their designs, students benefit from frequent and informative feedback as they iterate. However, teacher attention may be limited or inadequate, both during COVID-19 and beyond. We present Aladdin, a web-based computer-aided design (CAD) platform for engineering design with a built-in artificial intelligence teaching assistant (AITA). We also present two curriculum units (Solar Energy Science and Solar Farm Design), where students explore the Sun-Earth relationship and optimize the energy output and yearly profit of a solar farm with the help of the AITA. We tested the software and curriculum units with over 100 students in two Midwestern high schools. Pre- and post-survey data showed improvements in understanding of science concepts and self-efficacy in engineering design. Pre-post analysis of design performance gains reveals that AI helped lower achievers more than higher achievers. Interviews revealed students’ values and preferences when receiving feedback. Our findings suggest that AITAs may be helpful as an additional feedback mechanism for geoscience and engineering education. Future efforts should focus on improving the usability of the software and providing multiple types of feedback to promote inclusive and equitable use of AI in education. 
    more » « less
  3. Computer-aided design (CAD) programs are essential to engineering as they allow for better designs through low-cost iterations. While CAD programs are typically taught to undergraduate students as a job skill, such software can also help students learn engineering concepts. A current limitation of CAD programs (even those that are specifically designed for educational purposes) is that they are not capable of providing automated real-time help to students. To encourage CAD programs to build in assistance to students, we used data generated from students using a free, open-source CAD software called Aladdin to demonstrate how student data combined with machine learning techniques can predict how well a particular student will perform in a design task. We challenged students to design a house that consumed zero net energy as part of an introductory engineering technology undergraduate course. Using data from 128 students, along with the scikit-learn Python machine learning library, we tested our models using both total counts of design actions and sequences of design actions as inputs. We found that our models using early design sequence actions are particularly valuable for prediction. Our logistic regression model achieved a >60% chance of predicting if a student would succeed in designing a zero net energy house. Our results suggest that it would be feasible for Aladdin to provide useful feedback to students when they are approximately halfway through their design. Further improvements to these models could lead to earlier predictions and thus provide students feedback sooner to enhance their learning. 
    more » « less
  4. The quest to incorporate digital games into US classrooms has been pervasive in educational communities over the last two decades. Educational video games have been studied as a mechanism for enhancing the engagement and performance of underrepresented groups (UGs) in spatial learning, physics, computer science, general engineering, software and electrical engineering, mechanical engineering (ME) computer aided design, and aerospace engineering. Less than a handful of these studies have explored games’ appeal, efficacy or UG performance as a function of gender. Preliminary findings on a study that explores the appeal, efficacy, and performance of UGs in engineering-based educational video games as a function of gender and those of intersectional backgrounds is discussed. Emphasis is placed on elucidating these students' perceptions of serious game structure, design and content, and how these factors motivate their learning of engineering concepts and self-identification as engineers. This work builds upon the Technology Acceptance Model. 
    more » « less
  5. This Research Work In Progress Paper examines empirical evidence on the impacts of feedback from an intelligent tutoring software on sketching skill development. Sketching is a vital skill for engineering design, but sketching is only taught limitedly in engineering education. Teaching sketching usually involves one-on-one feedback which limits its application in large classrooms. To meet the demands of feedback for sketching instruction, SketchTivity was developed as an intelligent tutoring software. SketchTivity provides immediate personalized feedback on sketching freehand practice. The current study examines the effectiveness of the feedback of SketchTivity by comparing students practicing with the feedback and without. Students were evaluated on their motivation for practicing sketching, the development of their skills, and their perceptions of the software. This work in progress paper examines preliminary analysis in all three of these areas. 
    more » « less