In undergraduate engineering programs, recent emphasis has been placed on a more holistic, interdisciplinary approach to engineering education. Some programs now teach product design within the context of the market, extending the curriculum to topics outside of scientific labs and computational analysis. This study analyzes survey and concept map data collected from 154 students in a third-year engineering design course. The aim is to evaluate the impacts of student backgrounds and experiences on their mental models of product design. Data were gathered from surveys on student backgrounds and experiences, along with concept maps that were generated by the students on the first day of a product design class. The concept maps were analyzed in a quantitative manner for structural and thematic elements. The findings show that several background attributes influence student conceptions of product design. Academic major appeared to have the largest impact on a variety of variables. Additionally, prior work experience, enrollment in a master’s program, and the presence of an engineering role model at home all showed significant impacts on design conceptions. By analyzing and understanding unique backgrounds of students, educators can adjust their curricula to more effectively teach design concepts to students of various backgrounds and experiences.
]]>Traditional undergraduate engineering programs emphasize technical knowledge, with courses in mathematics, physics, mechanics, thermodynamics, and other quantitative topics. While these subjects are undoubtedly critical for aspiring engineers, they can often overpower the importance of design education that also includes non-technical factors such as the markets in which designed artifacts must thrive [1]. This has prompted many institutions to reevaluate their design education curricula, making room for a more holistic approach to engineering design that emphasizes both technical skills and business acumen. Examples of design-related engineering education initiatives include the conceive, design, implement, and operate (CDIO) approach [2]; integrative STEM education [3]; the proliferation of capstone design courses [4], and the rise in project-based learning [5]. Many of today's engineering students are now receiving some level of training in interdisciplinary design topics such as market analysis, financial feasibility, and business planning to supplement their technical skills.
It is widely accepted that students' individual backgrounds and experiences influence their initial knowledge and conceptions surrounding a topic prior to beginning coursework [6][7][8][9][10]. This study focuses specifically on how engineering student backgrounds influence the breadth and depth within their conceptions of product design prior to beginning a course on the topic. The primary research question is: How do the backgrounds and academic profiles of engineering students influence their conceptions of product design? Specifically, conceptions of product design are analyzed through individual concept maps generated by the students, and the following background and academic information is considered: previous work experience, for example through internships or co-ops; parents or role models with engineering degrees or professions; academic major; and intentions to pursue a master's degree.
At the beginning of a third-year undergraduate engineering design course, data were collected from 154 students through a survey and a concept mapping activity. The survey gathered details about the students' backgrounds and academic profiles, and the concept maps were generated individually around the central concept of "product design." These maps were explored in a quantitative manner, analyzing both the structural and thematic elements of the concept maps. Significant correlations are then identified through analyses of variance (ANOVAs) with the background and academic data as independent variables and the concept map metrics as dependent variables. The findings are discussed in the context of their fundamental contributions to knowledge about student learning as well as their implications to support engineering design education improvement.
To provide a framework for the analysis, this section presents an examination of the existing literature on design education, concept mapping, and the general impact of unique student backgrounds on learning.
Engineering Design Education Engineering design education has undergone substantial changes over the last 50 years, as studies began to indicate a skills gap in trained engineers. It became apparent that engineering education was not providing enough "real-world" knowledge, leaving new graduate engineers with a surplus of technical skills, but a deficit in market understanding and financial literacy [1]. As engineering programs have evolved, more emphasis has been placed on more engaging classroom experiences as well as multidisciplinary topics such as holistic design, business, and sustainability.
One early documented effort was a longitudinal study comparing active and cooperative classroom styles against traditional lecture teaching [11]. The study investigated two separate groups of chemical engineering students: Each student took different versions of the same courses, but the experimental group's courses included multiple methods of instruction to provide a more holistic learning experience, including open-ended questioning and more multidisciplinary integration within the course material. The control group took traditional lecture courses. The study concluded that the experimental students had a 20-percent higher 5-year graduation rate than those in the traditional lecture courses (85 percent vs. 65 percent). This research alludes to the successes of multidisciplinary problem formulation in STEM (science, technology, engineering, and math) fields. Rather than merely lecturing and examining students on technical material, students were introduced to broader concepts beyond their specific field of study that helped them understand course material in better context and create a more robust, applicable expertise of course material.
Another example of this was the introduction and growth of capstone design projects and project-based learning (PrBL) methods. In these types of courses, students are simultaneously introduced to material while applying the same concepts to realworld tasks. According to a 2005 survey, the most popular format of PrBL includes one-to two-semester design experiences with lectures and projects being conducted simultaneously, iterating on the project each week. These types of courses have shown increasing effectiveness on students' academic achievement in recent years [5]. Furthermore, these PrBL methods in college courses aim to address the aforementioned skills gap in young engineers entering the workforce. One study followed several students into their careers following their completion of engineering programs [12]. These students were surveyed regarding their work activities in the first 12 weeks of their jobs. Over 75 percent said that they engaged in regular team meetings, and more than 50 percent said they engaged in planning activities and design refinement within their first 12 weeks. Team meetings, project management, and refining designs based on customer needs are all key elements of project-based and holistic design curricula. By practicing these skills in various contexts in school, students are more prepared to enter the workforce.
Concept Mapping Concept maps are organizational tools similar to flowcharts, but more limited in that they only contain one class of elements (concepts). They are constructed by creating nodes consisting of nouns or noun phrases, and then connecting those nodes together with linking verb phrases [13]. As a concept map grows, nodes and linking phrases begin to link with concepts from other fields of knowledge. In many contexts, concept mapping can act as an alternative to exams and other more traditional evaluation methods [14]. The value of concept maps stems from their ability to display interdisciplinary relationships among concepts. Typically, concept maps originate with a focus question or topic and branch outward. For example, Figure 1 provides an example concept map starting with the central concept of product design. Through concept mapping, individuals or groups can express and organize complex connections between different ideas in their minds, ultimately developing a more holistic, robust understanding. One study has shown that young students that practice regular concept mapping learn more effectively [15].
Concept maps are evaluated and assessed differently than more traditional learning evaluation methods. For students, the FIGURE 1. Example concept map on product design process of creating concept maps is a powerful method to synthesize knowledge, as it graphically displays and organizes knowledge of a student's thoughts surrounding a particular concept or field [16]. In the literature and in practice, concept maps are analyzed in many different ways, depending on the purpose of the exercise. Generally, numerically assessing node counts and looking at the network density is a common approach to understanding and evaluating concept maps from a structural perspective [17]. In the context of studying the progression of students over time, a greater number of relationships between nodes has been found to be an indicator of more comprehensive understanding [18]. In many experiments, concept maps are evaluated by comparing to a master map, which includes concepts and links that align with the viewpoints of subject matter experts. Student-generated maps are then compared against these master maps to evaluate thoroughness of understanding [19,20]. While these methods have been proven useful in other studies, the study reported here differs in that the analyses do not include a desired outcome or expert map. This is because design is inherently ubiquitous and context-driven, with no absolutely correct approach [21].
Backgrounds in Education Previous studies have analyzed how different backgrounds and environmental factors impact student performance. One study in Indonesia looked at parental education backgrounds of young students learning the English language. The results of the study showed that higher parental education levels were significantly correlated with better performance on English assessments. This indicates that parental support is an influencing factor in education outputs. The same study also suggested that there are other factors that may contribute to student success, including teachers, friends, and environmental factors [7].
Other studies have been conducted focusing on scientific backgrounds of students in STEM courses. In one study, interactive teaching methods were tested with two groups of students: one with strong science backgrounds and one with little to no scientific backgrounds. The teaching methods included both traditional one-way lectures, and an approach with class participation and frequent interaction. The results of the study found that the interactive teaching methods have an especially profound effect on students with less scientific experience [8]. While both groups of students positively responded to the more active teaching methods, it was the lessexperienced students that saw the greatest improvement in performance.
Another study in Australia analyzed the impact of paid work experience for high school students. This study measured the career maturity of different Australian students, some who had paid work experience and some who did not have such experiences. Career maturity measures a student's readiness to make appropriate career decisions and manage critical tasks associated with career success [22]. The Australian study used Career Development Attitude (CDA) and Career Development Knowledge (CDK) metrics to identify career maturity. This method was derived from the original American career maturity metrics [23]. The study found that students with paid work experience have consistently higher CDA scores than those without. The results also suggest that paid work experience can be associated with increased thoughtfulness in career maturity [24].
The present experiment uses similar background factors as the reviewed studies, but it includes different dependent variables. In contrast to response variables such as academic performance and career readiness, this analysis uses concept map data from students prior to taking a design course. In doing so, the study will identify trends in the ways different students conceptualize product design based on their backgrounds.
This study analyzes data from surveys and concept maps generated on the first day of a third-year engineering design course. The survey asked questions regarding students' backgrounds coming into the course, and the concept maps mapped out the students' conceptualizations of product design. The survey data were then compared with the concept map contents to explore correlations between backgrounds and conceptualizations.
Course context The study was conducted at Stevens Institute of Technology, a private STEM university located in the northeastern United States. All undergraduate engineering students at Stevens follow the university's Design Spine course progression: This is an eight-course series through which students learn and apply different aspects of design in conjunction with other engineering topics. The first five courses are project-based and focus on general engineering topics such as mechanics, dynamics, and materials. The sixth course, Engineering Design VI, is disciplinespecific and is the final course of the Design Spine before students begin the year-long capstone design project. This course brings together topics from previous course in a PrBL experience that mirrors the process students will go through in their capstone project, with more emphasis on instruction and guidance.
The participants of this study were all entering their thirdyear Engineering Design VI course. Survey and concept map data were collected from students in three different disciplines: Engineering Management (EM), Industrial and Systems Engineering (ISE), and Mechanical Engineering (ME). The EM and ISE students took this course together in one combined section, and therefore their concept maps were grouped together.
Data Collection Survey and concept map data were collected from 154 students (125 ME and 29 EM/ISE) during the first week of the Engineering Design VI course. The survey asked about the students' backgrounds and experiences, and the concept maps were generated around the students' internal conceptions of "product design." The data instruments were approved by the Stevens Institutional Review Board (IRB) under protocol 2017-016(21-R1).
To gather data about the students' backgrounds and experiences, a survey was administered on the first day of the course. The survey collected data about previous work experience (e.g., internships, co-ops, and research assistantships), intentions regarding whether to pursue a master's degree, courses that have been completed previously, education level of parents/guardians, and whether they grew up with a parent, guardian, or close adult role model who had an engineering background. Regarding the previous work experience, information was requested about the timing and specific job roles in those work experiences. The complete text of the survey questions and response options are provided in the Appendix.
To measure how students conceptualize product design, they were tasked on the first day of the course to generate a concept map around the central theme of "product design." Prior to constructing these maps, the students were given a brief tutorial on how to construct a concept map, and they constructed a group example concept map on the topic of "personal health." Following this exercise, they were asked to construct their own using the following prompt: Draw a concept map that embodies the concept of "product design." There is no right or wrong answer, as we just want to explore how you think about product design and the factors that are important to consider in product design. Please use the entire 15 minutes to add/revise elements and refine the structure and connections. Remember, concept maps include concepts (in boxes) and relationships (along arrows).
As this course took place during the Spring of 2021 in the midst of the COVID pandemic, the course was held entirely over Zoom. Therefore, the students constructed their concept maps digitally using the Lucidchart online diagramming software [25]. The resulting concept maps were submitted, anonymized, and subsequently analyzed.
The concept maps were analyzed in two ways: structurally and thematically. The structural analysis viewed each concept map as a quantitative network, looking at the number of nodes, the number of links, and the network density. The thematic analysis involved categorizing the contents of the nodes and evaluating the relative presence of different themes. This resulted in dependent variables for subsequent statistical analyses, and four binary independent variables from the surveys were used to evaluate their predictive capabilities: academic major, work experience, plans to enter a master's program, presence of an engineering parent or role model.
The structural dependent variables included in the analysis were node count, link count, and map density. The node count is simply the number of concepts the student included in their map, and the link count is the number arrows. Network density is a ratio of actual links to potential links in a concept map, given the number of nodes. Density (ρ) is calculated using Equation (1), where e is the number of edges and n is the number of nodes.
Factorial Analysis of Variance (ANOVA) tests were conducted to analyze each dependent variable with respect to the four categorical independent variables [26]. These tests identified whether each independent variable had a significant influence on the dependent variable. Furthermore, it provided insight into interaction effects for combinations of independent variables that might have otherwise been missed using other methods such as t-tests and regression analysis. The resulting analysis identifies which factors significantly influenced the structure of the concept maps and to what extent.
In addition to analyzing the structure of the concept maps, it was critical to also look into the themes present. One of the most common methods of evaluating concept maps is to identify the presence of certain root themes and terms within the maps [27]. When analyzing concept map content in engineering design contexts, there are a variety of different methods. Some research indicates that words should be broadly categorized into three buckets: technology, business, and people [28]. Other researchers have taken a more specific approach, categorizing words in more specific themes including things like design knowledge, theory, and finance [29]. In the study reported here, these two methods were combined, allowing researchers to search for the presence of broad themes and also specific categorical terms. In a previous study as part of this project [30], the terms that appear in product design concept maps were categorized into three thematic areas, each with four associated sub-themes, summarized in Table 1.
Business Society
Prototyping & testing Operations Ethics
Project management
The analysis began by building a comprehensive list of every word from every concept map. Each substantive node was manually categorized and sub-categorized. This process resulted in a dictionary of every term that appeared in any of the concept maps, along with that term's theme and subtheme. Then, the percentage of terms in a given concept map in each theme and sub-theme is calculated. For example, if a concept map has ten total terms, and three of them were categorized as Engineering, the resulting Engineering term ratio is 0.30.
Thematic and sub-thematic ratios were the dependent variables in the ANOVA thematic analysis tests. The goal was to identify which, if any, of the background factors led to significant differences in the ratios of specific themes and subthemes.
Limitations There are notable limitations to this study. First, the sample size is limited to the 154 students who participated in the study. With four independent variables plus their six interaction effects, this was a limited sample that was constrained by the participant pool. Additionally, since there is no consensus on what specific topics, links, and themes should be present in a "correct" concept map of product design, this analysis does not evaluate the quality of student understanding of product design. Rather, the study provides insight into what types of themes students of different backgrounds include in their maps, and what gaps these students may have in their initial understandings.
The results show how the structural and thematic contents of student concept maps correlate with a variety of background factors. Individual influences of independent variables were studied, along with interaction effects between every pair of independent variables. Table 2 summarizes which independent variables (columns) exhibited significant (p < 0.1) correlations with each dependent variable (rows), with the corresponding pvalues when applicable. Of the independent variables, academic major was a significant factor in the highest number of dependent variables (seven). The interaction between academic major and presence of an engineering role model did not significantly explain any differences.
The structural analysis revealed several significant differences among the dependent variables, all relating to the students' field of study and their prior work experience. Table 3 shows the one-way ANOVA tests with significant differences, corresponding with the top three rows of Table 2. ME majors included significantly more nodes than their EM/ISE counterparts. Additionally, students with prior internship or coop experience included significantly more nodes in their maps on average, while also having lower network densities. The twoway ANOVA tests with significant findings are provided in Table 4. Edge count and density interactions were found between prior work experience and enrollment in master's programs. Students with no prior work experience who had plans to enter a master's program had the most edges in their concept maps on average. Students with no prior work experience with no plans to enter a master's program had the most dense concept maps on average.
Of the four major independent variables included in this analysis, only the academic major had a significant impact on the top-level themes found in the concept maps. Student majors had a significant impact on all three themes: Engineering, Business, and Society. ME students used a significantly higher percentage of Engineering terms than EM/ISE students, whereas EM/ISE students had a higher ratio of both Society and Business terms. Since these ratios are correlated-e.g., as one goes up, others must go down-this is not surprising. These results are provided in Table 5.
Experimental parameters with significant effects shown; empty cells indicate no significant correlation, and numbers represent the p-values of significant effects (p < 0.1)
Sub-Thematic Analysis Because the sub-thematic level includes 12 categories, there are inherently fewer words per category than with the broader themes. There is also a wider array of usage with subthemes. Some sub-themes were included often, while some were rare. For example, the average student included over 20 percent Conceptual development terms in their maps, whereas the average student only included 0.2 percent Governance terms.
Several significant results were found within the subthematic analysis, with select results from Table 2 expanded on in this section. Much like in the thematic analysis, students' field of study influences several sub-thematic categories. Regarding the first-order interactions, EM/ISE students used significantly more Operations and Sustainability terms than their ME counterparts. These are shown in Table 6.
Additionally, several interaction effects were identified among the independent variables that significantly influenced some of the sub-themes. One involves student majors and their work experience, where it was discovered that EM/ISE students with no prior work experience used the most Ethics terms in their terms. This can be seen in Table 7.
Another interesting result from this analysis is in the Project management category. When looking at students' work experience and the presence of engineering role models, there was a significantly higher Project management ratio among students who had engineering role models at home and also had prior work experience.
The results provide insights into the primary research question posed at the beginning of this article. The analyses identified significant ways that student backgrounds and academic profiles are correlated with conceptualizations of product design.
The structural analysis resulted in several unexpected and difficult to explain correlations. When looking at the number of nodes in the student-generated concept maps, ME students averaged more than their EM/ISE counterparts. Additionally, students with prior work experience included more terms than those without prior work experience. One common explanation for such findings is that more nodes represent higher knowledge or understanding [17,18]; in this case, the results may indicate that ME students and those with work experience are relatively more knowledgeable about product design.
The network density results were more difficult to interpret. There was a significant interaction effect discovered between enrollment in master's programs and previous work experience. Of this group, students not enrolled in a master's program and also with no work experience had the highest network density. This Significance levels: * p < 0.10, * * p < 0.05, * * * p < 0.01 result is challenging to interpret, but it may be related to more experienced students having the ability to concisely portray their ideas. Studies have pointed to the importance of being concise in messaging, so students with more professional experience have likely had more practice communicating in a more efficient manner [31]. Students with no work experience or plans to pursue a master's degree have likely had fewer opportunities to practice concise messaging, which may explain their propensity to generate denser maps than those students with more experience. The edge count results similarly had interactions between the master's program and work
experience, but in this case students with no work experience and master's degree intentions had the highest number of average edges, while those with no work experience and no master's intentions had the lowest average edge count. As there is a direct mathematical correlation between edge count and density, it is unclear how this is related to the previouslymentioned trends.
In response to the research question, it is evident that factors like academic major and prior work experience influence how students conceptualize product design. ME students and students with work experience tend to conceptualize product design with more breadth, including a greater number of concepts. This differs from EM/ISE students, and also those students without work experience, who tend to conceptualize product design at a more abstract level, using fewer terms to express their conceptualizations. The factor with the highest structural influence on student conceptions of product design was work experience, followed by academic major.
The thematic analysis revealed various insights into student conceptualizations, based on the types of terms they chose to include in their maps. The most significant observations were those that pertained to student majors. When considering toplevel themes, EM/ISE students included a higher percentage of Business and Society terms than ME students, while the ME students included relatively more Engineering terms. Breaking down further into these Business and Society thematic trends, EM/ISE students had significantly higher ratios for both Operations and Sustainability terms, which likely drove the higher percentages in the large themes. These differences may be explained by the varying course curricula (prior to the Engineering Design VI course) between ME and EM/ISE students, as well as the predispositions of students who choose to pursue these major fields of study. By the time students reach their sixth academic term, EM students have taken courses such as project management, accounting and business analysis, and logistics and supply chain management, while ME students have taken courses such as fluid mechanics, design of machine components, and ME thermodynamics. When only looking at student majors, these course differences may explain the observed disparity between term usage ratios. Another explanation for this finding is that students choosing to study ME are more inclined to focus on the technical engineering topics, whereas those choosing EM and ISE tend to think more about the broader system, including non-engineering factors.
A less obvious but equally interesting result found in the thematic analysis has to do with the usage of Ethics terms. While none of the factors individually explained differences in Ethics terms, two separate interaction effects led to significant differences. The first measured the interaction effect between work experience and academic major. EM/ISE students with no work experience included significantly more Ethics terms than the others. Conversely, ME students with work experience used the lowest percentage of Ethics terms. This observation suggests a possible lapse of emphasis on ethics education within technical environments (both education and job roles). Similar to the previous discussion, the differences across majors may have to do with predispositions to considering non-technical factors. The results may indicate that ME courses prior to the Design VI course spend less time emphasizing ethics, or that EM/ISE students are predisposed to considering ethics in design. Interestingly, students with work experience may not be introduced to engineering ethics principles in their co-op and internship job roles, perhaps because their job roles focus more on the technical experiences.
Looking further into the sub-themes, use of Project Management terms was influenced by an interaction effect between student work experience and the presence of an engineering role model. Students with both prior work experience and an engineering role model showed highest ratio of Project Manage-Copyright © 2022 by ASME ment terms of the observed population. Students with frequent exposure to engineering concepts both at home and through professional work experiences focus more on project management when it comes to product design, whereas those without these influences may not sufficiently consider this critical component of engineering practice.
Considering the number of themes and subthemes which each independent variable from the survey was a significant predictor, academic major appears to be the most impactful variable. Academic major influenced 6 of the 15 categories, whereas work experience, master's program intentions, and role model presence influenced 2, 3, and 1 category, respectively. The highest number of significant interaction terms was at the intersection of academic major and master's program intentions (4 significant effects), followed by master's program and role model (3), major and work experience (2), work experience and role model ( 2), and work experience and master's (1).
Recommendations for Design Education The findings support further analysis of students' prior experiences, both within their academic programs and outside the classroom. One recommendation for instructors of future design courses is to collect data at the beginning of the course, and then tailor the course syllabus to the gaps in student conceptual models. This could be done in a comprehensive way through concept map collection and analysis, as was done in this study. However, this is time intensive, and so instructors may more easily survey their students about their backgrounds and infer learning needs from the correlations revealed in this and other similar studies.
Furthermore, educational institutions may consider implementing more holistic methods of teaching engineering concepts at an earlier level of undergraduate education. Moreover, they may consider devoting additional resources toward promoting internship and co-op experiences, which have significant impacts on broadening student conceptualizations of product design. Such actions may lead to students who are better able to put their technical training into context, and institutions will build a stronger, more wellrounded pipeline of students who can approach design problems in holistic ways.
Future Research Opportunities This research creates a foundation upon which further studies may build. The methodology and findings presented in this paper reveal several opportunities to supplement and expand on this domain. As the study took place in three programs at one private institution with a high proportion of white students, one direction is to expand to a more diverse population. Including students from different institutions and in different fields of study would yield more robust results that may provide additional support to generalize (or differentiate against) the findings in this paper.
Furthermore, future research may include more depth in the demographic, background, and academic profile variables. In the study reported here, each independent variable was binary (e.g., yes/no, EM/ISE or ME). However, there are further details about the students which could be expanded into additional or more complex independent variables (e.g., type of work experience, education level of parents/guardians). While the relatively small sample size in the present study made this unlikely to produce statistically meaningful results, as the subsets of students would be quite small, a larger sample may make such a follow-up study more suitable. This would also open the door to include additional types of data that could not be reduced to a simple binary response.
Lastly, an opportunity is presented to further refine the methods by which the concept maps are analyzed. Through more advanced network analysis strategies and/or concept map analysis tools, further research may uncover additional findings beyond the dependent variables utilized in this study. Two specific ideas are to investigate specific node pairings and to research trends in the ways certain themes connect with other themes within the concept maps. Furthermore, research to rigorously develop an industry-based expert concept map around the topic of product design could enable a dependent variable that measures concept map quality in a meaningful way.
This study took an exploratory approach to identify in what ways student backgrounds may influence their conceptions of product design. Structurally, the most influential factors included academic major and work experience, such as through internships and co-ops. Generally, ME students and students with prior work experience included more nodes in their maps. However, students with prior work experience also created less dense concept maps, with fewer connections per node. Thematically, the same two factors showed the most significant findings, but there were also significant differences among students based on their enrollment in master's programs and the presence of engineering role models at home. EM/ISE students tended to include more Business and Society terms in their maps, whereas ME students used more Engineering terms. Students with engineering role models present at home and prior work experience included the highest ratio of Project Management terms. findings provide insights on the gaps in students' knowledge about holistic product design, the ways that outside factors and experiences may or may not be able to fill those gaps, and a baseline upon which educators can use to design improved engineering curricula for today's students.