In order to effectively solve complex problems in design and manufacturing, students need to develop metacognitive skills. This research suggests measuring metacognitive skills in engineering students by having them participate in hands-on and virtual reality activities that are related to design and manufacturing. The study uses common established measures of metacognitive and problem-solving skills. Students took different measures to assess their conceptual knowledge, problem-solving skills, and metacognition. These measures include:
1) The Metacognitive Activities Inventory (MCAI) [1] 2) The Metacognitive Awareness Inventory (MAI) [2] 3) The Metacognitive Scale (MCS) [3] 4) The State Scale of Metacognition (SAQ) [4] For MCAI, five of the eight categories found a negative association between the pre and post assessment. We also found that procedural, conditional, planning, information, and comprehension to be lesser after the simulation. The Cronbach's alpha of the measure was between 0.77 and 0.89 which suggests that there was a good reliability of the scale.
For MCAS, we also found a similar lack of a positive change in this measure. In this scale, we found a Cronbach's alpha of 0.86 in the pretest and a Cronbach's alpha of 0 .87 in the post test. No significant difference was found between any of the categories before or after the simulation despite the good reliability.
For MCS, we found no significant difference between the pre and post with a Cronbach's alpha ranging from a pre-survey reliability score of 0.75 and a post survey reliability score of 0.94. All three scales are considered trait metacognition scales.
For SAQ, there is a discussion of metacognition existing as either or both a trait and a state. Other constructs in education share this dichotomy between trait and state [5]. The idea that metacognition is a slowly changing trait or a temporary state that can be imposed by an instructional strategy is in question. There seems to be evidence that metacognition is a state that incorporates many dimensions such a problem-solving, strategies, awareness, and planning.
In SAQ, which we consider a state measure, it was found that there is a positive change in all four subcategories: Awareness, Cognitive Strategy, Planning and Self Checking. Future iterations of the experiment will focus on including a pre and post SAQ measure along with measures of flow state and problem-solving ability.
Other measures the students took include NASA Task Load Index (TLX), Revised Purdue Spatial Visualization Tests (PSVT-R), Flow State Scale (FSS), and Task Analyzer Questionnaire (TAQ).
While metacognitive theories have expanded to fields outside of education such as how metacognition impacts the workplace and how to use metacognitive theory to improve work [6], metacognition is still a staple in education research as an approach to enhancing and understanding student learning through activities that improve a student's metacognitive skills [7]. Metacognition, or "thinking about thinking", is positively correlated with problem-solving skills and is essential to a student when it comes to learning skills needed to excel in many different fields [8]. In the development of an expertise, metacognition is invaluable in understanding the differences in each person and each field [9].
Research has shown that there is a gap of knowledge between metacognitive theory and putting that knowledge into practice in higher education [10]. Metacognition is essential to collaborative learning [8] and is an important quality for learning required skills in different fields. In the development of an expertise, metacognition is invaluable when it comes to creating an expertise and understanding the differences of metacognition in each person [9]. Metacognition, through monitoring and control of our abilities, influences decision making in our complex world. When it comes to researching metacognition in learning environments, there are complementary processes between the educator, the student, and the environment. Understanding these dynamics is complex as a change in one creates changes in the others. Changes are not always beneficial to student [8]. In addition, it is unclear if metacognition is a temporary state or an enduring trait [11]. Metacognition can enhance problem-solving skills [12]. Problem-solving research suggests that there are several different types of problem-solving. It is thought that as novices become experts, they intuitively begin to understand when to use what type of problem-solving strategy. The link between problem-solving and metacognition has been established in Kim and Kim [13]. In their study, they explored the types of problem solvers in connection to metacognitive awareness. They identified four types of problem solvers based on students' talking aloud.
During problem solving, solvers often reach a flow state. A flow state is when an activity is so engaging that the motivation for engagement is the activity itself. One may say that this is the ultimate metacognitive state where the immersion in the activity inspires the person to seek more information and regularly check what is known and not known during engagement. Engeser and Schiepe-Tiska [14] suggest that this is a common occurrence in artists who work without rest or food until the creation is finished. During the creation process, the artist engages two types of metacognition: the skill knowledge and the creation knowledge. In this sense, it could be that other types of problem solvers engage different types of metacognition as indicated by flow rather than by a metacognition scale.
One of the most cited researchers in flow is Csikszentmihalyi [15]. Csikszentmihalyi describes the merging of action and awareness, the centering of attention, the loss of self-consciousness, the feeling of control, coherent demands, and the autotelic nature as indicators of a flow state during an activity. The last two items; the coherent demands and the autotelic nature describe the smooth processing of information from the environment to the creator. In future work, the addition of the distortion of time was added. In Jackson and Marsh [16] they developed a survey measuring these components in different subsets.
As in metacognition, researchers are split as to whether flow is a temporary state or an enduring trait. Engeser and Schiepe-Tiska [14] suggest that further work needs to be done to examine the differences between a temporary or enduring flow state. The authors suggest that differences between individuals and between tasks are expected as different types of optimization required to reach the flow state depends on prior experience and motivation. As Engeser and Schiepe-Tiska [14] noted, there is a role of metacognition when individuals achieve flow; general metacognition regarding emotion/motivation and goal restructuring.
The literature search conducted to develop the study found four articles on eye tracking and metacognitive monitoring. These were by Merten and Conati [17], Orquin, Ashby and Clarke [18], Bondareva, Conati, Feyzi-Behnah, Harley, Azevedo, and Bouchet [19], and Zhou and Ren [20]. These four articles, especially Orquin, Ashby and Clarke [18] outlined the use of eye tracking to measure attention and metacognition during a problem-solving task. The eye tracking studies had a median number of participants of 18 (SD = 18). There were 28 references that contributed to our understanding of metacognition. Of these, there were two that discussed metacognition and problems solving [13,21]. Chalmers [21] focused on group metacognition while the Kim and Kim [13] article focuses on cognitive style in idea generation. Within these studies there were two types of studies: survey creation and observational studies. In the survey creation studies, the average number of participants was 298 (SD =219), In the observational studies, the average number of participants was 31 (SD = 17). There were 30 references that contributed to our understanding of problem-solving and 11 that contributed to our understanding of signal detection theory as it is related to analyzing the eye tracking data. Some of these studies used a large sample with an average of 115 participants (SD = 140). Other studies used a small sample size, average of 20 (SD = 12). Across all articles, the majority were published in psychology journals, books and conference proceedings with 75 from psychology and 6 from engineering.
In this study, there were two hypotheses. Hypothesis 1: Does the design and manufacturing simulation activities affect students' metacognition in terms of monitoring, awareness, planning, self-checking, or strategy selection. Hypothesis 2: Are there other components of metacognition that are affected by this activity? The activity used in this study was inspired by an industrial engineering class activity taught by one of the authors. The team develop hands-on and virtual reality manufacturing simulations that are conducted by students individually and in groups. We collect data via traditional paper-based measures as well as eye tracking technology. This study only presents the results and analysis from the paper-based measures. The eye tracking data will be analyzed utilizing signal detection theory and presented in future research publications.
We tested a total of 34 undergraduate engineering students who were invited to participate from an introductory engineering design course. The study had institutional review board (IRB) approval, all students signed an informed consent, and they were compensated with gift cards. Of the 34 participants, 32 were male and 2 were female students. All participants stated that they were interested in pursuing a career in engineering. The average age of the students was 18.46 (SD = 3.7 years). The data was collected by semester with the first semester comprising a group of 24 participants and the second semester had 10 participants. After the analysis of the first semester's data, changes were made and submitted for IRB approval. The changes were approved and then implemented with the second group of the participants.
In both semesters, students built a car toy with small plastic bricks either in groups of 3-4 (mass production) or individually (craft production). The steps for the car toy assembly are shown in Figure 1. In the mass production simulation, the students were timed at 20 minutes. In the craft production paradigm, students were randomly assigned to either the physical build or the virtual reality build of the plastic brick car toy in a craft production simulation. The time for the craft production activity was also 20 minutes. The two simulations were two weeks apart. Students took a measure of their understanding of the key concepts in the simulation before and after each simulation. In addition, the students took NASA TLX [22], PSVT-R [23][24], FSS [16], TAQ [25], and several other metacognitive measures (see Section 1).
In both simulation activities, students build the car toy according to a set of customer requirements shown in Table 1. The simulation activities also require that all the tasks are performed by one student for the individual activity (craft production) and by four students for the group activity (mass production). The student(s) need to minimize the total cost of producing the car toy while satisfying the requirements of the customer. Hence, there are four main functions: design, sourcing, manufacturing, and inspection. The simulation also involves a customer and a supplier (see Figure 2). The descriptions of the four jobs are as follows: (1) Design Engineer: the design engineer will translate customer requirements into specifications and design the product based on the customer needs. The design engineer will then create a drawing for the product design to be used by the manufacturing and sourcing engineers, (2) Manufacturing Engineer: the manufacturing engineer will identify and design the manufacturing processes for producing the product based the design created by the design engineer. The manufacturing engineer will then assemble the Lego car once s/he gets the parts from the sourcing engineer, (3) Sourcing Engineer: the sourcing engineer plans and purchase the raw materials (plastic bricks) that will be used to produce the car toy. The sourcing engineer will provide the manufacturing engineer with bill of material along with the costs of the parts, (4) Quality Engineer: the quality engineer develops a system to ensure the products are designed and produced to meet customer requirements. The quality engineer will test and inspect the final products to determine if the customer requirements are met. For the craft production activity, each student develops his or her own design and may negotiate with the customer before finalizing the design and starting the production. In the mass production activity, each student in a group develops one design and the group evaluates the designs and select the best one. Figure 3 shows sample designs for the car toy.
Figure 3. Sample designs for the car toy 3.3 Materials. Students used plastic bricks in the physical simulations and instructions of how to create the car toy using a mass production method as partially shown in Figure 1. The instructions asked that the students create a car with specific weight, color, and cost. Each plastic brick was assigned a cost. In the virtual reality simulation, a similar approach was used but each instruction sheet used the craft production method that focused on one person building the same sort of car.
Figure 4 shows sample pictures for the simulation activities and Figure 5 shows sample car toys.
Simulation kit for hands-on activity A researcher testing virtual reality simulation Checking car weight in physical environment Checking the car weight in VR environment
In both simulations, students were asked to build the same car either as a group or individually. We found that if students did the craft production first, then they misunderstood the mass production directions and would build the cars individually again in the mass production. For this reason, we asked students to do the mass production simulation first.
Students took the NASA TLX to explore which parts of the two simulations were most challenging (Figure 6). Overall, it was found that both simulations were the most taxing in terms of temporal (time) and effort. Within the craft production activity there was a physical simulation and a virtual reality (VR) simulation (Figure 7). Students were in one or the other version. Again, there was a difference in how they perceived the virtual or the physical simulation. Surprisingly, none of the students in the virtual simulation rated it as frustrating. When students rated the mass simulation or the craft simulation, their ratings overall were very similar. A paired samples t-test reveals that there were no significant differences t (7) = 0.4595, p > .05. The mean NASA-TLX rating for the craft production was 11.142 (SD = 3.777). The mean rating for the mass production was 11.758 (SD = 2.849). We expected that if our population was representative of most engineering undergraduate populations, then our sample would have a mean score of 50% correct or greater. The range on the revised PSVT-R was between 33% correct to 94% correct with a mean of 73% (SD = 0.08%). In the first 24 students, there was a ceiling effect in items numbered 5, 6, 11, 12, 14, 15, 19, 20, 21, 23, 24, and 25. For the next semester of students, these items were omitted. In the second semester of students, they had an average of 77% (SD = 19.5%).
In order to measure how much the students learned during the simulations, we asked eight content-based questions before and after each simulation. The content knowledge questions were different for the mass simulation and the craft simulation. In the first semester, we found a ceiling effect of the questions. Students had a mean increase of 0.48 points (SD = 0.37 points) between the pre-concept knowledge and the post-concept knowledge.
In the second semester, nearly all students increased in knowledge from the pre simulation concept check to the post simulation concept check as shown in Table 2 with a mean score of 65% (SD = 0.16%). For the craft production, there was a mean score of 62.5% (SD = .37%). In the repeated measures ANOVA below, both mass and craft were significantly different. Between the students, there was also a significant difference indicating that student performance varied as a result of the simulation. mass simulation had an F (1,8) = 87.51, p = 0.0001, partial eta squared = 0.92, craft simulation had an F (1,8) = 38.87, p = 0.0001, partial eta squared = 0.83.
In the first semester, for MCAI, students rated their agreement with 27 statements on a 5-point scale of 1 = strongly disagree, 2 = somewhat disagree, 3= neither agree nor disagree, 4= somewhat agree, 5= strongly agree. For each question, 1 was the possible lowest score and 5 was the possible maximum score. In the pretest, we found a Cronbach's alpha for the MCAI scale from this current sample to be 0.77. In the post-test we found a Cronbach's alpha for the MCAI scale to be 0.89. We found that the answers did change significantly from pre to post but in the wrong direction [t (28) =-2.310, p = 0.029, r (28) = .551, p = 0.002]. According to the paired t-test for the original study, pre-and post-MCAI scores were not significantly different at the 95% level [2].
Inventory. MAI measure was given to the first semester's participants and was taken by 16 students both before and after the simulation. MAI is a trait measure of metacognition with a series of 51 questions to which the students answer true or false. We chose to use only the separated into the categories of declarative metacognition, procedural metacognition, and conditional metacognition as the other categories were not relevant to our initial hypothesis [1]. Table 3 shows the results.
In the pre activity data set we found a Cronbach's alpha or 0.86. In the post activity data set we found a Cronbach's alpha of 0.87. Because of the lack of change, we omitted this survey from the second semester of the study. [3] survey. This survey measures metacognition according to the following constructs: 1. metacognitive knowledge, metacognitive experience, metacognitive goal orientation, metacognitive strategy, and metacognitive monitoring. In this Likert scale survey, there are seven points with 1 = strongly disagree and 7 is strongly agree. A Cronbach's alpha ranged from .75 to .94 for each scale. Table 4 shows the PIP results. The mean score before the simulation for the craft students was 5.87 (SD = 1.24), the mean score after the simulation was 5.97 (SD = 1.25). There was a strong correlation between the two sets of scores of r (144) = 0.68, but there was not a significant increase in metacognition after the simulation t (143) = -1.27, p = 0.208. We decided to retain the study for the second semester and give it as a pre the mass production and after the craft production. In the second semester, we examined each construct.
For the second semester, the students took it before the mass and after the craft production. In this case, all measures decreased with the exception of metacognitive experiences. However, the increase in experiences was not large enough to be significant.
In the first semester, SAQ survey was taken once after the students had finished the craft simulation. This was the only state metacognition measure. The students rated their agreement with 20 statements on a 4-point scale of 1 = not at all, 2 = somewhat, 3= moderately so, 4= very much so. For each question, 1 is the possible lowest score and 4 is the possible maximum score. The SAQ measured metacognition in four areas: Awareness, Cognitive Strategy, Planning, and Self Checking (Figure 8). In this sample, we found a Cronbach's alpha of 0.90. In each category there was a maximum score available of 20. Here are the mean scores for each category along with the standard deviation for each category. In our dataset, for Awareness students scored a mean of 83% (raw µ =16.5, SD =2.57); cognitive strategy a mean of 78% (raw µ = 15.6, SD = 2.30), planning a mean of 82% (raw µ = 16.35, SD = 2.76) and self-checking with a mean of 65% (raw µ = 12.9, SD = 2.53).
Figure 8. SAQ mean percentage score in each category O'Neil and Abedi [4] correlated achievement on a mathematics achievement test to determine the construct validity between achievement and metacognition. The authors hypothesized that students who did better on math achievement tests also had better metacognitive abilities. They broke these abilities into four components: awareness, strategy, planning, and self-checking. Awareness was the understanding of one's own thinking process; strategy was the conscious use of a strategy to solve the problem; planning was the understanding of the task before beginning the task; and selfchecking was checking your work as you went along. O'Neil and Abedi [4] found a correlation of .26 to .36 with academic achievement. The reliability scores were in the .77-.81 range.
In the second semester, the SAQ measured metacognition before the mass production and after the craft production. The following table compares the answers before the mass production activity to after the craft production activity to look for a significant change between the two measures. None were significant as shown in Table 5. There was an increase in all categories but not in the feedback and transcendence categories as shown in Table 6. In the challenge/skill balance, the concentration on the task at hand, and transformation of time, students reported a significant change as a result of the simulation as shown in Table 7. This suggests that students did enter a flow state and that problem solving as related to flow is measurable. While the grading procedure in the original TAQ differed, assessing the answers by theme according to grounded theory [26] provided another perspective on the answers. The answers to the first semester of testing is named Spring 2019 and second semester is named Fall 2019. In the Fall, students had not had any engineering design classes; many were straight from high school.
In the Spring, students had been in at least one engineering design class and had discussed initial concepts. For goals, students in Spring group were most interested in the specifications while students in Fall group were most interested in meeting the constraints as (Figure 9). For question 2, students reported a trial and error or step by step procedure to solve problems (Figure 10). For question 3, students reported plastic bricks as the substance used most (Figure 11). For question 4, the Fall students reported kinetic energy most often while the Spring students were unsure what the question meant (Figure 12). For question 5, students reported the idea of understanding how to build cars with plastic bricks or having a pre-existing schema as the information that helped them most (Figure 13). For question 6, students identified the kinds of they used in solving the problem (Figure 14). For questions 7-8, students repeated the themes of specifications, constraints, and optimization as they understood them in the simulations (Figures 1516). 4.10 Overall Discussion. We found in the concept knowledge that students did increase their subject knowledge as a result of the simulation. This demonstrated that they were learning something and experiencing a change in understanding before and after the two simulations. However, hypothesis 1 was not supported and the metacognition surveys failed to detect change.
It could be that the sample size is too small for the effect or that metacognitive change in engineering is more complex than general metacognitive change that might be experienced in a K-12 learning classroom. The sample size of 34 was consistent with other sample sizes in our literature search that had a mean of 31 students. It could be that the short time period of two weeks was not enough time for students to experience a change in their metacognitive state.
Of all the metacognitive measures, the SAQ had the highest reliability with the fewest questions.
As it seems that students experienced survey fatigue, shorter is preferred. In the original articles where the scales were developed, the researchers typically used a pre-post design and measured the change or the reliability of the scale with the goal of students NOT changing their answers. This design is suitable for scale development. Once a scale has been validated, it is a useful way of measurement however with a short time period, it may not be sensitive enough. There are mixed results on the state or trait argument in the construct of metacognition with researchers such as Hong [11] stating that there is a difference between a current increase in metacognition due to an activity (state) and Schraw and Moshman [27] who infer that it is a trait that builds over time beginning in childhood and effected by social learning, individual construction, peer interaction, and collective reasoning. Schraw and Moshman [27] state that measurement problems are endemic with a construct as complex as metacognition.
For hypothesis 2, support for problem solving was found in the Flow State Scale [16], the students reported aspects of problem solving associated with intense concentration. This is consistent with the literature on problem solving and suggests that either the metacognitive measures were not sensitive to this change or that problem solving is a pre-condition to emergent metacognition in a developing engineer. In the TAQ, there were several themes that indicated students were developing their problem-solving skills.
This study presented a simulation-based assessment of metacognitive skills in engineering students by engaging them in design and manufacturing hands-on and virtual reality activities. The study provides insights that will contribute to fields of engineering education and design and manufacturing. We conclude that problem solving skills contribute to metacognitive skills and may develop first in students before larger metacognitive constructs of awareness, monitoring, planning, self-checking, and strategy selection. Problem solving skills and expertise in solving engineering problems is needed before other skills emerge or can be measured. While we are sure that these students have awareness as well as the other metacognitive skills in reading, writing, science, and math, they are still developing in relation to engineering problems. More work needs to be done to examine how problem solving relates to metacognition as well as developing finer scales to measure metacognition in engineering students. The team collected eye tracking data from the virtual reality activities and future work of this study will focus on analyzing this data to determine if students are attending to the correct elements while solving engineering problems in the virtual environments.