Research has demonstrated that there is a gap between the problem types engineering students are given in academic settings and the workplace. Problems encountered by professionals in the engineering industry are typically described as ill-structured [1], wicked [2], [3], ill-defined [4], complex [5], or workplace [1] problems. Although the terms used vary, they have similar meanings; they describe problems that have no right and wrong solutions, are not easily defined, have no clear rules, and typically require iteration to develop a final solution.
Engineering classroom problems, are defined as being typically given by the course instructor in a well-documented, well-defined written form, and having a single, "correct" solution [3]. Workplace problems, however, typically include only smaller amounts of information regarding how to approach the problem, often with vague or conflicting information and nonengineering constraints [3]. To better prepare engineering students for their future engineering careers which require skills to solve such ill-structured problems, the Accreditation Board for Engineering and Technology (ABET) [5] has underlined the importance of integrating solving these engineering problems into the civil engineering curriculum. ABET identifies complex problem solving skills (Outcome 1) as one of its learning outcomes, as well as the importance of understanding non-engineering (i.e. non-technical) constraints (Outcome 2). To further support and improve the civil engineering curriculum moving forward, how these ill-structured problems are perceived and interpreted by engineering students should be examined, as well as the factors that influence those perceptions. This is important, as students should understand the nature of illstructured problems and what types of problems they will face in workplace environments.
Although considerable research has been conducted regarding ill-structured problems, particularly how engineering students and practicing engineers approach such problems [1], [6][7][8][9][10][11][12][13][14][15][16], less work has been done on how these problems are perceived by undergraduate engineering students. In one study, Pan and Strobel [17] developed a survey to examine how engineering students perceived the level of difficulty of ill-structured problems. Ninety-three engineering students' responses indicated that solving ill-structured problems were more difficult and challenging than classroom problems, particularly among non-white students. In another study, McNeill et al. [18] investigated undergraduate engineering students' beliefs about solving classroom problems, including two closed-ended and two open-ended problems. Their findings revealed that students identified a distinction between workplace and classroom problems and believed that different problem solving approaches are required when solving different types of problems. Students described ill-structured problems as complex and having a number of variables. Students also reported they felt that they received little exposure to these types of problems during their undergraduate studies. While some students believed that they enjoyed the flexibility of illstructured problems, some felt uncomfortable due to the problems' ambiguity. [19] explored how engineering students perceived problem solving and their future goals, and found that students used different problem solving approaches depending on how well the problem aligned with their future goals. They also found that students felt that context played an important role in their problem solving processes. In addition, Jocuns et al.'s [20] findings showed that freshmen had little idea about what type of work they would be doing when they graduate. In another study, Adams et al. [21] investigated engineering students' and professionals' perceptions of problem solving and creativity and found that both students and professionals had a hard time defining creativity. Participants tended to see creativity as "thinking out of the box" and an innate ability. These studies show that although students are aware of different problem types, they find solving ill-structured problems more difficult and challenging and believe that they are not exposed to workplace problems as much as well-defined, classroom problems. Given that engineering students will solve ill-structured problems when they work in the engineering industry and that these problems are different from classroom problems, it is important to understand students' perceptions about solving ill-structured problems.
The goal of this study is to explore the perceptions of undergraduate students in civil engineering regarding solving an ill-structured problem as part of an ongoing larger research study. This is guided by the following research question: What factors do students perceive to influence their solving of an ill-structured civil engineering problem? Based on students' responses to a series of questions after solving an ill-structured engineering problem, we develop themes of these responses inductively, with the goal of using these insights to help engineering educators, to better teach students ill-structured problem solving skills throughout their curricula.
Eighteen undergraduate students in civil engineering at two universities in the Midwest were asked to solve an ill-structured problem and interviewed after solving the problem. Fourteen of the students came from a large doctoral university and four of them were students at a small university. As part of a larger study on ill-structured problem solving, only seniors and freshmen were selected for this study to better understand similarities and differences in students' beliefs in terms of their class level within a civil engineering program. Of these 18 students, 14 were seniors and four were freshmen. Characteristics of participants are provided in Table 1. These characteristics are taken from students' responses to a demographics survey. All participants took part in the study on a voluntary basis and were given $25 for their participation.
Each student was asked a series of thirteen questions, seven of which were related to their perceptions associated with an ill-structured problem and the associated problem solving process as shown in Table 2. These questions were the same for all participants, and were developed by research team members based on the review of responses in the pilot study and the literature [17], [18]. These questions covered topics including what impacted problem solving abilities, the time to solve the problem, and evaluation of solutions by students. These interviews lasted approximately 20-25 minutes in total across all questions. Additional, clarifying questions were asked, as needed, when students did not answer the question, or provided general answers. Each interview was recorded for transcription purposes. All participants also completed a demographics survey (Table 1). Information about demographics was included to provide further clarity on who was studied in the research, although it was limited in terms of sample size.
For context, it is important to note that prior to answering the interview questions considered in this paper, students completed a problem solving task where they were asked to solve an ill-structured engineering problem over a 30 minute period. Verbal protocol analysis (VPA) was used to collect data as participants solved the problem. VPA has been widely used in the literature to examine problem solving [9], [15], [22][23][24] which require participants to think aloud while working on a problem. Participants were given pieces of paper to solve the problem without any use of external resources such as the Internet or textbooks. The problem utilized focused on removing trash from a river; this problem was developed by research team members along with other ill-structured problems [25]. This problem was selected based on the project's advisory board members' comments and discussions among the researchers. However, we only focus on interview data for the purpose of this study.
Data for this study are from eighteen transcripts of the student interviews. After interviews were conducted, audio recordings of interviews were transcribed for data analysis and each transcript was coded by three coders. Open, axial, and selective coding outlined by Corbin & Strauss [26] were used to develop a codebook. During the open coding stage, each transcript was read several times and using the participants' own words, initial labels were created by each team member separately that summarize students' responses to each interview question. Each line and/or sentence within each transcript was read to understand what it was about and describe what was described within that transcript. During the stage of axial coding, categories were related to their subcategories to identify the relationship between open codes along with further development of categories. For example, initial labels such as "it is not the best solution", "require further testing", and "long-term concerns about their solution" were related to the overall code of solution critique. Through axial coding, we were able to merge similar codes by reducing the number of open codes. During the following stage, selective coding, a core category was selected around which related subcategories were unified. As Corbin & Strauss [26] discuss, this was the last stage in the process of codebook development. One example of selective coding used in the study is that rating of solution was selected as one of the core categories that consisted of problem critique, personal critique, and solution critique as subcategories of the core category. Interview questions also informed the development of core codes during the stage of selective coding. Table 3 shows the main code schemes used in the study. The codes were developed by five members of the research team iteratively throughout a semester. Each transcript was coded by three coders who met weekly to discuss each transcript and compare codes. When the coders had a disagreement or a question, they consulted a fourth coder to reach a consensus. 80% inter-coder reliability was achieved. A qualitative data analysis tool, MaxQDA, was used to code the transcripts. After each transcript was coded, all of the transcripts were merged into a single MaxQDA file for comparison. The coders met to discuss the coded segments and construct themes from data. The following themes emerged during qualitative analysis and they will be discussed in the results section. As recommended by American Psychological Association (APA) [27], the generic singular pronoun "they" will be used throughout the rest of the paper to refer to individual participants.
How students rate their final solution, including (a) on a scale of 1-10 and (b) why they rate their solution this way Resources used for solution development
Theme 1. Students tend to rate their solutions as being decent, while recognizing there is a lot of opportunity for improvement.
The average rating of students' self-evaluated solution quality was 6.1 out of 10, ranging from a score of 3 to 8. This includes including an average rating of 5.5 for freshmen and 6.3 for seniors. When asked to justify their rating, students' reasoning can be summarized into three main groups. These included comments on the problem, their final solution, and themselves as a problem solver.
With respect to the problem, some students (27%), all of whom were seniors, stated that the time given to solve the problem was not enough, indicating that they needed more time. One senior student mentioned "If I had more time, maybe I could have fleshed this idea out more." Another senior student also made a comment about not having enough time by saying "I think that, because part of that thing of only having limited time is, I just have to pick something and roll with it, and just build off of that rather than come up with a million obscure solutions and try and narrow it down." These student comments show that many students, seniors in particular, recognize that more than 30 minutes is needed to create a sufficiently detailed solution to an illstructured problem, including the need to add more details to their solutions and consider multiple ideas.
Nearly all students (88%), including four freshmen and 10 seniors, also made one or more comments on attributes of their final solutions as justification for their rating in response to this question. The majority of the students (67%) explicitly stated positive comments on their solutions, using words such as decent, efficient, practical, cost-effective, innovative, and effective. This is shown in the following examples "I think it's pretty easy to implement. All things considered, it's probably fairly effective" and "I think the solutions that I came up with are pretty efficient because the problem that they mentioned here is very storm water management-related and just the general cleanliness of the city. All my solutions are very proactive and active. They will help clean in a short time and in the long term as well." Students, however, also recognized that there was room for improvement in their final solutions. Eleven (68%) had negative comments, including six (33%) that explicitly indicated their solution was not good. Two students reported that their solutions did not cover all the problem requirements, while six thought it was not the best solution. One senior student recounted: "I think it could work, but I think there are probably a lot of variables about it that may not be the best solution." Many of the students that had negative comment, however, also stated positive comments (44%), and vice versa. Some (25%) students, including those that made negative and those that made positive comments about their solutions, also suggested improvements that could be made. These included improvements to aesthetics, the construction methodology, conducting further testing and data collection, and concerns about the long-term performance of the proposed solution. One of the senior students expressed their concern about the aesthetics of the solution stating "I don't think it's very attractive. It would look like a big eyesore in the river. If it doesn't work very well, then it'll just look like a bunch of trash. People will see more trash than they did before because they'll all be collected." All students that suggested improvements were seniors.
In addition, five students (28%) critiqued themselves as problem solvers, most of which were seniors. Three of them commented that they did not know whether their assumptions that they made during the problem solving process were correct or not as expressed by one of the seniors "I think the design, in theory, if all of my assumptions were correct, would be a 9 or 10. I don't actually know if they're correct, so that's why I'm having it lower." Several students also reported that they did not have enough experience with the content of the problem and were not given enough information in the problem statement, which is why they thought their solution deserved a lower rating. In addition, one student stated that not seeing the area given in the problem statement in person made it more challenging to solve. One also expressed that they did not remain on track while solving the problem. These responses of students showed that the problem itself, their perceived quality of their final solutions, and themselves as problem solvers played an important role in the evaluation of their solutions. Overall, students believed that their solutions would work, were decent and practical, but they also stated that they could develop better quality solutions if they had more experience in the area provided in the problem statement, were given more time, and were more confident with their assumptions. This suggests that students have the ability to critique an illstructured problem's solution. It was interesting that only seniors critiqued the problem itself, and it was mostly seniors that critiqued themselves. In addition, while both freshmen and seniors critiqued attributes of their solution, only seniors made suggestions on how the solution could be improved. Seniors also, on average, rated their solutions as higher. Such observations may suggest that seniors have a better grasp on what solving an ill-structured problem entails, how their skills compare to what is needed to solve such a problem, and opportunities for improving their solutions.
It was observed that students indicated that classes they had taken (89%), internship experiences (61%) and personal experiences (56%, including non-internship work experience and background knowledge) positively impacted their problem solving processes.
For classes taken, civil engineering classes including senior design, geotechnical engineering, and water-related civil engineering courses as well as class design projects were mentioned as helping students familiarize themselves with the ill-structured problem solving process. Students stated that in their classes they had been given an ill-structured problem which helped them learn the problem solving process (eight students, 44%), as in the following example: "I was a part of an engineering academy and senior year, we were given the free rein to come up with the problem and fix it. We could come up with anything that we wanted at all and we had to go through the whole process…" A different set of eight students (44%), and 50% of seniors also mentioned that solving a similarly ill-structured problem in class had helped them solve the given ill-structure problem. Six students (33%), including 36% of seniors surveyed stated that solving ill-structured problems in their classes made them feel more comfortable solving these problems and helped them learn to make assumptions. Two of the four freshmen responded that they had not been asked to solve ill-structured problems in their classes. Three seniors and one freshman also thought that high school classes such as algebra and geometry and experiences such as being part of an engineering academy and robotics team in high school influenced their problem solving processes. In addition, five students, including three freshmen, believed that having solved illstructured problems in their classes had no impact on their solutions to the given problem.
In addition to classes, it was found that internship experiences influenced students' problem solving, as discussed by 12 students (67%), all of which were seniors (86% of seniors). Students' internship experiences discussed ranged from working for the state department of transportation (DOT) and railroad companies to engineering consulting firms. Two students indicated that solving ill-structured problems during an internship made them feel comfortable solving illstructured problems. Two students indicated their internship(s) helped them to pre-plan their solution, stated by one of the seniors: "I think that's helped me take a step back and look at it a little more closely before I just jump right into it." One student also stated their internship helped them to use outside resources as in the following example: "… and I realized how many different solutions and how easy it is to call a representative and especially how easy it is to call other people and be like, "What have you used and what products would you recommend?" So, that's probably what I would have done on this." It should also be noted that while 12 students emphasized a positive impact of solving ill-structured problems as part of an internship on their problem solving, four students were not sure if it impacted their problem solving processes. Additionally, freshmen were found to have no engineering-related internship experience.
Personal experiences also played an important role in the students' problem solving processes, as discussed by 56% of students, including eight seniors and two freshmen. Subcategories of such experience included background knowledge, previously living in a polluted area, being influenced by other people, and non-internship work experience. Seven students mentioned their background knowledge. For example, one senior student stated that growing up on a farm impacted their solution: "I grew up on a farm, so I've definitely waited in streams and then that kind of stuff before. That probably influenced like, yes, use posts, use more hands-on materials." Another example of background knowledge was international experiences, as in the following example: "I've had a lot of international experiences. Those have prepared me well for coming up with a best solution given limited data, which is what I felt like this problem had. Because we obviously didn't have the exact layout of the river and where the trash was entering. A lot of those international experiences have helped me a lot." It was also observed that four students were influenced by other engineers and/or family members when solving an ill-structured problem. For instance, one student mentioned that they were influenced by another student's design "Of course, I knew about that one design that this Danish student made and dropped out of school, and he's now millionaire on to a million company over doing this exact same thing, but on a larger scale in the Pacific Ocean. That's not really known, which is kind of sad, but it's a great thing that he's doing. I want to a mere slide like that. It's not the same design, his design is simply just a giant floating tube, but of course, his tube is several miles long, so to log a little bit more practical for it to just be ginormous tube in the ocean, this is smaller social spot."
In summary, it was found that students' personal experiences such as prior work experience and background knowledge in addition to the classes they had taken and internship experiences influenced the majority of students' problem solving processes. These results show that both academic and non-academic experiences starting from childhood helped students as they developed solutions to ill-structured problems. They also indicate the importance of students participating in internships throughout their undergraduate careers, in developing their illstructured problem solving skillsets. It also suggests that there are experiences that the large majority of students have had in their courses that they feel has helped them be able to solve an illstructured problem. Theme 3. Most students state that they would use the Internet as a starting point, specifically to gather more information, look for similar problems, and develop initial ideas related to the problem and that their solution would be different if they had access to the Internet and other resources.
Students were not allowed to use any external resources except a calculator to solve the illstructured problem, but when asked, 12 students (68%), all of which were seniors, stated they would use the Internet. Other resources suggested included using a map of the area (three students), talking to their teacher(s) (one student), using a software program (one student), and visiting the site personally (three students). Of the 12 students who would prefer to use the Internet, responses ranged from "I would google fish", "Wikipedia", "going on a google earth", "I would look up material costs" to "I would google river pollution solutions." Looking up a similar solution/problem on Google was the most popular response among students. While this is not surprising, nor necessarily incorrect, it points to students seeing the Internet as a main source of a diversity of information.
With respect to the purpose of resources they stated they would use, 12 students (68%), expressed that they would look for similar problems that had been solved before. One freshman stated that "I'd probably look at past designs, and then see how effectively they work in similar solutions." The majority of the students chose to look up similar past designs to see what other people and/or cities had done and what possible solutions they had utilized. Four students (22%) also expressed the importance of reducing the impact of their solution on wildlife and the community. Other students suggested using the resource(s) to look up what materials to use and the dimensions of the river/stream, and to brainstorm more ideas. When asked whether their solutions would be different if they had access to external resources, 12 (68%) of them responded "yes". One senior student's response was "… and then maybe I would've had a different idea, but this is my idea without thinking of what something else", while another senior stated "Definitely more put together. I don't know if it would be very different, but more so more well put together. Yes, more. More clarification, I guess. Like it'd be clearer what I meant." One freshman student, however, responded that they would not need to use a computer or software program for such problems because they did not think it would be beneficial to them.
Overall, it was found in students' responses that the majority would choose to use the Internet as their main source of information to look up either similar problems, solutions and/or location specifications, and/or to brainstorm more ideas. They also believed that their solutions would be either different or more put together if they had access to these resources. This points to students understanding that there is a wealth of information available via the Internet and this is the main source they look to for information, and they also recognize the importance of using other information as references in the development of an engineering design rather than creating a design in isolation. It is interesting to note, however, that resources such as design guides, standards, and other resources that a practicing engineer may look to for design guidance were not mentioned or discussed. This may point to an opportunity to increase students' awareness of such resources, where to find them, and how they can be used. In this study students were given 30 minutes to solve the ill-structured problem. However, all indicated that they needed more time to come up with a solution. The amount of time they suggested they would use, however, ranged very broadly for both freshmen and seniors, including from 50 minutes to over a year. When estimating how much time they would need, students also considered a broad diversity of factors, such as whether there would be an actual testing, size of the project, site visit, and asking for guidance from universities and other institutions.
It was observed that four students calculated the time that they would need by breaking down the steps of idea development, testing, and implementation separately. For example, one student stated "I would say, give me a week to actually get a real design with real drawings, and then a year to test it and actually make sure it works." Similarly, another student said "feel like it'll take me, I'm thinking one day to get the design down just on paper to where I'm happy with it. I'm going to want to prototype it. Let's, say a day to build the prototype. I feel like I'm overestimating it but I feel it is better to overestimate than underestimate. Let's say a day to build the prototype, probably half a day to run the tests, to run all the tests I want to a set and then and if I'm happy with it, probably another half a day to just come check over everything and finalize my design and get to a point where it's presentable." It was also observed that most students chose to use the time they suggested to research (33%), brainstorm ideas (22%), and build and/or test a prototype (33% and 39% respectively).
These student responses showed that the majority of the students felt that they needed more time than the allocated 30 minutes. However, the most common amount of time ranged widely, indicating there may be misconceptions or lack of knowledge in how long an engineering problem can take to solve. While the most commonly discussed answer was expressed in days, this is likely shorter than the amount of time needed in a real-world scenario. Theme 5. Creativity rating identified by students and the rating of their final solution showed similarities signaling a relationship between self-rated creativity and solution.
Students were asked to rate their creativity and it was found that their average self-rated creativity score was 6.1 out of 10, ranging from scores of 1 (least creative) to 8. This includes an average score of 5.8 for freshmen and 6.3 for seniors. After stating their suggested scores, five students identified themselves as creative, while nine, including three freshmen, thought they were not creative. When asked whether creativity impacted how they went about solving the problem, 13 students (72%) stated that creativity influenced their problem solving processes. We also noted a positive correlation between the self-rating of their solution and their self-rated creativity score (correlation coefficient of 0.24).
Students who identified themselves as creative suggested that this helped them come up with both multiple solution options, and ultimately an original/unique solution. One senior associated creativity with developing original ideas and stated "I think creativity is everything about coming up with original ideas, so it absolutely influenced." Likewise, a freshman thought their idea was unique, although they came up with only one solution. Three students said they were very creative because they approached the problem from different angles and had a lot of ideas, stating "I would say that I am creative because I can look at different ideas and then pick the best one" and "I like to list a bunch of solutions and list their pros and cons, not just like go with the first one that comes to my head."
Students who identified themselves as not creative reported that they did not develop multiple solutions, were realistic but not innovative, used existing ideas, and their solutions were not original. Four students, similar to those who did develop multiple ideas and did indicate they were creative, associated creativity with developing multiple ideas. For example one senior felt they were not creative, as they did not come up with alternate solutions, stating "because then I would assume if you're creative enough to come up with multiple solutions, they would all be different and you could figure out which one's best. I'm just not the creative type that would sit here and think of multiple solutions." In addition, three students felt they were very analytic, realistic, and data-driven; therefore they did not develop "crazy" ideas. One student expressed "Solid five because I am realistic, I try to create things that will work physically so I think that's what's holding me from being really creative because I don't throw any crazy ideas around saying, "This would be really cool." It won't really work because of gravity." Similar to this response, another student reported that they always wanted to be safe more than being innovative.
In summary, these student responses indicated that most of the students associated being creative with being innovative, developing "crazy" and original ideas, approaching a problem from different angles, and developing multiple solutions. They felt that their creativity influenced their problem solving processes and solutions. It also suggests that students' self-rated scores are related to students' self-rated level of creativity.
This study was intended to explore the perceptions of undergraduate civil engineering students about solving an ill-structured problem and what factors they think influence their solving such a problem. This study demonstrated that the majority of students tended to rate their solutions as practical and decent, but at the same time they felt there was significant opportunity for improvement. In general, students believed that internships and classes in which they were asked to solve ill-structure problems and personal experiences with complex problems helped them solve the given engineering problem. A few students stated that they were not familiar with the problem, which demonstrated that context played an important role during the problem solving process.
Another discovery from our study was that students believed creativity played a role in their problem solving processes and influenced their solutions. They considered creativity as formulating multiple ideas which are "innovative" and "out of the box", which aligned with the findings of Adams et al. [21]. In addition, it was found that students felt they needed more than the allocated time to solve the problem and thought if they could use external resources such as the Internet, their solutions would be different or more "put together". It was observed that a few students paid attention to protecting the wildlife and community as they were formulating a solution and demonstrated an ability produce solutions with consideration of non-engineering constraints, which is one of the student outcomes designated by ABET [5]. It was also found that it was seniors who critiqued the problem itself and made suggestions on how to improve the solution, which indicated that seniors may have a better understanding of the problem solving process compared to freshmen. Our findings suggest that if given more time and resources along with opportunities to help students increase their creativity, students believe they can produce better solutions.
This study can help to inform engineering curricula and educators who intend to incorporate ill-structured problems into their classroom and help them better understand what factors influence engineering students' beliefs about solving complex engineering problems. It should be noted that students' grades and academic performance were not taken into account when analyzing the results. In future studies it may be beneficial to consider these factors in relationship to the collected data. Another limitation of our study is our sample size, which is limited, particularly for women and freshman. As this is part of an ongoing study, we will continue collecting more data from female and first-year students. This research effort will continue as part of an ongoing research project and the research team will collect more data interviewing also sophomores and juniors as well as more freshmen and analyze results across participant groups. Ultimately, our goal is to evaluate students' solution quality and compare results with findings from this study. For future research we also will consider faculty and practicing engineers' beliefs about solving an ill-structured problem and compare these results to those of this study.