Complex problemsolving is a vital skill prevalent to thrive in the workforce along with creativity and conceptual thinking. Homework problems allow engineering students to practice problem solving, and writing new problems can be a creative process for students. Our previous research found that implementing alternative, studentwritten homework problems, referred to as YouTube problems, led to better learning attitudes. YouTube problems are course related; homeworkquality problems generated by reverse engineering publicly available videos. Comparing learning experiences of students solving YouTube versus Textbook problems is the focus of the current study. Impacts of solving YouTube problems are examined based on perception of difficulty as well as students’ problemsolving skills displayed by students. To enable testing, students were assigned one textbook and three YouTube problems. Perception of problem difficulty across problems was examined using the NASA Task Load Index. Additionally, problem solving aptitudes while solving homework problems was assessed using a previously validated rubric called PROCESS: Problem definition, Representing the problem, Organizing the information, Calculations, Solution completion, and Solution accuracy. A new case study compares Textbook and YouTube problems related to reacting systems with recycle, which is one of the most difficult course concepts. A correlation between problem rigor and problem solving wasmore »
Assessing the Reliability of a Chemical Engineering Problemsolving Rubric when Using Multiple Raters
This evidencebased practices paper discusses the method employed in validating the use of a project modified version of the PROCESS tool (Grigg, Van Dyken, Benson, & Morkos, 2013) for measuring student problem solving skills. The PROCESS tool allows raters to score students’ ability in the domains of Problem definition, Representing the problem, Organizing information, Calculations, Evaluating the solution, Solution communication, and Selfassessment. Specifically, this research compares student performance on solving traditional textbook problems with novel, studentgenerated learning activities (i.e. reverse engineering videos in order to then create their own homework problem and solution). The use of studentgenerated learning activities to assess student problem solving skills has theoretical underpinning in Felder’s (1987) work of “creating creative engineers,” as well as the need to develop students’ abilities to transfer learning and solve problems in a variety of real world settings. In this study, four raters used the PROCESS tool to score the performance of 70 students randomly selected from two undergraduate chemical engineering cohorts at two Midwest universities. Students from both cohorts solved 12 traditional textbook style problems and students from the second cohort solved an additional nine studentgenerated video problems.
Any large scale assessment where multiple raters use a rating tool requires more »
 Award ID(s):
 1712186
 Publication Date:
 NSFPAR ID:
 10105181
 Journal Name:
 ASEE Annual Meeting
 Sponsoring Org:
 National Science Foundation
More Like this


Problem solving is a signature skill of engineers. Here, problem solving is employed when students apply course concepts to reverse engineer YouTube videos and solve new studentwritten, homeworkstyle problems (YouTube problems). Replacing textbook problems with YouTube problems, this research focuses on examining the rigor of YouTube problems as well as students’ problemsolving skills on textbook and YouTube problems. A quasiexperimental, treatment/control group design was employed, and data was collected and evaluated using multiple instruments. First, rigor of homework problems was examined using the NASA Task Load Index. Also, problem solving was assessed using a previouslydeveloped rubric called PROCESS Problem definition, Representing the problem, Organizing the information, Calculations, Evaluating the solution, Solution communication, and Selfassessment. PROCESS was modified to independently measure completeness and accuracy of student responses, as well as identify errors committed in material and energy balances. In the treatment group, students were assigned ten textbook problems and nine YouTube problems. In addition to obtaining an evidencebased assessment of problem solving via PROCESS, students’ learning attitudes, overall and with respect to problem solving, were measured via a selfreported survey known as Colorado Learning Attitudes about Science Survey (CLASS). Utilizing YouTube problems in classroom did not influence learning attitudes of studentsmore »

Problem solving is a signature skill of engineers. Here, problem solving is employed when students apply course concepts to reverse engineer YouTube videos and solve new studentwritten, homeworkstyle problems (YouTube problems). Replacing textbook problems with YouTube problems, this research focuses on examining the rigor of YouTube problems as well as students’ problemsolving skills on textbook and YouTube problems. A quasiexperimental, treatment/control group design was employed, and data was collected and evaluated using multiple measurement instruments. First, rigor of homework problems was examined using the NASA Task Load Index. Also, problem solving was assessed using a previouslydeveloped rubric called PROCESS: Problem definition, Representing the problem, Organizing the information, Calculations, Evaluating the solution, Solution communication, and Selfassessment. PROCESS was modified to independently measure completeness and accuracy of student responses, as well as identify errors committed in material and energy balances. In the treatment group, students were assigned ten textbook problems and nine YouTube problems. While the control group obtained higher PROCESS scores at the beginning of the study, both groups exhibited similar problemsolving skills near the end. Also, the rigor of studentwritten YouTube problems was similar to textbook problems related to the same course concepts.

Faculty often utilize homework problems as a means to help students practice problem solving. Recently, with textbook solutions manuals being freely available online, students are prone to copying/cheating, which can severely limit improvements in problem solving. One hypothesis is that YouTube problems could serve as alternatives to textbook problems to significantly reduce cheating and promote better problem solving. YouTube problems are studentwritten problems that were inspired by events in a video publicly available online. While our previous studies have showcased positive attitudes related to engineering, high engagement, and rigor of the YouTube problems, the current study examines a subset of problems related to one major course topic, namely vaporliquid equilibrium. The cohorts include engineering students from a public university who were assigned homework problems as part of a material and energy balance course. Two constructs were explored: problem solving and perception of problem difficulty. The study adopted an established and validated rubric to quantify performance in relevant stages of problem solving, including problem identification, representation, organization, calculation, solution completion, and solution accuracy. While problem solving can be influenced by perception of problem difficulty, the widely used NASA Task Load Index was adopted to measure the problem rigor. This paper willmore »

This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers who included students, professors, and professionals, solving illstructured problems. The participants solved two illstructured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a teambased approach is advantageous especially when dealing with large amounts of data. A teambased approach involves an iterative process of interrater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis. Initial codes were created from existing literature, which were refined through an iterative process. Thismore »