skip to main content

Explore Research Products in the NSF-PAR

Title: Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters
This evidence-based practices paper discusses the method employed in validating the use of a project modified version of the PROCESS tool (Grigg, Van Dyken, Benson, & Morkos, 2013) for measuring student problem solving skills. The PROCESS tool allows raters to score students’ ability in the domains of Problem definition, Representing the problem, Organizing information, Calculations, Evaluating the solution, Solution communication, and Self-assessment. Specifically, this research compares student performance on solving traditional textbook problems with novel, student-generated learning activities (i.e. reverse engineering videos in order to then create their own homework problem and solution). The use of student-generated learning activities to assess student problem solving skills has theoretical underpinning in Felder’s (1987) work of “creating creative engineers,” as well as the need to develop students’ abilities to transfer learning and solve problems in a variety of real world settings. In this study, four raters used the PROCESS tool to score the performance of 70 students randomly selected from two undergraduate chemical engineering cohorts at two Midwest universities. Students from both cohorts solved 12 traditional textbook style problems and students from the second cohort solved an additional nine student-generated video problems. Any large scale assessment where multiple raters use a rating tool requires the investigation of several aspects of validity. The many-facets Rasch measurement model (MFRM; Linacre, 1989) has the psychometric properties to determine if there are any characteristics other than “student problem solving skills” that influence the scores assigned, such as rater bias, problem difficulty, or student demographics. Before implementing the full rating plan, MFRM was used to examine how raters interacted with the six items on the modified PROCESS tool to score a random selection of 20 students’ performance in solving one problem. An external evaluator led “inter-rater reliability” meetings where raters deliberated rationale for their ratings and differences were resolved by recourse to Pretz, et al.’s (2003) problem-solving cycle that informed the development of the PROCESS tool. To test the new understandings of the PROCESS tool, raters were assigned to score one new problem from a different randomly selected group of six students. Those results were then analyzed in the same manner as before. This iterative process resulted in substantial increases in reliability, which can be attributed to increased confidence that raters were operating with common definitions of the items on the PROCESS tool and rating with consistent and comparable severity. This presentation will include examples of the student-generated problems and a discussion of common discrepancies and solutions to the raters’ initial use of the PROCESS tool. Findings as well as the adapted PROCESS tool used in this study can be useful to engineering educators and engineering education researchers.  more » « less
Award ID(s):
1712186
NSF-PAR ID:
10105181
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
ASEE Annual Meeting
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ( , ASEE Annual Conference proceedings)
    null (Ed.)
    Complex problem-solving is a vital skill prevalent to thrive in the workforce along with creativity and conceptual thinking. Homework problems allow engineering students to practice problem solving, and writing new problems can be a creative process for students. Our previous research found that implementing alternative, student-written homework problems, referred to as YouTube problems, led to better learning attitudes. YouTube problems are course related; homework-quality problems generated by reverse engineering publicly available videos. Comparing learning experiences of students solving YouTube versus Textbook problems is the focus of the current study. Impacts of solving YouTube problems are examined based on perception of difficulty as well as students’ problem-solving skills displayed by students. To enable testing, students were assigned one textbook and three YouTube problems. Perception of problem difficulty across problems was examined using the NASA Task Load Index. Additionally, problem solving aptitudes while solving homework problems was assessed using a previously validated rubric called PROCESS: Problem definition, Representing the problem, Organizing the information, Calculations, Solution completion, and Solution accuracy. A new case study compares Textbook and YouTube problems related to reacting systems with recycle, which is one of the most difficult course concepts. A correlation between problem rigor and problem solving was found. 
    more » « less
  2. ( , Master of Science Thesis)
    Problem solving is a signature skill of engineers. Here, problem solving is employed when students apply course concepts to reverse engineer YouTube videos and solve new student-written, homework-style problems (YouTube problems). Replacing textbook problems with YouTube problems, this research focuses on examining the rigor of YouTube problems as well as students’ problem-solving skills on textbook and YouTube problems. A quasi-experimental, treatment/control group design was employed, and data was collected and evaluated using multiple instruments. First, rigor of homework problems was examined using the NASA Task Load Index. Also, problem solving was assessed using a previously-developed rubric called PROCESS Problem definition, Representing the problem, Organizing the information, Calculations, Evaluating the solution, Solution communication, and Self-assessment. PROCESS was modified to independently measure completeness and accuracy of student responses, as well as identify errors committed in material and energy balances. In the treatment group, students were assigned ten textbook problems and nine YouTube problems. In addition to obtaining an evidence-based assessment of problem solving via PROCESS, students’ learning attitudes, overall and with respect to problem solving, were measured via a self-reported survey known as Colorado Learning Attitudes about Science Survey (CLASS). Utilizing YouTube problems in classroom did not influence learning attitudes of students negatively. Students reported that YouTube problems possessed similar rigor as Textbook problems. Instead, students solving YouTube problems measured small effect size improvement in problem- solving skills. 
    more » « less
  3. ; ; ; ( , ASEE Annual Meeting)
    Problem solving is a signature skill of engineers. Here, problem solving is employed when students apply course concepts to reverse engineer YouTube videos and solve new student-written, homework-style problems (YouTube problems). Replacing textbook problems with YouTube problems, this research focuses on examining the rigor of YouTube problems as well as students’ problem-solving skills on textbook and YouTube problems. A quasi-experimental, treatment/control group design was employed, and data was collected and evaluated using multiple measurement instruments. First, rigor of homework problems was examined using the NASA Task Load Index. Also, problem solving was assessed using a previously-developed rubric called PROCESS: Problem definition, Representing the problem, Organizing the information, Calculations, Evaluating the solution, Solution communication, and Self-assessment. PROCESS was modified to independently measure completeness and accuracy of student responses, as well as identify errors committed in material and energy balances. In the treatment group, students were assigned ten textbook problems and nine YouTube problems. While the control group obtained higher PROCESS scores at the beginning of the study, both groups exhibited similar problem-solving skills near the end. Also, the rigor of student-written YouTube problems was similar to textbook problems related to the same course concepts. 
    more » « less
  4. ; ; ( , 2021 ASEE North Central Section Conference)
    null (Ed.)
    Faculty often utilize homework problems as a means to help students practice problem solving. Recently, with textbook solutions manuals being freely available online, students are prone to copying/cheating, which can severely limit improvements in problem solving. One hypothesis is that YouTube problems could serve as alternatives to textbook problems to significantly reduce cheating and promote better problem solving. YouTube problems are student-written problems that were inspired by events in a video publicly available online. While our previous studies have showcased positive attitudes related to engineering, high engagement, and rigor of the YouTube problems, the current study examines a subset of problems related to one major course topic, namely vapor-liquid equilibrium. The cohorts include engineering students from a public university who were assigned homework problems as part of a material and energy balance course. Two constructs were explored: problem solving and perception of problem difficulty. The study adopted an established and validated rubric to quantify performance in relevant stages of problem solving, including problem identification, representation, organization, calculation, solution completion, and solution accuracy. While problem solving can be influenced by perception of problem difficulty, the widely used NASA Task Load Index was adopted to measure the problem rigor. This paper will compare textbook and YouTube problem with respect to overall problem-solving ability as well as in each stage of problem solving. Furthermore, we will investigate whether disparities exist in students’ perceptions when solving vapor-liquid equilibrium problems. 
    more » « less
  5. ; ; ; ; ; ; ; ; ( , ASEE Annual Conference proceedings)
    This research paper elaborates on the process used by a team of researchers to create a codebook from interviews of Civil Engineers who included students, professors, and professionals, solving ill-structured problems. The participants solved two ill-structured problems while speaking aloud their thought process. In addition to recording the participant verbalization, the solution to their problems were also collected with the use of a smart pen. Creating a codebook from interviews is a key element of qualitative analysis forming the basis for coding. While individuals can create codebooks for analysis, a team-based approach is advantageous especially when dealing with large amounts of data. A team-based approach involves an iterative process of inter-rater reliability essential to the trustworthiness of the data obtained by coding. In addition to coding the transcripts as a team, which consisted of novice, intermediate, and experts in the engineering education field, the audio and written solution to the problems were also coded. The use of multiple data sources to obtain data, and not just the verbatim transcripts, is lesser studied in engineering education literature and provides opportunities for a more detailed qualitative analysis. Initial codes were created from existing literature, which were refined through an iterative process. This process consisted of coding data, team consensus on coded data, codebook refinement, and recoding data with the refined codes. Results show that coding verbatim transcripts might not provide an accurate representation of the problem-solving processes participants used to solve the ill-structured problem. Benefits, challenges and recommendations regarding the use of multiple sources to obtain data are discussed while considering the amount of time required to conduct such analysis. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email