<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Assessing the Reliability of a Chemical Engineering Problem-solving Rubric when Using Multiple Raters</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>2019</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10105181</idno>
					<idno type="doi">10.18260/1-2--32124</idno>
					<title level='j'>ASEE Annual Meeting</title>
<idno></idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Timothy Duckett</author><author>Matthew Liberatore</author><author>Uchenna Asogwa</author><author>Gale Mentzer</author><author>Amanda Malefyt</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[This evidence-based practices paper discusses the method employed in validating the use of a project modified version of the PROCESS tool (Grigg, Van Dyken, Benson, & Morkos, 2013) for measuring student problem solving skills. The PROCESS tool allows raters to score students’ ability in the domains of Problem definition, Representing the problem, Organizing information, Calculations, Evaluating the solution, Solution communication, and Self-assessment. Specifically, this research compares student performance on solving traditional textbook problems with novel, student-generated learning activities (i.e. reverse engineering videos in order to then create their own homework problem and solution). The use of student-generated learning activities to assess student problem solving skills has theoretical underpinning in Felder’s (1987) work of “creating creative engineers,” as well as the need to develop students’ abilities to transfer learning and solve problems in a variety of real world settings. In this study, four raters used the PROCESS tool to score the performance of 70 students randomly selected from two undergraduate chemical engineering cohorts at two Midwest universities. Students from both cohorts solved 12 traditional textbook style problems and students from the second cohort solved an additional nine student-generated video problems.Any large scale assessment where multiple raters use a rating tool requires the investigation of several aspects of validity. The many-facets Rasch measurement model (MFRM; Linacre, 1989) has the psychometric properties to determine if there are any characteristics other than “student problem solving skills” that influence the scores assigned, such as rater bias, problem difficulty, or student demographics. Before implementing the full rating plan, MFRM was used to examine how raters interacted with the six items on the modified PROCESS tool to score a random selection of 20 students’ performance in solving one problem. An external evaluator led “inter-rater reliability” meetings where raters deliberated rationale for their ratings and differences were resolved by recourse to Pretz, et al.’s (2003) problem-solving cycle that informed the development of the PROCESS tool. To test the new understandings of the PROCESS tool, raters were assigned to score one new problem from a different randomly selected group of six students. Those results were then analyzed in the same manner as before. This iterative process resulted in substantial increases in reliability, which can be attributed to increased confidence that raters were operating with common definitions of the items on the PROCESS tool and rating with consistent and comparable severity. This presentation will include examples of the student-generated problems and a discussion of common discrepancies and solutions to the raters’ initial use of the PROCESS tool. Findings as well as the adapted PROCESS tool used in this study can be useful to engineering educators and engineering education researchers.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Introduction</head><p>Engineers require precision and reliability in the tools they use to conduct research. For instance, the optimal design of planning vessels that transport goods around the world relies on the consistency of repeated particle image velocimetry measurements of flow characteristics around a ship <ref type="bibr">[1,</ref><ref type="bibr">2]</ref>. Yet much work is still required to develop tools for use in engineering education that meet the same rigorous standards of accuracy and repeatability when it comes to the assessment of student outcomes <ref type="bibr">[3]</ref><ref type="bibr">[4]</ref><ref type="bibr">[5]</ref>.</p><p>The attempts in engineering education to meet the demands of accountability and to provide assurances in the assessment of student knowledge have been marked by several components. There are institutes and committees comprised of engineering professors from across the country who develop and validate cognitive/declarative knowledge exams that serve as summative course assessments <ref type="bibr">[6]</ref>. In response to the call for more robust learning outcomes, many science and engineering departments have integrated professional development programs that bolster faculty familiarity with course evaluation concepts <ref type="bibr">[7,</ref><ref type="bibr">8]</ref>. Incorporating multiple types of student assessment in classroom instructional design has been found to increase proficient practice in the field <ref type="bibr">[9]</ref>. Methods of student assessment often incorporate rater-mediated assessment <ref type="bibr">[10]</ref><ref type="bibr">[11]</ref><ref type="bibr">[12]</ref><ref type="bibr">[13]</ref>. These methods of assessing student knowledge move beyond traditional notions of student grades that are just the calculation of correct responses divided by total possible items on a formative test. In rater-mediated assessment, student performance on a given task (e.g. presentation, homework solution, concept paper) is judged by a rater along any number of domains through the use of a rating scale <ref type="bibr">[14]</ref>. The inclusion of these additional components provides the prospect of more nuanced and detailed student assessments, but also the threat of greater inconsistency. Efforts need to be made to ensure that the rubric used for rating students represents the intended construct. This task necessitates the development of a continuum where students can be placed according to their possession of less or more of the latent construct attempted to be measured indirectly through their performance on the given task. Rating scales need to distinguish between distinct levels of performance. Raters need to be consistent in their use of the rating scales.</p><p>Figure <ref type="figure">1</ref> provides a model of rater-mediated assessment of problem-solving. This model shows that students are placed along the continuum of problem-solving ability by raters using rating scale judgments of student performance on a set of tasks.</p><p>Figure <ref type="figure">1</ref>. A model for rater-mediated assessment. Adapted from <ref type="bibr">[14]</ref>.</p><p>The qualitative levels defined by each category on a rating scale represent unequal intervals along the latent construct <ref type="bibr">[15]</ref>. The conveniently adopted ordinal level ratings given to the qualitative categories (e.g. a "score" of 1 for "Inadequate," "2" for "Acceptable," etc.) need to be converted into linear measures before they can be used in any meaningful way as proxy measures of student ability levels. For example, one would be hard pressed to argue that "Disagree" minus "Strongly Disagree" equals an integer value of 1. Therefore, each facet of the assessment situation as shown in Figure <ref type="figure">1</ref>, above, becomes a parameter that is estimated in a many-faceted measurement approach. Figure <ref type="figure">2</ref> describes the many-facet Rasch model (MFRM) developed by Linacre based upon the Rasch measurement paradigm <ref type="bibr">[15]</ref><ref type="bibr">[16]</ref><ref type="bibr">[17]</ref>. This approach treats the assigned ordinal ratings in an assessment as the outcomes of the linear combination of the parameters. A comparison of the empirical variance encountered during parameter estimation with the level of measurement error expected by the model indicates how well the data fit the model. Unlike other item response and classical test theory traditions that try to fit measurement models to the data, the Rasch model is built on the fundamental measurement property of invariance: the measurement of persons must be independent of particular items used for measuring (item-invariant person measurement) and the calibration of items must be independent of particular persons used for calibration (person-invariant item calibration) <ref type="bibr">[18,</ref><ref type="bibr">19]</ref>. The task of measurement in the Rasch paradigm becomes an investigation of how well a particular data set adhere to the principles of invariant measurement embedded in an ideal-type model <ref type="bibr">[14,</ref><ref type="bibr">20]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>log ( &#119875;&#119899;&#119894;&#119895;&#119895;&#119896; &#119875;&#119899;&#119894;&#119895;&#119896;-1 ) = &#119861;&#119899; -&#119863;&#119894; -&#119862;&#119895; -&#119865;&#119896;</head><p>where: Pnijk is the probability of examinee n, when rated on item i by judge j, being awarded a rating of k. Pnijk-1 is the probability of examinee n, when rated on item i by judge j, being awarded a rating of k-1. Bn is the ability of examinee n. Di is the difficulty of item i. Cj is the severity of judge j. Fk is the extra difficulty overcome in being observed at the level of category k, relative to category k-1. Figure <ref type="figure">2</ref>. Equation for the many-facet Rasch model. The purpose of this paper is to estimate the reliability of rater-mediated assessments of undergraduate engineering student problem-solving. Latent constructs such as problem-solving ability and content mastery comprise the domain of learning outcomes and variables of interest in the field of education. The MFRM was developed using philosophical principles similar to those that underpin the physical measures in engineering. Use of the MFRM can determine the fairness and objectivity of the estimations of student problem-solving by accounting for all of the aspects of the measurement process that can introduce error into that estimation, such as poorly functioning items, ill-defined rating categories, and differing levels of rater severity. Establishing reliability in rater-mediated assessments provides evidence that the scores obtained on the test actually represent the latent construct instead of being an artifact of rater discrepancies <ref type="bibr">[21]</ref>. This paper argues that the MFRM provides necessary evidence toward the validity of inferences that can be made regarding student learning outcomes in engineering education.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Methods</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Participants</head><p>A total of 113 students were enrolled in an undergraduate Material and Energy Balance chemical engineering course as part of a control cohort (23 students; 22% female) and a treatment cohort (93 students; 41% female) at two Midwest Universities. Table <ref type="table">1</ref> shows different distributions for highest mathematics courses completed by cohort. This discrepancy can be explained as a consequence of the course sequence occurring in the sophomore year for the control cohort (fall and spring semesters) compared to the spring semester of the freshmen year for the treatment cohort. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Instrument</head><p>The PROCESS was used to score students' homework problem solutions <ref type="bibr">[10]</ref>. The PROCESS was developed using several theoretical frameworks that consider the conceptual, analytical, and phenomenological process demands and cognitive skills involved in problem solving <ref type="bibr">[22]</ref>. PROCESS was modified to assess the problem-solving process for solved handwritten homework problems, which differs from its original use where participants' solutions were collected on tablets and custom software to see erasing and other details <ref type="bibr">[23,</ref><ref type="bibr">24]</ref>. The tool was modified to suit material and energy balance problems. The modified PROCESS consists of a 6stage rubric assessing: Problem definition, Representing the problem, Organizing information, Calculations, Solution completion and Solution accuracy. Each item in the revised PROCESS consists of four scaling levels ranging from 0 to 3 with the following categories to rate student performance on each of the six the stages of problem solving: missing, inadequate, acceptable, and accurate. Any identification regarding group identity was removed prior to scoring and replaced with a project-assigned ID number to maintain privacy and to mask group membership from raters.</p><p>A complete rating plan was proposed where four raters would use the PROCESS tool to score all solutions submitted by all students from both cohorts. The four raters consisted of one chemical engineering faculty member, one high school science teacher, and one graduate and one undergraduate student in chemical engineering. All students completed ten traditional textbook problems during the respective courses.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Analyses</head><p>Initial inter-rater reliability was assessed in line with best practices as a means to evaluate how consistently raters measured student problem-solving ability <ref type="bibr">[25]</ref>. The first assessment involved the PROCESS scores that five raters assigned to 20 randomly selected students for one textbook problem. An additional chemical engineer faculty member joined the four raters above to provide a benchmark reference point. Figure <ref type="figure">3</ref> presents a portion of the problem that was purposefully selected for piloting the use of the rubric. This specific problem was chosen in part because the research team decided it was of average difficulty and representative of the ten textbook problems assigned. The FACETS <ref type="bibr">[16]</ref> computer program was used to produce parameter estimates for the facets involved in the rater-mediated assessment (rating scale function, rater severity, item difficulty, etc.). Subsequently, qualitative focus group meetings were conducted where raters deliberated rationale for their ratings and their understanding of the underlying continuum of problemsolving. Discrepancies in ratings were resolved by recourse to the theoretical framework of the problem-solving cycle that informed the development of the PROCESS <ref type="bibr">[22,</ref><ref type="bibr">26]</ref>. Raters were then rescore that problem in light of their refined understanding of the latent trait and function of PROCESS. Those results were then analyzed in the same manner as before using the FACETS program to estimate parameters. The resulting logit scores were rescaled to conform to the original scale of 0 points (a rating of "missing" for all six PROCESS items, representing the lowest problem-solving ability level) through 18 points (a rating of "accurate" for all six PROCESS items, representing the highest problem-solving ability level).</p><p>These interval level measures were then used to calculate Cohen's kappa and intraclass correlation coefficients as extra measures of inter-rater reliability in addition to the standard errors and fit statistics provided by FACETS. Several types of descriptive statistics were calculated to assess the inter-rater reliability of the four raters using the PROCESS. The goal was to ensure that all of the raters used the rating scale consistently so that differences in student performance can be attributed to different problem-solving abilities and not a result of receiving a rating from a more or less severe rater.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Results</head><p>The problem-solving continuum developed by FACETS as a result of the parameter estimations of rater severity, student ability level, item difficulty, and rating scale function are displayed in Figure <ref type="figure">4</ref>. Students with higher scores indicate more advanced problem-solving skills, such that student 4835 was identified as exhibiting the most advanced problem-solving skills and student 1874 as the least advanced. For the PROCESS Item column, the higher the item is on the "ruler," the more difficult it is for students to answer it correctly. Therefore, "Final Solution Accuracy" was the most difficult item, with "Representing the Problem" and "Final Solution Completion" being the easiest items (i.e. students scored the highest on these items). The Rater column places the more severe raters (i.e. gave the lowest scores-Raters 3 and 5) higher on the ruler and the more lenient raters (i.e. awarded the highest scores-Rater 4) lower on the ruler. Figure <ref type="figure">5</ref> provides more descriptive statistics regarding the estimation of the rater severity parameter. The raters are ordered in rows from most to least severe, with their overall measure being reported in the fifth column. Of particular interest is the Strata statistic of 2.48 (highlighted in yellow). This indicates that there were two distinct groups of raters, a more severe group and a more lenient group.</p><p>Several types of rater agreement could be achieved <ref type="bibr">[16]</ref>. If the desire is to have raters agree exactly with each other then we would expect the third to last column in Figure <ref type="figure">5</ref> (Exact Agree. Obs. %) to be greater than 90%. This would mean that raters were agreeing on exact scores for student performance on individual PROCESS items greater than 90% of the time. As can be seen, in this instance this is not the case. Of greater concern to most measurement contexts is the determination of similar leniency/severity in rater assessments. This is reported by the 'Reliability (not inter-rater)' statistic, highlighted in red in Figure <ref type="figure">5</ref>, and calculated by taking [1 -Separation Reliability], [1 -0.72 = 0.28], with numbers closer to 0 being best. While there are no hard and fast guidelines, 0.28, in conjunction with other evidence, suggests that the raters were rating with different levels of severity. Similarly, the null hypothesis for the "Fixed (all same) chi-square tests" shown in the third row from the bottom of Figure <ref type="figure">5</ref>, assumes that all raters are rating the same. The significant statistic chi-square value (highlighted in green, Figure <ref type="figure">5</ref>) indicates that we must reject this null hypothesis, providing further evidence that raters are rating with different levels of severity and therefore the raters are bringing different interpretations of the rating scale into their scoring of student problems   Further diagnosis revealed some of the overarching areas of disagreement. For example, Table <ref type="table">2</ref> reveals statistically significant bias regarding how Rater 5 scored the first PROCESS item, "Identify the Problem," and Rater 3's rating of the second item, "Represent the problem." The scores the raters gave on those respective items across all 20 students they rated were statistically significantly lower than expected by the model given the estimated student problem-solving ability level and item difficulty. The t-statistics were computed to test the hypothesis that there was no bias present in the ratings besides what was to be expected by measurement error. This analysis indicated bias in how these raters scored these particular items; specifically, they were harsher when scoring these items compared to other PROCESS items.</p><p>- </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 2 Bias in rater interaction with PROCESS items</head><p>Comparisons between how pairs of raters scored specific items can lead to fruitful conversations regarding the characteristics of responses that belong to each rating category on the items, i.e. "What does an answer to Item 1, Identify the problem, need to look like to be considered in the acceptable category (score of 2)?" The data in Table <ref type="table">3</ref> suggest that the raters employed different understandings of item 1, "Identify the problem." This was the greatest source of disagreement leading to different levels of rater severity. For example, row 1 reports that Rater 1 (overall severity estimate of -0.48 logits on item 1) has a statistically significantly more lenient understanding/rating of item 1, Identify the problem when compared to Rater 5's (0.84 logit) more strict position on that item, t (35) = -2.55, p =.015. This indicates that a student solution that was assessed by Rater 5 was likely to receive a significantly lower score on the "Identify the problem" item than if that same solution was assessed by Rater 1.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 3 Pairwise comparison of rater discrepancies in scores assigned to PROCESS items</head><p>The qualitative meeting between raters examined the discrepancies highlighted above, in addition to those found in the raw scores raters provided to some of the students, displayed in Table <ref type="table">4</ref>. Highlighted cells show areas of considerable discrepancy in ratings that could potentially represent different understandings of the underlying construct and its measurement. It can be seen that the majority of the ratings provided by the raters were similar across the PROCESS items for most of the students. The solution provided by student 4036, a moderate performing student, provided difficulties that led to fruitful conversations about the different characteristics of responses in the "inadequate" and "acceptable" rating scale categories. The meeting offered the opportunity for the raters to clarify any fundamental disagreements or misunderstandings pertaining to the latent construct of problem-solving ability. Following the qualitative meeting, the four primary raters were asked to rescore the student solutions for the "methanol reactor" problem described above. Inter-rater reliability statistics were computed to assess the extent to which the raters understood and scored "problem-solving ability" in a consistent manner with each other. The results in Table <ref type="table">5</ref> report the two forms of Cohen's kappa that were calculated. The second column reports the standard Cohen's kappa for absolute agreement. This statistic quantifies how many instances of exact agreement occurred for the ratings (e.g. both raters would have to give a particular student the same rating on a specific item). This method is best suited to determine absolute level of agreement, essentially treating the ratings as binary outcomes, and therefore only has limited applicability here. It is included in the present study just to provide a baseline for comparison. Column 3 reports the quadratic weighted kappa statistics. These take into account the nature of ordered categories and adjust for the fact that adjacent categories are more alike than non-adjacent (i.e. ratings of 0 from one judge and 1 from another are more similar than ratings of a 0 and a 3). Moderate levels of agreement (kappa statistic in the range of .60 -.79) mark half of the rater relationships; specifically all of those relationships that do not involve Rater 3. Table <ref type="table">5</ref> highlights the need to follow up with Rater 3 to discuss their understanding of the latent variable and potentially provide additional training on their understanding of the rating scale categories that comprise the assessment tool. Additionally, intraclass correlation coefficients (ICC) were computed to assess how similar the group of raters as a whole (rather than rater pairs) scored problem-solving ability for the student solutions. This correlation shows the reproducibility of the measurement of student problemsolving ability. Table <ref type="table">6</ref> reports the results of the intraclass correlation coefficient computed using the interval level estimates produced by FACETS as the measure. The ICC coefficient was calculated using the initial assessment of the raters and then again using the ratings of the second round of assessment. Specifically, a two-way mixed effects, multiple-raters model was employed. The initial ICC coefficient of .844, indicates good reliability. Yet, when taking into consideration the 95% CI [.570, .973], there was a wide range from very low moderate to excellent reliability in the raters' scores. The ratings from the post-qualitative meeting show appreciable increases in reliability and a narrower confidence interval. Provided the raters score student solutions with a similar conceptualization of problem-solving ability, then adding more student solutions will help reduce the high standard error caused by the relative small sample size in this analysis.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Table 6</head><p>Intraclass correlation coefficient for rater agreement Figure <ref type="figure">7</ref> maps the rater-mediated student scores used to calculate the post-qualitative meeting ICC coefficient. Data came from six randomly selected students. This plot provides a snapshot of how raters differed in the sum scores given to each of the students. It should be noted that Rater 3 shows significant separation from the other three raters on three of the six assessments (A, C, and F), while Rater 2 was drastically lower for the ratings on student B. This provides further explanation for the variation in the 95% CI of the ICC coefficient.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Discussion</head><p>This study estimated the reliability of scores from a rubric designed to measure chemical engineering problem-solving ability. The analyses mark an important step in the validation of the PROCESS itself which has only been validated previously using traditional correlational techniques. The many-facet Rasch model (MFRM) was used to explore a set of rater-mediated data. This evaluative approach and choice of measurement models was designed to meet the increasing demands of accountability in engineering, and in this case specifically chemical engineering, education <ref type="bibr">[3,</ref><ref type="bibr">6,</ref><ref type="bibr">27]</ref>. The models in the Rasch measurement paradigm are particularly well suited to the task of evaluating the defensibility of measures pertaining to the assessment of student learning outcomes. The demands of specific objectivity, as set out by Rasch, require person-free item calibration and item-free person measures <ref type="bibr">[18]</ref>. The Rasch models also expect the measures resulting from data to meet other requirements similar to those maintained for the physical measures that define the field of chemical engineering. Examples include the requirements of monotonicity and local independence. These conditions demand that items with increasing levels of difficulty require increasing presence of the latent variable in order for an individual to succeed on that item/receive a higher rating <ref type="bibr">[28]</ref>.</p><p>The initial findings show promise for the validity of the measures of problem-solving ability produced by the PROCESS. Qualitative meetings discussed the raters' conceptions of the latent construct and how the rating scales mapped progress along the continuum of lower to greater levels of the construct. The conversations produced a more stable understanding of the thresholds of each of the rating categories, e.g. the hallmarks of an "inadequate" (rating of 1) response to a PROCESS item and at what point that response became "acceptable" (rating of 2). Future analyses will monitor the function of the rating scale categories as different chemical engineering problems are scored. The inclusion of more rater-mediated assessments will make for more precise parameter estimations and therefore lead to student assessments that more accurately represent actual student ability. Fortunately, the MFRM evaluation process is iterative in nature and can (and should) be conducted as assessments are ongoing <ref type="bibr">[25]</ref>. This evaluation process can identify sources of measurement error in any of the facets estimated, including the parameter of rater severity. Discrepancies in use of the rating scale on the PROCESS tool can provide opportunities for additional training. These evaluative steps can increase not only the accuracy of the scores among the raters, but also the precision of those scores in measuring the latent construct, provided the raters maintain fidelity in their use of the rating scale rubric and the operationalization of the problem-solving framework.</p><p>Reports suggest that there will only be an increase in the call for authentic, meaningful measures of student outcomes in engineering programs as the 21 st Century proceeds <ref type="bibr">[29]</ref>. Novel methods of engaging students in the content and methods of engineering appear promising <ref type="bibr">[11,</ref><ref type="bibr">30,</ref><ref type="bibr">31]</ref>. The validity of the pedagogical interventions and the inferences that can be drawn from resulting measures will be enhanced through the use of robust measurement and evaluation techniques.</p><p>Engineering educators who demand measures as sturdy as the measures used to build the machines that cultivate alternative energy <ref type="bibr">[32]</ref> and fuel the next modes of transportation <ref type="bibr">[33]</ref> need to implement a rigorous system of evaluation of their pedagogical assessments through the use of a measurement model that makes such demands on the data. To that end, the implementation of Rasch measurement models will provide robust validation for the measures of student learning outcomes, which in turn can improve course curricula by accurately targeting domains and transferable skillsets critical to the development of this generation's chemical engineers.</p></div></body>
		</text>
</TEI>
