skip to main content

Title: Scoring with classroom observational rubrics: a longitudinal examination of raters’ responses and perspectives
This study examines the utilization of cognitive interviews longitudinally over a one-year period to collectively trace raters’ response processes as they interpreted and scored with observational rubrics designed to measure teaching practices that promote equity and access in elementary and middle school mathematics classrooms. We draw on four rounds of cognitive interviews (totaling 14 interviews) that involved four raters at purposeful time points spread over the year. Findings reported in this study focus on raters’ responses about one rubric, positioning students as competent. The findings point to the complexities of utilizing observational rubrics and the need to track response processes longitudinally at multiple time points during data collection in order to attend to rater calibration and the reliability and validity of resulting rubric scores.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Objective. Maternal stress is a psychological response to the demands of motherhood. A high level of maternal stress is a risk factor for maternal mental health problems, including depression and anxiety, as well as adverse infant socioemotional and cognitive outcomes. Yet, levels of maternal stress (i.e., levels of stress related to parenting) among low-risk samples are rarely studied longitudinally, particularly in the first year after birth. Design. We measured maternal stress in an ethnically diverse sample of low-risk, healthy U.S. mothers of healthy infants (N = 143) living in South Florida across six time points between 2 weeks and 14 months postpartum using the Parenting Stress Index-Short Form, capturing stress related to the mother, mother-infant interactions, and the infant. Results. Maternal distress increased as infants aged for mothers with more than one child, but not for first-time mothers whose distress levels remained low and stable across this period. Stress related to mother-infant dysfunctional interactions lessened over the first 8 months. Mothers’ stress about their infants’ difficulties decreased from 2 weeks to 6 months, and subsequently increased from 6 to 14 months. Conclusions. Our findings suggest that maternal stress is dynamic across the first year after birth. The current study adds to our understanding of typical developmental patterns in early motherhood and identifies potential domains and time points as targets for future interventions. 
    more » « less
  2. Research prior to 2005 found that no single framework existed that could capture the engineering design process fully or well and benchmark each element of the process to a commonly accepted set of referenced artifacts. Compounding the construction of a stepwise, artifact driven framework is that engineering design is typically practiced over time as a complex and iterative process. For both novice and advanced students, learning and applying the design process is often cumulative, with many informal and formal programmatic opportunities to practice essential elements. The Engineering Design Process Portfolio Scoring Rubric (EDPPSR) was designed to apply to any portfolio that is intended to document an individual or team driven process leading to an original attempt to design a product, process, or method to provide the best and most optimal solution to a genuine and meaningful problem. In essence, the portfolio should be a detailed account or “biography” of a project and the thought processes that inform that project. Besides narrative and explanatory text, entries may include (but need not be limited to) drawings, schematics, photographs, notebook and journal entries, transcripts or summaries of conversations and interviews, and audio/video recordings. Such entries are likely to be necessary in order to convey accurately and completely the complex thought processes behind the planning, implementation, and self-evaluation of the project. The rubric is comprised of four main components, each in turn comprised of three elements. Each element has its own holistic rubric. The process by which the EDPPSR was created gives evidence of the relevance and representativeness of the rubric and helps to establish validity. The EDPPSR model as originally rendered has a strong theoretical foundation as it has been developed by reference to the literature on the steps of the design process through focus groups and through expert review by teachers, faculty and researchers in performance based, portfolio rubrics and assessments. Using the unified construct validity framework, the EDDPSR’s validity was further established through expert reviewers (experts in engineering design) providing evidence supporting the content relevance and representativeness of the EDPPSR in representing the basic process of engineering design. This manuscript offers empirical evidence that supports the use of the EDPPSR model to evaluate student design-based projects in a reliable and valid manner. Intra-class correlation coefficients (ICC) were calculated to determine the inter-rater reliability (IRR) of the rubric. Given the small sample size we also examined confidence intervals (95%) to provide a range of values in which the estimate of inter-reliability is likely contained. 
    more » « less
  3. In this study, we investigated the impact of the evaluation capacity building (ECB) efforts of an organization by examining the evaluation plans included in funding proposals over a 14-year period. Specifically, we sought to determine the degree to which and how evaluation plans in proposals to one National Science Foundation (NSF) program changed over time and the extent to which the organization dedicated to ECB in that program may have influenced those changes. Independent raters used rubrics to assess the presence of six essential evaluation plan elements. Statistically significant correlations indicate that proposal evaluation plans improved over time, with noticeable differences before and after ECB efforts were integrated into the program. The study adds to the limited literature on using artifacts of evaluation practice rather than self-reports to assess ECB impact.

    more » « less
  4. Key points

    Right heart catheterization data from clinical records of heart transplant patients are used to identify patient‐specific models of the cardiovascular system.

    These patient‐specific cardiovascular models represent a snapshot of cardiovascular function at a given post‐transplant recovery time point.

    This approach is used to describe cardiac function in 10 heart transplant patients, five of which had multiple right heart catheterizations allowing an assessment of cardiac function over time.

    These patient‐specific models are used to predict cardiovascular function in the form of right and left ventricular pressure‐volume loops and ventricular power, an important metric in the clinical assessment of cardiac function.

    Outcomes for the longitudinally tracked patients show that our approach was able to identify the one patient from the group of five that exhibited post‐transplant cardiovascular complications.


    Heart transplant patients are followed with periodic right heart catheterizations (RHCs) to identify post‐transplant complications and guide treatment. Post‐transplant positive outcomes are associated with a steady reduction of right ventricular and pulmonary arterial pressures, toward normal levels of right‐side pressure (about 20 mmHg) measured by RHC. This study shows that more information about patient progression is obtained by combining standard RHC measures with mechanistic computational cardiovascular system models. The purpose of this study is twofold: to understand how cardiovascular system models can be used to represent a patient's cardiovascular state, and to use these models to track post‐transplant recovery and outcome. To obtain reliable parameter estimates comparable within and across datasets, we use sensitivity analysis, parameter subset selection, and optimization to determine patient‐specific mechanistic parameters that can be reliably extracted from the RHC data. Patient‐specific models are identified for 10 patients from their first post‐transplant RHC, and longitudinal analysis is carried out for five patients. Results of the sensitivity analysis and subset selection show that we can reliably estimate seven non‐measurable quantities; namely, ventricular diastolic relaxation, systemic resistance, pulmonary venous elastance, pulmonary resistance, pulmonary arterial elastance, pulmonary valve resistance and systemic arterial elastance. Changes in parameters and predicted cardiovascular function post‐transplant are used to evaluate the cardiovascular state during recovery of five patients. Of these five patients, only one showed inconsistent trends during recovery in ventricular pressure–volume relationships and power output. At the four‐year post‐transplant time point this patient exhibited biventricular failure along with graft dysfunction while the remaining four exhibited no cardiovascular complications.

    more » « less
  5. null (Ed.)
    This study explored seven engineering graduate students’ collaborative problem-solving (CPS) skills while working in interdisciplinary teams. Students worked in two different teams, in face-to-face and online environments, to solve complex manufacturing design challenges posed by their instructor. The students were assessed using an observational rubric with four dimensions: peer interactions, positive communication, tools and methods and iteration and adaption, and scored via each dimension’s associated attributes, and subsequently interviewed. Six students scored emergent or proficient in CPS and had slightly higher CPS scores during the second observation. One student demonstrated a limited ability for CPS and the observable CPS skills decreased during the project. Interviews revealed the importance of (1) relying on instructor and student chosen technologies for collaborative tasks, (2) recognising and drawing on peer expertise early in the project, (3) building trust during and outside of team meetings and (4) valuing off-site and online collaborative work. Findings advance the understanding of how graduate students working in interdisciplinary teams rely on particular features of collaboration to solve engineering design challenges, which may assist in developing future skills and fostering productive teamwork. 
    more » « less