An official website of the United States government
Constructed responses can be used to assess the complexity of student thinking and can be evaluated using rubrics. The two most typical rubric types used are holistic and analytic. Holistic rubrics may be difficult to use with expert-level reasoning that has additive or overlapping language. In an attempt to unpack complexity in holistic rubrics at a large scale, we have developed a systematic approach called deconstruction. We define deconstruction as the process of converting a holistic rubric into defining individual conceptual components that can be used for analytic rubric development and application. These individual components can then be recombined into the holistic score which keeps true to the holistic rubric purpose, while maximizing the benefits and minimizing the shortcomings of each rubric type. This paper outlines the deconstruction process and presents a case study that shows defined concept definitions for a hierarchical holistic rubric developed for an undergraduate physiology-content reasoning context. These methods can be used as one way for assessment developers to unpack complex student reasoning, which may ultimately improve reliability and validation of assessments that are targeted at uncovering large-scale complex scientific reasoning.
more »
« less
A Search to Capture and Report on Feasibility of Implementation
This brief report describes the conception, development, and use of a rubric in evaluating the feasibility of a new program. The evaluators searched for a meta-analytic tool to help organize ideas about what data to collect, and why, in order to create a detailed story of feasibility of implementation for the client. The main advantage of using the rubric-based tool is that it lays out key evaluative criteria that are defined as concretely as possible. The article gives a brief overview of the literature on the use of rubrics in evaluation, illustrates the use of a feasibility of implementation rubric as a tool for development, analysis, and reporting, and concludes with recommendations emergent from the use of the rubric.
more »
« less
- Award ID(s):
- 1625215
- PAR ID:
- 10229736
- Date Published:
- Journal Name:
- American Journal of Evaluation
- Volume:
- 41
- Issue:
- 1
- ISSN:
- 1098-2140
- Page Range / eLocation ID:
- 145 to 155
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Constructed responses can be used to assess the complexity of student thinking and can be evaluated using rubrics. The two most typical rubric types used are holistic and analytic. Holistic rubrics may be difficult to use with expert-level reasoning that has additive or overlapping language. In an attempt to unpack complexity in holistic rubrics at a large scale, we have developed a systematic approach called deconstruction. We define deconstruction as the process of converting a holistic rubric into defining individual conceptual components that can be used for analytic rubric development and application. These individual components can then be recombined into the holistic score which keeps true to the holistic rubric purpose, while maximizing the benefits and minimizing the shortcomings of each rubric type. This paper outlines the deconstruction process and presents a case study that shows defined concept definitions for a hierarchical holistic rubric developed for an undergraduate physiology-content reasoning context. These methods can be used as one way for assessment developers to unpack complex student reasoning, which may ultimately improve reliability and validation of assessments that are targeted at uncovering large-scale complex scientific reasoning.more » « less
-
Despite the recent proliferation of research concerning integrating computational thinking (CT) into K-5th grade curriculum, there is little literature concerning how to evaluate the quality of CT integrated curricula, especially curricula integrating CT into language arts and social studies content areas. In this paper, we present a theoretically derived rubric for the evaluation of CT integrated curricula for grades K-5 across the curriculum (math, science, language arts, social studies). Our rubric is divided into two sections. The first section provides guidelines for identifying the integration type (disciplinary, multidisciplinary, interdisciplinary, or transdisciplinary). The second section presents six categories of evaluation that further subsume nine sub-categories. The principal categories of evaluation include the following: conceptual coherence, role of computational technology, assessment, use of multiple representations, play, and equity. We include the play category as an aspect of developmental appropriateness. Play is an important pedagogical approach for learning in the early grades. Our work takes place in the context of the Computer Science (CS) for All initiative in the United States which emphasizes the goal of improving racial and gender diversity in CS participation. Therefore, creating integrated lessons that address equity is important. Our paper describes rubric development from the theoretical perspectives that underlie the inclusion of each type, category, and sub-category. Our evaluative rubric can guide future efforts to integrate CT/CS into the elementary curricula. Researchers can utilize our rubric to evaluate and analyze CT-integrated curricula, and educators can benefit from using this rubric as a guideline for curriculum development.more » « less
-
More than 27 years have passed since the National Science Board identified Broader Impacts as one of two merit criteria for National Science Foundation proposals. Yet many researchers remain less certain of how to develop, implement, and assess a broader impact plan. This multi-method study of a Broader Impacts (BI) rubric analyzed expert panels that included BI professionals and researchers for both content validity and reliability. Focus groups with researchers explicate the challenges researchers face regarding BI plans and the potential value of the rubric as a tool for use. It revealed the challenges researchers have in weighing proven strategies versus innovative strategies, a bias documented by other scholars. Researchers stated concern with how to weigh the different facets of the rubric to arrive at a single score. Moreover, researchers reported that their disciplinary field influenced how they interpreted the audiences whose needs and interests may be met through BI plans. These distinctions represent a range of different types of community engaged scholarship (e.g., public information network, community-campus partnership, K-12 schools’ partnerships). Finally, researchers found the BI rubric useful in evaluating and developing their own BI plans as well as their role in panels to ultimately strengthen the field of funded BI work.more » « less
-
The demand to provide high-quality computer science (CS) education to K-12 students across the United States continues to grow due to societal transformations driven by AI and cybersecurity. However, the impact of state initiatives and mandates on district leaders’ decision making remains an under-explored area in the literature. In 2022, CSforALL began work in Tennessee, a state poised to enact CS education policy, as part of a Research Practice Partnership (RPP). This study investigates the first eight school districts who participated in the Strategic CSforALL Resource and Implementation Planning Tool (SCRIPT) workshops in 2022 and 2023, setting goals based on the SCRIPT rubric. The study takes a general qualitative approach underpinned by the Capacity, Access, Participation, and Experience (CAPE) Framework [14] to develop a coding scheme analyzing the districts’ related rubric scores and goals, and to investigate the impacts on equity indicators. The districts participated in three SCRIPT workshops held in 2022 and 2023, and this study dives deeply into the initial goals as well as analyzing the ways the SCRIPT rubric aligned to the CAPE Framework to investigate how district leaders make decisions which impact teacher and student outcomes which lead to equitable high-quality CS education.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1177/1098214019878784
