skip to main content


Title: Utilizing Evidence-Centered Design to Develop Assessments: A High School Introductory Computer Science Course
Evidence-centered design (ECD) is an assessment framework tailored to provide structure and rigor to the assessment development process, and also to generate evidence of assessment validity by tightly coupling assessment tasks with focal knowledge, skills, and abilities (FKSAs). This framework is particularly well-suited to FKSAs that are complex and multi-part (Mislevy and Haertel, 2006), as is the case with much of the focal content within the computer science (CS) domain. This paper presents an applied case of ECD used to guide assessment development in the context of a redesigned introductory CS curriculum. In order to measure student learning of CS skills and content taught through the curriculum, knowledge assessments were written and piloted. The use of ECD provided an organizational framework for assessment development efforts, offering assessment developers a clear set of steps with accompanying documentation and decision points, as well as providing robust validity evidence for the assessment. The description of an application of ECD for assessment development within the context of an introductory CS course illustrates its utility and effectiveness, and also provides a guide for researchers carrying out related work.  more » « less
Award ID(s):
1639946
NSF-PAR ID:
10290098
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
Frontiers in Education
Volume:
6
ISSN:
2504-284X
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Evidence has shown that facilitating student-centered learning (SCL) in STEM classrooms enhances student learning and satisfaction [1]–[3]. However, despite increased support from educational and government bodies to incorporate SCL practices [1], minimal changes have been made in undergraduate STEM curriculum [4]. Faculty often teach as they were taught, relying heavily on traditional lecture-based teaching to disseminate knowledge [4]. Though some faculty express the desire to improve their teaching strategies, they feel limited by a lack of time, training, and incentives [4], [5]. To maximize student learning while minimizing instructor effort to change content, courses can be designed to incorporate simpler, less time-consuming SCL strategies that still have a positive impact on student experience. In this paper, we present one example of utilizing a variety of simple SCL strategies throughout the design and implementation of a 4-week long module. This module focused on introductory tissue engineering concepts and was designed to help students learn foundational knowledge within the field as well as develop critical technical skills. Further, the module sought to develop important professional skills such as problem-solving, teamwork, and communication. During module design and implementation, evidence-based SCL teaching strategies were applied to ensure students developed important knowledge and skills within the short timeframe. Lectures featured discussion-based active learning exercises to encourage student engagement and peer collaboration [6]–[8]. The module was designed using a situated perspective, acknowledging that knowing is inseparable from doing [9], and therefore each week, the material taught in the two lecture sessions was directly applied to that week’s lab to reinforce students’ conceptual knowledge through hands-on activities and experimental outcomes. Additionally, the majority of assignments served as formative assessments to motivate student performance while providing instructors with feedback to identify misconceptions and make real-time module improvements [10]–[12]. Students anonymously responded to pre- and post-module surveys, which focused on topics such as student motivation for enrolling in the module, module expectations, and prior experience. Students were also surveyed for student satisfaction, learning gains, and graduate student teaching team (GSTT) performance. Data suggests a high level of student satisfaction, as most students’ expectations were met, and often exceeded. Students reported developing a deeper understanding of the field of tissue engineering and learning many of the targeted basic lab skills. In addition to hands-on skills, students gained confidence to participate in research and an appreciation for interacting with and learning from peers. Finally, responses with respect to GSTT performance indicated a perceived emphasis on a learner-centered and knowledge/community-centered approaches over assessment-centeredness [13]. Overall, student feedback indicated that SCL teaching strategies can enhance student learning outcomes and experience, even over the short timeframe of this module. Student recommendations for module improvement focused primarily on modifying the lecture content and laboratory component of the module, and not on changing the teaching strategies employed. The success of this module exemplifies how instructors can implement similar strategies to increase student engagement and encourage in-depth discussions without drastically increasing instructor effort to re-format course content. Introduction. 
    more » « less
  2. null (Ed.)
    To meet the rising demand for computer science (CS) courses, K-12 educators need to be prepared to teach introductory concepts and skills in courses such as Computer Science Principles (CSP), which takes a breadth-first approach to CS and includes topics beyond programming such as data, impacts of computing, and networks. Educators are now also being asked to teach more advanced concepts in courses such as the College Board's Advanced Placement Computer Science A (CSA) course, which focuses on advanced programming using Java and includes topics such as objects, inheritance, arrays, and recursion. Traditional CSA curricula have not used content or pedagogy designed to engage a broad range of learners and support their success. Unlike CSP, which is attracting more underrepresented students to computing as it was designed, CSA continues to enroll mostly male, white, and Asian students [College Board 2019, Ericson 2020, Sax 2020]. In order to expand CS education opportunities, it is crucial that students have an engaging experience in CSA similar to CSP. Well-designed differentiated professional development (PD) that focuses on content and pedagogy is necessary to meet individual teacher needs, to successfully build teacher skills and confidence to teach CSA, and to improve engagement with students [Darling-Hammond 2017]. It is critical that as more CS opportunities and courses are developed, teachers remain engaged with their own learning in order to build their content knowledge and refine their teaching practice [CSTA 2020]. CSAwesome, developed and piloted in 2019, offers a College Board endorsed AP CSA curriculum and PD focused on supporting the transition of teachers and students from CSP to CSA. This poster presents preliminary findings aimed at exploring the supports and challenges new-to-CSA high school level educators face when transitioning from teaching an introductory, breadth-first course such as CSP to teaching the more challenging, programming-focused CSA course. Five teachers who completed the online CSAwesome summer 2020 PD completed interviews in spring 2021. The project employed an inductive coding scheme to analyze interview transcriptions and qualitative notes from teachers about their experiences learning, teaching, and implementing CSP and CSA curricula. Initial findings suggest that teachers’ experience in the CSAwesome PD may improve their confidence in teaching CSA, ability to effectively use inclusive teaching practices, ability to empathize with their students, problem-solving skills, and motivation to persist when faced with challenges and difficulties. Teachers noted how the CSAwesome PD provided them with a student perspective and increased feelings of empathy. Participants spoke about the implications of the COVID-19 pandemic on their own learning, student learning, and teaching style. Teachers enter the PD with many different backgrounds, CS experience levels, and strengths, however, new-to-CSA teachers require further PD on content and pedagogy to transition between CSP and CSA. Initial results suggest that the CSAwesome PD may have an impact on long-term teacher development as new-to-CSA teachers who participated indicated a positive impact on their teaching practices, ideologies, and pedagogies. 
    more » « less
  3. null (Ed.)
    This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non-treatment (n = 246) groups, wherein the former participated in a two-to-three-week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann-Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 
    more » « less
  4. null (Ed.)
    This research paper describes the development of an assessment instrument for use with middle school students that provides insight into students’ interpretive understanding by looking at early indicators of developing expertise in students’ responses to solution generation, reflection, and concept demonstration tasks. We begin by detailing a synthetic assessment model that served as the theoretical basis for assessing specific thinking skills. We then describe our process of developing test items by working with a Teacher Design Team (TDT) of instructors in our partner school system to set guidelines that would better orient the assessment in that context and working within the framework of standards and disciplinary core ideas enumerated in the Next Generation Science Standards (NGSS). We next specify our process of refining the assessment from 17 items across three separate item pools to a final total of three open-response items. We then provide evidence for the validity and reliability of the assessment instrument from the standards of (1) content, (2) meaningfulness, (3) generalizability, and (4) instructional sensitivity. As part of the discussion from the standards of generalizability and instructional sensitivity, we detail a study carried out in our partner school system in the fall of 2019. The instrument was administered to students in treatment (n= 201) and non- treatment (n = 246) groups, wherein the former participated in a two-to-three- week, NGSS-aligned experimental instructional unit introducing the principles of engineering design that focused on engaging students using the Imaginative Education teaching approach. The latter group were taught using the district’s existing engineering design curriculum. Results from statistical analysis of student responses showed that the interrater reliability of the scoring procedures were good-to-excellent, with intra-class correlation coefficients ranging between .72 and .95. To gauge the instructional sensitivity of the assessment instrument, a series of non-parametric comparative analyses (independent two-group Mann- Whitney tests) were carried out. These found statistically significant differences between treatment and non-treatment student responses related to the outcomes of fluency and elaboration, but not reflection. 
    more » « less
  5. Nehm, Ross (Ed.)
    To excel in modern science, technology, engineering, and mathematics careers, biology majors need a range of transferable skills, yet competency development is often a relatively underdeveloped facet of the undergraduate curriculum. We have elaborated the Vision and Change core competency framework into a resource called the BioSkills Guide, a set of measurable learning outcomes that can be more readily implemented by faculty. Following an iterative review process including more than 200 educators, we gathered evidence of the BioSkills Guide’s content validity using a national survey of more than 400 educators. Rates of respondent support were high (74.3–99.6%) across the 77 outcomes in the final draft. Our national sample during the development and validation phases included college biology educators representing more than 250 institutions, including 73 community colleges, and a range of course levels and biology subdisciplines. Comparison of the BioSkills Guide with other science competency frameworks reveals significant overlap but some gaps and ambiguities. These differences may reflect areas where understandings of competencies are still evolving in the undergraduate biology community, warranting future research. We envision the BioSkills Guide supporting a variety of applications in undergraduate biology, including backward design of individual lessons and courses, competency assessment development, and curriculum mapping and planning. 
    more » « less