Computer science and digital literacy (CSDL) education is essential to preparing all students as the next generation of digital citizens as well as to building a more diverse workforce in computing or in fields that depend on computing. The CS Pathways RPP is a researcher and practitioner partnership (RPP) collaboration among the University of Massachusetts Lowell, the University at Albany, and three urban school districts in MA (Lowell and Methuen) and NY (Schenectady). All three districts have substantial populations of students who are underrepresented in STEM fields, including computer science [1]. Funded under the NSF CSforAll: RPP program [2], the project partners are establishing inclusive and sustainable middle school computer science and digital literacy (CSDL) programs that serve all students. The work is built on a previous project that developed an 18-hour middle school computer science curriculum focused on students developing mobile apps for social and community good [3,4].
The project started its first year with RPP team building and teacher professional learning of foundational CSDL knowledge and skills during the 2019-2020 school year [1]. The responsive shift to remote and hybrid learning during the COVID-19 pandemic brought forth the immediate need to develop curriculum resources that support remote learning. As teachers transitioned to remote instructions, we observed increased teacher anxieties related to student engagement, especially those from underrepresented groups [1]. To address these challenges and needs, the project's second year has been dedicated to developing curriculum resources that explicitly embed culturally relevant pedagogy [5,6] and support teachers adapting the curriculum in remote and hybrid learning settings.
The project used collaborative curriculum design (codesign) as a curriculum development and implementationfurthering strategy [7]. This paper explores teachers' experience in the co-design approach, professional learning and the implementation of the co-designed CSDL curriculum. In particular, we are interested in exploring the following research questions:
1. How do the project teachers implement the co-designed curriculum in their own classrooms?
2. How does the co-design process foster teacher professional learning and curriculum implementation?
3. What are the student learning outcomes, such as their interest and confidence in creating apps and learning computer science overall? II. BACKGROUND
Collaborative curriculum design (co-design) is a highlyfacilitated, team-based process in which a group of teachers, researchers and developers engage in iterative cycles of design, implementation, testing, and re-design to develop curriculum materials [8]. Research reports that co-design positively affects both teachers' professional development and curriculum innovations [7,9,10]. The co-design process enables and supports professional learning by having teachers enact an active role in the design process. By involving the professionals who engage daily with students, co-design leads to contextspecific innovative curricula that have more relevance and therefore increase the chance for success in schools [10,11].
Co-design is also well aligned with the core principles of design-based implementation research (DBIR) approaches for RPP projects [12,13]. DBIR focuses on addressing problems of practice from multiple stakeholders' perspectives through iterative, collaborative design and systemic inquiry; it emphasizes developing collaborative relationships between practitioners and researchers in developing, implementing and sustaining educational innovations [8,12]. Our approach seeks to support teachers and the whole RPP team for professional learning and agency while introducing and establishing computer science to the partnered middle schools. The co-design team included six returning teachers who completed professional learning from the first year of the project, three district leads from the partner districts, and the project's computer science and education researchers. To facilitate the codesign work, the project also organized monthly group meetings with district leads and (bi) weekly one-on-one (O3) coaching with individual teachers to develop and manage the human, material, and structural resources to support this collaborative design approach [14,15].
The bi-weekly co-design meetings provided a continual opportunity for all RPP members to build consensus on curriculum goals and learning objectives, address issues related to remote teaching and learning, explore ways of integrating culturally relevant pedagogy into CSDL, share teachers' experience of piloting the curriculum, as well as to identify the needs of resources and support from the project and the partner districts.
The project hosted twelve co-design meetings in total during the school year. The meeting topics were centered around curriculum co-design through Unit 1 to Unit 5, culturally relevant pedagogy, and remote learning. The meeting agenda was developed at the weekly research leadership meeting. The co-design meetings were usually structured into the following sessions: (1) Teacher presentation: Teachers were given homework to prepare slides to share their experience in designing and piloting a specific lesson, as well as any relevant teaching strategy or pedagogy. They then received feedback from the whole team. (2) Curriculum co-design: The whole team discussed whether and how the proposed lesson(s) or module fit into the project curriculum and planned the next steps. (3) District update: District leads shared their opinions on how the curriculum design and implementation served their own districts' needs. (4) Research team update: The research team's role was to support the co-design process and sustain the curriculum co-design products through building a Google Classroom site as the curriculum repository. Therefore, the research team gave updates on the progress of the Google Classroom work.
The project adapted a one-on-one (O3) meeting technique developed by Manager Tools (MT) [14], a business management consulting and training company. The goal of the O3 meetings was to provide ongoing support to teachers while they implemented the curriculum. The O3s were regularly scheduled, weekly to bi-weekly, semi-structured meetings between one teacher and one researcher, with an occasional second researcher to help with documentation. The frequency of the meetings and the equal sharing of teacher and researcher agendas can build trust and mutual understanding between them. O3s combined with the co-design meetings and monthly project group meetings with all RPP members supported an iterative process in which teachers, administrators and researchers gave, received, and acted on frequent feedback regarding the curriculum design process and products.
The project is informed by the three types of culturally relevant pedagogy (CRP) practices in its curriculum design and implementation: connecting with students' culture and life experience; fostering relationships with students, families, and communities; and empowering students to become change agents [6]. With the project curriculum, students learn the principles of computer science by creating "socially useful" mobile apps, broadly speaking, that matter to them and their communities. Rooting the CSDL curriculum and pedagogy in the cultural experiences and the social identities of students allows them to engage in and learn about computer science in meaningful ways. By establishing the curriculum in required middle school courses, the project is aimed to engage all of the districts' middle school students in learning CSDL.
In the co-design process, the team reviewed the recommended CRP practices [6] and discussed ways of application through the project curriculum. Researchers shared additional resources (e.g., [16]) at multiple meetings. Teachers also came up with their own ideas, enacted in their classrooms and then shared among the co-design group.
Supporting Curriculum Implementation. The project provided ongoing support to teachers while they piloted the curriculum through (bi)weekly O3s and bi-weekly co-design meetings. The project researchers also provided in-class support as requested. During the classroom visits, researchers assisted teachers in instruction and helped solve technical problems as well as provided feedback to teachers after the class.
The main product of the collaborative design is the project curriculum with resources housed on the project's Google Classroom site. It is an approximately 18-hour CSDL curriculum consisting of five units with two to six modules in each unit. The curriculum units are organized into modules, which provide video tutorials, curated lessons and recommended unplugged activities. One of the distinct features of the curriculum is that culturally relevant pedagogy is explicitly integrated throughout the units. Table I presents the overview of the units and modules connecting with the MA and NY state CSDL learning standards [17,18].
Each unit provides a clear statement of its focus and goals, which are summarized as follows. Unit 1 introduces the impact of computing and apps for social good. Unit 2 aims to help students make the first app to introduce computer science in a way that motivates students with different levels of computer science experience. Unit 3 focuses on developing students' problem-solving and programming skills so that students will be able to create an app independently. In Unit 4, students are introduced to more computer science concepts such as conditionals and functions. At the end of this unit, students are expected to create and remix more complex apps with multiple functions. Unit 5 provides guidance for teachers to organize an app showcase for students to share the apps they develop for community and social good.
This project collected three data sets to understand the status of curriculum implementation, teacher professional learning, and student learning experiences: (1) co-design meeting notes and teacher reflections shared at co-design meetings; (2) semistructured interviews with teachers who co-designed and piloted the curriculum; (3) student pre-and post-survey responses on their attitude and interest in learning CSDL.
The project researchers took notes for each co-design meeting and collected all the teacher presentations and reflections from those meetings. During the past school year, six teachers piloted the project curriculum. Five of them accepted the end-of-schoolyear interview invitations, including two computer teachers, two technology teachers and one science teacher. The primary aim of this interview was to further understand teachers' professional learning and curriculum implementation experiences. Sample interview questions include: "How has the project prepared or supported you in teaching the CSDL curriculum? What support/resources do you feel are the most helpful? and "How did you implement the curriculum this year? Did you make any specific changes to the project model curriculum?" The interviews were conducted through Zoom meetings and each lasted around an hour. The interviews were recorded and then transcribed verbatim. Two researchers analyzed the transcriptions. The results were triangulated with co-design meeting notes (including teacher presentations and reflections) and synthesized into themes focused on curriculum implementation status and teacher professional learning experiences, as reported in Sections IV.
We used pre-and post-online surveys to collect student data regarding their use of digital devices, confidence and interest in tasks to be facilitated by the program (e.g., creating apps) and attitudes towards computer science on a 5-point scale (1 = low or negative, 5 = high or positive). In addition, in the post-survey, we asked students' summative perceptions about the extent to which the curriculum changed their understanding and interest in computer science and developing apps (1 = greatly decreased, 5= greatly increased). The pre-survey and post-survey were distributed through Qualtrics by teachers during the first and last classes they taught the curriculum, respectively. The COVID-19 school restrictions complicated the process of obtaining parental consent and survey administration, resulting in a low response rate. Of the 481 participating students, 107 students replied to the pre-survey (21%); 55 students replied to the post-survey (11%), resulting in 51 students replying to both pre-and post-surveys. Two sets of analyses were conducted using R software. The first set used responses from 51 students who answered both pre-and postsurveys to examine two time-point mean score differences in student confidence in coding or programming, creating their own apps, using and creating apps to help others. Changes were also examined separately by gender and race. With the sample limitations, we were unable to compare specific ethnic groups. Instead, we categorized students as White or Non-White. The second set used post-survey responses from 55 students on their summative perceptions of changes in understanding of and interest in computer science and creating apps. Considering the small sample size and skewed data, the first set of analyses applied the non-parametric Wilcoxon signed-rank test to compare pre-and post-survey responses, while the second set used the Wilcoxon rank sum test to study the summative perceptions. Hedge's g was calculated to indicate the effect sizes.
Implementation Overview. With the COVID-19 pandemic related disruptions, the project encountered a significant challenge of teacher continuity. Teachers from one district had to discontinue their participation, as they either lost their jobs or were reassigned to new positions. Many teachers from the other two districts were also overwhelmed by their new assignments with the shift to remote teaching. In total, six teachers from four middle schools piloted the CSDL curriculum in computer, technology and science classrooms, reaching 481 students.
Note. a Teacher A and D had the same students rotating with them during the school year.
Table II presents the overall implementation information. All the participating schools started teaching remotely, but one district switched to a hybrid mode in April, 2021. At School I-A, Teacher E piloted the curriculum in the computer class. At school I-B, the technology curriculum focuses substantially on digital technology. Therefore, when implementing the CSDL curriculum, Teacher A and Teacher D collaborated, with 7th and 8th grade students rotating between the technology course and the computer course through the school year. At school II-B, the technology curriculum focuses on basic engineering, problemsolving and hands-on activities. Teacher B introduced computer science as part of the technology curriculum. Two science teachers from School II-A piloted it as an optional unit for 7th and 8 th graders at the end of the school year. Implementation Process. Using the interview data supplemented with meeting notes, teacher presentations and reflections collected from the co-design meetings, we further analyzed how teachers implemented the project curriculum. All six teachers except Teacher F participated in the interview. While starting with the 18-hour model curriculum [3], teachers were given full autonomy in adapting and piloting the curriculum. They were also encouraged to remix materials shared by other teachers during the co-design process. Notably, when collaborating on teaching the curriculum: Teacher A focused on introducing the basics of CSDL; Teacher D worked with students on further developing apps.
Table III summarizes the total instruction time for the CSDL key concepts [17,18] and specific topics introduced by each teacher. These topics were spread into four of the five key concepts of the state standards: impact of computing, computational thinking & programming, cybersecurity, and digital literacy. In particular, Teacher B was able to implement more CS activities than expected as the remote learning setting limited the feasibility of physical engineering and building activities for the technology course. Teacher C was only able to start a few introductory CSDL lessons in the science class after the state tests were completed. Teachers also reported varied paces in teaching this curriculum. Overall, teachers started with two to four periods (approximately 50 minutes per period) introducing the field of computer science and apps for social good, one period on selecting and using sound and images, one or two lessons on cybersecurity, and more than half of the total time on computational thinking and programming focused on creating apps. All the teachers concluded the course with an app showcase and discussion session where the students presented their final projects and shared ideas.
Teaching Strategy and Pedagogy. All teachers started Unit 1 by introducing the broad view of computer science, using inspirational videos and/or virtual classroom visits from computer science professionals with diverse backgrounds, to motivate students and connect computer science to real-world examples. For example, one teacher had a virtual visit from two female engineers who discussed women in computer science and the gender pay gap. The class then completed discussion boards on what they learned from this visit.
Teachers also tried a few pedagogies introduced by the project, including the use of CS unplugged activities [19], pair programming [20], and culturally relevant pedagogy (CRP) [6]. When implementing CRP practices, one teacher specifically focused on addressing the issue of the gender gap in computing, while the other teachers mainly focused on community. Starting with helping students identify personal interest and passion and understand their own culture and community needs, teachers facilitated students to develop ideas and then create apps as the final project. For example, Teacher B explicitly introduced the concept of community, and had the class discuss what they considered as their community and how it could be different for different people. The students created vision boards to outline their app ideas. Another teacher connected with the school's social study resources, through which the students interviewed people in their own community to understand issues the community was facing.
Teachers reported overall positive experiences in both the co-design process and curriculum implementation. They enjoyed the remote professional learning experience being flexible but also productive. Here we present the results focused on how the co-design process supported community building, teacher learning and curriculum implementation as well as some challenges and outcomes of the curriculum pilot.
Co-design for Community Building. First, teachers reported that the co-design process enhanced their sense of community with peer teachers. The co-design meetings helped build the relationships among the project RPP members through continual conversations. Notably, teachers appreciated that the project brought the district leads into the meetings and fostered communication across the whole team. For example, Teacher E mentioned (s)he enjoyed the meetings where teachers got a chance to present their efforts, ask questions and share opinions in front of the district leads.
Teacher E: "[It] was helpful to hear from district leads, to hear from other districts what their struggles were …[we] also got a chance to share our efforts… It was good to bounce ideas off each other. Because of the slide-share format, everyone gets a chance to have a say."
In addition to the bi-weekly co-design meetings, teachers also appreciated the regular check-in meetings (O3s) with researchers, which also helped break the isolation they faced.
Teacher B: "In the beginning, I kind of felt like I was on my own… Then I started meeting with other teachers in the codesign meetings… Also, I started my weekly check-ins. Those were huge for me. Those were super helpful."
Teachers spoke highly of the O3 meetings in providing individual support. While group meetings allowed broad discussions on collective needs, the one-on-one setting was less intimidating and more customized for each teacher while teachers started piloting the project curriculum. The O3 meetings provided a forum for constructive dialogues between teachers and researchers.
Teacher C: "The most useful thing that I had was being able to get that feedback week to week and kind of let [the research team] know what I was doing and check in on the sense of how what I'm doing matches up or doesn't match up with what [the project] is trying to do… I can ask questions if there's something that I realized I don't know how to do as I'm trying to set up a curriculum."
Teachers all reported they gained ideas and resources for implementation from this year's professional learning. The codesign meetings provided a pathway for teachers to see multiple ways of teaching the project curriculum. Teachers learned from each other and brought ideas back to their own classrooms. For example, one teacher shared the experience of having two female Amazon engineers visit the class at one co-design meeting. Another teacher got inspired and arranged virtual visits from local computing professionals to her own classes. Teachers uniformly reported gaining more confidence and a deeper understanding of the CSDL curriculum. They also felt strongly supported while piloting the curriculum. Teacher B: "I'm not as scared as I was a year ago. I'm much more confident in my own abilities than I was a year ago... I'm much more confident and comfortable. I don't have all the answers, but I know where to get them."
In addition, teachers found their co-design efforts on creating the curriculum resources paid off as they were able to access the resources easily and potentially benefit more teachers. The project's Google Classroom site provided a guide of the curriculum sequence as well as a collection of resources for teachers to create their own curriculum materials.
When asked what they enjoyed most about implementing the project curriculum, teachers shared some positive outcomes they observed in the classrooms. First, teachers observed that the majority of their students were excited about learning the CSDL modules, which were different from what they had been learning in the rest of the course during the pandemic. The excitement supported student engagement in the class. Notably, two teachers commented many of their students would like to spend more time working on further developing the apps. Since the curriculum focused on introducing the basics of CSDL, Teacher C reported she did not plan enough time for students to advance their apps. This teacher reported that challenge as "a good problem to solve" for next year. Teacher A also reported (s)he was thrilled by students asking permission to continue working on the projects after school even if (s)he did not assign any homework. Those students realized the importance of learning CSDL and really enjoyed making and improving their own apps.
Teachers also shared that they were really impressed by their students' problem-solving skills and creativity in creating the various apps. For example, Teacher D shared one example of student apps: Teacher D: "They were interested in the idea of litter and recycling and how to sort trash. So, they actually created a game for this... They got to do some of the things they wanted to do, like the gaming aspect of it, but also with a way to make it interesting for someone else to actually use the app and see how they could disseminate it."
Meanwhile, teachers shared a few challenges they faced when implementing the curriculum. Many of the challenges were related to remote teaching in general. For example, teachers felt it was difficult to either know student engagement or monitor learning progress in an online class. In terms of implementing CS pedagogies, teachers also reported it was more difficult to run CS Unplugged activities or organize pair programming remotely.
Beyond those remote teaching related challenges, teachers also suggested an action item for organizational support. While teachers appreciated the benefits of the project's Google Classroom site as the curriculum repository, the technical settings within the districts did not allow teachers to easily integrate it within their local Google Classrooms, which were administrated by each district. This remains a challenge to the project and requires technical support from each partner district.
In addition to the teacher self-reports through interviews and co-design presentations and reflections, the student surveys provide another data source to understand the outcomes of the curriculum implementation. Table IV presents the demographic information of the 51 students who answered both pre-and post-surveys and the 55 students who filled out the post-surveys. Table V presents the race/ethnicity information of the 51 students and the students who filled the pre-survey. Among the 51 students, 21.6% were White students, 72.5% were Non-White students, and 5.9% were of indeterminate race. For gender, 49% of the students identified as Boys, 41.2% as Girls, 5.9 % preferred not to say, and 3.9% did not answer this question. Note: 1 Unknown indicates no answer, prefer not to answer, or unable to determine.
Based on the results of pre-and post-survey analyses, we found a statistically significant increase in students' confidence in creating their own apps, with a medium effect size (diff. = 0.64, SE = 0.17, p<.05, Hedge's g = 0.55). Analysis by gender suggested the increase in the confidence held for both girls (diff. = 0.79, SE = 0.25, p<.05, Hedge's g = 0.70) and boys (diff. = 0.67, SE = 0.25, p<.05, Hedge's g = 0.52), while girls had a relatively greater effect size. Alternatively, analysis by race suggested the increase in the confidence held only for Non-White students (diff. = 0.66, SE = 0.18, p<.05, Hedge's g = 0.52) but not for White students (diff. = 0.64, SE = 0.45, p = .22, Hedge's g = 0.39).
On the other hand, both students' confidence in coding or programming (diff. = 0.26, SE = 0.16, p = .15, Hedge's g = 0.23) and confidence in creating their own apps to help others (diff. = 0.31, SE = 0.17, p = .09, Hedge's g = 0.27) were slightly increased but these increases were not statistically significant for overall samples. The analysis by gender or by race also suggested no statistically significant increases with one exception. That is, only White students perceived a statistically significant increase in their confidence in creating their own apps to help others (diff. = 0.82, SE = 0.33, p < .05, Hedge's g = 0.70).
Results of the four summative post-survey items indicate that the means of these items were all above 3.0: understanding of computer science (M = 3.46, SD = 0.97); understanding of creating apps (M = 3.45, SD = 0.97); interest in computer science (M = 3.23, SD = 1.24); and interest in creating apps (M = 3.26, SD = 1.28). The means were computed using scores on a 5-point scale and means above 3 suggested students perceived increases in both understanding of, and interest in and creating apps. In addition, results suggested the aforementioned increases were perceived equivalently by boys and girls as well as by White and Non-White students with an exception: White students perceived greater increase in their understanding of computer science than Non-White students (diff. = 0.74, SE = 0.14, p < .05, Hedge's g = 0.79).
V. SUMMARY AND DISCUSSION While the COVID-19 pandemic and the shift to remote learning brought many implementation challenges, the project was able to implement a collaborative design approach to facilitate curriculum development and foster teacher learning and curriculum implementation. Teachers adapted and piloted the co-designed CSDL curriculum into computer, technology and science classes with various curriculum scope, duration and pedagogies. We also observed some positive preliminary results in both teacher learning and student learning. The co-design process facilitation to achieve its dual goals. Integrating the one-on-one coaching with the co-design group activities can better meet the needs individual teachers as well as the collective efforts of the whole RPP team.
It is noteworthy that this study encountered a significant challenge with low response rates of parental consent and student surveys. Because of the sample limitations, the analysis of student learning experiences should be considered exploratory. We hope to conduct more extensive studies on student learning outcomes with significantly larger samples of student data while more teachers are implementing the curriculum this school year.
Currently, all three partner districts have further conceptualized their district plans for middle school computer science: One district has decided to integrate the project curriculum into 8th-grade Civics; another district has offered the opportunity to all subject area teachers and has recruited a group of teachers in 6-8th grade technology, science, and ELA; the third district has decided to integrate the CSDL curriculum with a new course on equity and social inquiry for grade 6-8th, expecting to add newly hired teachers into the project.
Looking ahead, the project will focus on supporting the new cohort of teachers in professional learning and curriculum adoption. With the leadership of the teacher leaders from cohort one and the district leads' support, we expect to facilitate more co-design work and build professional learning communities within each district in the coming year.