Although the last global pandemic was over 100 years ago, other localized events have signiicantly disrupted education. There have been many studies focused on educational disruption due to natural disasters (e.g., hurricanes, tornadoes, war) [34,35]. Research into educational disruption has focused on the capacity of educational institutions, access of resources to provide adequate education to students, and interruption and delay of learning for students who were impacted [12]. In this section, we describe some of the emerging literature on K-12 education during COVID-19, implications for CS education especially with regards to equity, and a framework for evaluating the multifaceted impacts of COVID-19 on CS education speciically over the last year. Although we frame the larger problem as one of equity and cite work that includes racial disparities and inequitable access to resources based on the relative wealth of school populations [33], we note that these challenges are often correlated. As the literature at this time is limited due to the emerging nature of the research and recent emergence of global events, we speciically address this in the limitations section and use imperfect theoretical alignment to hypothesize about what types of impacts would be more relevant to perceive.
The impacts of the COVID-19 pandemic on primary and secondary education, while still evolving and emerging, have been profound and widespread [15,17,38,39]. Much of the available research indicates that the pandemic has magniied existing inequities in K-12 education systems across the globe [13,49], impacting school capacity to ofer adequate training and resources for teachers to move instruction partially or fully online, as well as student access to help, technology and stable, supportive learning environments. In turn, these have impacted student participation in online and hybrid courses and, ultimately, their engagement and academic performance. The overall implications of these impacts are hard to ignore, with one study [38] indicating that the current academic cohort could lose up to one-third of a year of instructional time, a loss that will undoubtedly be exacerbated for underserved student populations [13,14]. Subject-speciic learning losses are still being investigated, but some evidence [14,23,37,40] suggests that math and possibly other STEM subject-areas may see the brunt of these losses.
Participation has been a key issue for students. Onyema et al. examined the impacts of COVID on their education system and found that many educators and students relied on technology to continue learning online, but that online education was negatively impacted by factors such as poor infrastructure, technology inaccessibility and unavailability issues, and poor digital skills [36]. In Pakistan, Adnan and Anwar found something similarśthat COVID-induced online learning faced signiicant challenges to success to the vast majority of students were unable to access the Internet due to technical and monetary issues [2]. Public media in the U.S. has echoed similar observed challenges [46].
In the U.S., the Institute of Education Sciences (IES) conducted a national survey monthly from January through May 2021 to provide insights into learning opportunities ofered by schools during the COVID-19 pandemic [21]. Results from February 2021 ofer evidence of enrollment trends across diferent groups for fourth and eighth grade student cohorts. Some of the key trends included:
• Nationally, Black and Hispanic students in 4th and 8th grades were enrolled in fully remote learning models at higher rates (55-60% versus the average 42-45%), and at much higher rates than 4th and 8th grade White students (24-27%). The percentages of 4th and 8th grade Black and Hispanic students enrolled in fully remote learning models (55-60%) was similar to both the percentage of economically disadvantaged 4th and 8th grade students enrolled in fully remote learning models (48-52%), and the percentage of ELL 4th and 8th grade students enrolled in fully remote learning models (53%-58%).
• With respect to regional diferences, it is worth noting that in almost every category the Western U.S. had signiicantly higher percentages of both 4th and 8th grade students enrolled in fully remote learning models than the other three US regions (Northeast, South, Midwest). • With respect to school location diferences, it is worth noting the discrepancy in primary learning models between Urban/Suburban and Town/Rural locations. Most 4th and 8th grade students in Urban/Suburban locations were enrolled in fully remote learning models, whereas most students in Town/Rural locations were enrolled in full in-person learning models. This trend was even more pronounced for Black, Hispanic, Economically Disadvantaged ELL students, but less pronounced for White students.
This evidence indicates some key diferences in enrollment trends across remote, hybrid and in-person learning models for key student groups. What is still largely unknown at this point is the extent to which participation in diferent learning models impacts learning outcomes. There is early evidence that when students have had to move learning online their eventual success is mediated by their access to technology, academic and peer support, and stable learning environments. We also know from past evidence that Black, Hispanic, ELL, and Economically Disadvantaged students are more likely to experience challenges with access to technology, support and stable learning environments, making their eventual success in online learning environments less likely.
Coping with these changes has not been easy for students. Cao et al. examined the impacts of the pandemic on college students and found that just over 20% reported experiencing mild to moderate anxiety, an outcome that was positively mediated by living in urban areas, family income stability and living with parents [3]. The results also indicated that the economic efects, efects on daily life and delays in academic activities were positively associated with anxiety symptoms. Anxiety is known to negatively afect emotional engagement of students in learning [25], which can further exacerbate the negative impact of learning during the pandemic.
The COVID-19 pandemic has also impacted CS education worldwide. In the U.S., the CS Teachers Association (CSTA) and the Kapor Center [27] found that fewer than 20% of teachers reported temporarily suspending CS instruction, but higher rates were reported by teachers at schools serving rural, low-income, and Black, Latinx, and Indigenous students. Further, just over 40% of teachers reported distance learning as a major challenge to instruction, with teachers at schools serving rural, low-income and Black, Latinx, and Indigenous students more likely to face these challenges. In the U.K., Mooney and Becker investigated the impacts of the pandemic on CS students' sense of belongingness and how it changed as they switched to online learning [31]. They found statistically signiicant reductions in the belongingness of students identifying as men, as well as those not identifying as being part of a minority, but more nuanced trends among students with self-identiied gender and minority status.
Three other US studies investigated the impacts of having to move CS education online from diferent angles. Focusing on middle and high school students participating in a Girls Who Code camp, McDonald and Dillon found that despite a signiicant decrease in attendance when it was moved online, those who did participate reported high morale and satisfaction [28]. Skuratowicz et al. studied virtual summer PD to help elementary and bilingual teachers learn the fundamentals of computational thinking [41]. They found that it was important to proactively address elementary teachers' barriers to technology adoption and ask teachers to include the physical in the virtual whenever possible to keep the focus on hands-on learning. In another study, Computing Research Association found that faculty reported the greatest challenge for their students moving online were family obligations, insuicient or slow internet, and mental health issues [8]. Meanwhile, faculty reported that implementing their desired instructional style virtually was challenging as was the amount of time it took to teach online. [10]. Their indings indicated that those who work within the CS discipline were signiicantly more likely than educators in other disciplines to say that they felt prepared, conident, supported by their institution, held a good working knowledge of appropriate technologies, had access to appropriate technologies, and were conident that their students could access online lessons and assessments.
CS education, especially in the United States, has focused on the challenges of broadening participation, access, and initiating new programs across the landscape. The CS for All movement and the larger research community, often use data to show a deicit narrative, make visible the negative space of implementation, and close gaps of access and participation in communities [43]. More recently there is emerging sophistication of dialog regarding the diference between broadening participation and equity [50]. Over the last decade, K-12 CS education has reformed its measures to not just identify growth in numbers of student participation, but truly ask questions about who is missing, and for students who are participating if they are succeeding at similar rates to their White and Asian peers [43].
CS education in K-12 is entering a space already fraught with inequitable access, participation, and educational experiences and outcomes for students. The disaggregated lens of not just student participation but the full ecosystem of factors impacting the ability for access, participation, and experience to have comparable outcomes across student subgroups is an important and ongoing conversation in K-12 education in the United States [26]. Although international communities may not have to contend with the same structural bias and racism that persists in the United States due to more centralized control of education systems, the lessons learned here can be important to identifying hidden factors in how COVID-19 may have afected communities diferently across the globe, as many of the same diferences in outcomes (especially the participation of women) persist in international settings.
In this study, we use the CAPE framework to speciically disaggregate the impacts of COVID-19 on various components of students' CS education experience. This allows for clarity in the observations for problems of capacity, access, participation, and experience that help identify institutional barriers compared to individual barriers when systems are stressed. We again remind the reader that our work speciically disaggregates the types of schools by socioeconomic status of the majority of students [33], as measured by free and reduced lunch, but this measure often strongly correlates with rural schools or schools that serve a high population of BIPOC students.
It is clear from this review that COVID has disrupted many aspects of education systems worldwide. In particular, the massive and sudden shift to online learning has exacerbated many equity-related education issues, including capacity of schools to ofer CS courses and extracurricular activities, access to technology, stable and supportive home learning environments, students' sense of belongingness and anxiety, and students' learning experiences. CS education has not been spared these impacts, with several studies echoing challenges in other subject areas. It is essential then, when seeking a comprehensive understanding of the impacts of COVID on education and CS education, that a systems-level framework and approach is adopted so that evidence can be developed at multiple levels. Fletcher and Warner developed the CAPE framework to help education leaders, policymakers, and researchers evaluate equity in CS education at multiple levels of educational systems [16]. The framework (seen in Figure 1) takes a systems-level approach that compels us to consider student outcomes and how those outcomes are situated within a larger initiative and policy level environment 1 . This illustrates how the framework components are interrelated and rely upon previous components.
Students are more likely to have positive CS learning experiences when they elect to participate in CS courses and programs, and they choose to participate in CS when they have access to CS courses and programs. Schools can provide students access to CS courses and programs when they have the capacity to ofer CS courses and programs. Using CAPE as a disaggregation model of indings, the New York University (NYU) Research Alliance has used CAPE to investigate the New York City (NYC) CS4ALL initiative. Although NYC's initiative is a single district, the scale of the initiative makes it unique -in 2020, there were 160,000 students in over 1,200 schools enrolled in initiative-based CS coursework [52], giving indication that the framework is robust and useful.
To answer the research question, In what ways has the high school computer science educational ecosystem for students been impacted by COVID-19, particularly when comparing schools based on relative socioeconomic status of a majority of students?, we planned to conduct an explanatory sequential mixed methods design [9,22], with the irst part of the study focusing on the quantitative data collection followed by a qualitative study that ofered explanations to the quantitative study's results. For this study, we adapted the CAPE framework to ensure appropriate measures at multiple levels connecting student participation and experiences to district and school contexts where access and capacity are key factors (Figure 2). Here, we provide context for each component:
• Capacity: A district's or school's ability to ofer equity-focused policies, resources, and funding. This includes the extent to which school leadership, staf, and teachers are efectively prepared to implement equity-focused CS courses, advising, and extracurricular activities. • Access: Students' equitable access to CS courses, advising, and extracurricular activities.
• Participation: Students' awareness of and enrollment in CS courses, extracurricular activities. This includes the extent to which they enroll in them in equal proportions. • Experience: Equitable student outcomes in CS courses and engagement in CS-focused college and career options. This includes the extent to which the course is equally and positively impacting cognitive and noncognitive outcomes, including interest in attending college and awareness of career options. In studies like this, there is typically a basis for collecting the quantitative data, such as existing instrumentation or previous research that can be referenced for creating new instrumentation. However, there is limited research on the impact of COVID-19 (or similar) on education in general and, at the beginning of this study, no previous research on the impact of COVID-19 on CS education in high schools. That is, we had limited background research to build upon and no existing instrumentation. Further, since the CAPE framework is also relatively new with limited published instrumentation on measuring the various components, we chose to modify the research design (quantitative followed qualitative) as follows (Figure 3):
• Create an instrument to collect qualitative data from open-ended questions deeply rooted in the four CAPE components. Once created, collect data from CS high school educators. • Using the qualitative data, analyze the responses, code them across CAPE, and then cultivate them as more reined questions for integration into a second instrument. Once created, collect data from a wider set of CS high school educators. • Using the results from above, create a semi-structured interview protocol for qualitative data collection and interview 6 to 8 high school educators. Steps 1 and 2 comprise the modiied exploratory sequential mixed methods study [29] and are the focus of this paper. Step 3 will conclude the explanatory sequential mixed-methods study portion of the overall study, planned for later this year. All three parts of this study were approved as exempt human subjects research by an Institutional Review Board.
Though positionality statements are not often seen as common practice in quantitative studies, we believe they should be. As researchers involved in this study, we bring various interpretations of both the qualitative and quantitative data based on our experiences with conducting research, designing and implementing qualitative, quantitative, and mixed-methods studies, and promoting equitable CS education locally, regionally, and nationally within the United States (and also internationally). With equity-focused initiatives, a large part of these programs and initiatives across the educational ecosystem guided us in our creation of instruments, collecting data, and interpreting the results.
Members of our team have extensive experience studying inequities in computer science education, from the systems-level approach to impact on learning on various population subgroups (e.g., girls/women learning CS, race/ethnicity). Several of the team members have over 15 years experience conducting this type of research. We bring our unique blend of experience and understanding of the education ecosystem (including two have taught CS in high school and college) to this study.
We are irm believers that that the COVID-19 pandemic had a serious impact on all of our lives, and the shift to online communications for many of us changed how we share and process information. We were aware of early research that indicated that student learning was deeply afected by the pandemic and that this likely impacted underserved, underresourced and underrepresented students more than their counterparts. Although we thought we may uncover evidence of this through our study from the eyes of practitioners, we were unsure of the extent of this gap and what it meant for schools that were in the middle of the process of building CS education oferings at their schools. Our hope is that this research can provide some empirical groundings for others who are interested in how practitioners perceived the COVID-19 pandemic impacts on students and their schools.
This section describes our methodology for creating an instrument to collect qualitative data from open-ended questions deeply rooted in the four CAPE components (Step 1 in Section 3) and provides the results of data collection.
As described for Step 1, we designed the cross-sectional qualitative survey to provide information about the CAPE framework and to capture data to use in the quantitative survey. We planned to send the survey to a broad set of practitioners (e.g., teachers, administrators, counselors) to collect information on how COVID-19 has impacted their CS courses and extracurricular activities, instruction, recruitment and retention eforts. 4). We used the REDCap survey platform to conduct the survey [18,19]. The irst questions focused on the ways in which the respondent perceived that COVID-19 impacted their school's or district's ability to maintain or build capacity (impact on teachers, professional development opportunities, funding/resources, and policies) in computer science education (including computer sciencerelated extracurricular activities) in terms of equity. The second question focused on the ways in which the respondent anticipates that COVID-19 will impact their school's or district's ability to maintain or build capacity in computer science education (including computer science-related extracurricular activities) using an equity lens. We customized the survey to randomly choose only one of the components for each participant to answer.
For these two questions, we ask you to consider the impact of COVID-19 on the capacity for your school/district to ofer Computer Science in an equitable manner. This includes its impact on being able to support computer science courses and computer science-related extracurricular activities in the following areas: administrators and teachers (general impact), professional development opportunities, funding, and policies.
We ask you to consider equity in terms of how support might difer between boys and girls, students of various racial/ethnic groups, students from lower-income families, and students with disabilities as well as equity for your school compared to other schools. Please be as speciic as possible in your answers. Short, descriptive answers are ine and appreciated.
Looking back at the last few months, in what ways has COVID-19 impacted your school's or district's ability to maintain or build capacity (impact on teachers, professional development opportunities, funding/resources, and policies) in computer science education (including computer science-related extracurricular activities) in terms of equity?
Looking ahead to the remainder of this academic year (2020-21) and next academic year , in what ways, if any, do you anticipate that COVID-19 will impact your school's or district's ability to maintain or build capacity in computer science education (including computer science-related extracurricular activities) in terms of equity? In this way, we could maintain coverage across the CAPE components while simultaneously keeping the survey to approximately 10 minutes per response.
4.1.2 Participant Recruitment and Characteristics. We deined participants as high school educators in the U.S. who were involved in CS education. We aimed for 100 participants, 25 for each component. In December 2020 and January 2021, we recruited via social media, through an RPPforCSfoot_2 community newsletter, and through two webinars (RPPforCS and ECEPfoot_3 ). Each participant received a $10 gift card if requested.
We closed the survey in January 2021 with 21 responses. Though the submissions were much lower than anticipated, the feedback to the open-ended questions turned out to be rich. Further, although we presented only one CAPE component to each participant, the participants' responses often applied to more than one component.
The majority of the respondents (95%) selected one of their roles as CS Teacher, while 5% (1) selected principal, 5% selected Coordinator of Makerspace, and 5% preferred not to say. 29% of the respondents stated that they had been in this role for 1 to 5 years, 29% 6-10 years, 19% 11-15 years, 10% 16-20 years, and 14% more than 20 years. Nearly half (48%) stated that they were a woman, 38% man, and 10% preferred not to say. Only 5% stated that they had a disability. Participants responded: White (86%), Asian (10%), Hispanic/Latinx (5%), and preferred not to say (10% ).
We asked in which CS networks they engaged. 81% stated that they engaged with CSTA, 14% with ISTE, and 5% with each of ECEP, Research Experiences for Teachers, CSforALL, Digital Promise, Beauty and Joy of Computing, CUE, CBEA, and Code.org. 43% stated that their August-December 2020 semester was ofered online, 19% were hybrid (in-person and online), 10% started online and switched to hybrid, and 5% started as in-person and switched to hybrid. 19% also stated Other, specifying łBOTH a full-time (5 days a week) face-to-face school AND a Virtual school onlinež, łStarted as hybrid, and have temporarily switched to onlinež, łHybrid but just switched to remote and we're told we will be back to hybrid in Janž, and łCombinations -some are fully remote; some are hybrid; some are combos of both; right now it is all remote but will change if the tier color is ok. ž
As for the participants' schools for which participants responded (15 of the 21), 6 were Title I schools 4 and 5 (66%) had over 50% of students receiving free or reduced lunch 5 . Participants indicated their location as: cities (42%), rural communities (26%), suburbs (21%), and towns (11%). Nine of the schools had over 30% White students, six had over 30% Hispanic students, and one had over 30% Black students.
To code each response along the CAPE components, one researcher coded each statement as either the Capacity, Access, Participation, and/or Experience component. Once completed, a second researcher critically critiqued the codes by examining each response and providing detailed feedback. Based on this preliminary analysis, the researchers together settled on three ways to classify the interpretation of feedback: primary impact that relates directly to the feedback provided, a secondary inference from the primary impact, or a comment that may help establish the qualitative interview protocol for the last leg of this overall study. As an example, the feedback statement łCovid has made it diicult to ofer extra help for students who don't normally reach out for help, but in a classroom you can go to them and help them without drawing attention to them. ž had one primary impact that we aligned with the Capacity component as łCapacity to provide extra help to studentsž. We considered the inferences from this and were able to provide two secondary impacts under Experience, łStudent assignment completion ratesž and łStudent gradesž, and one statement under Experience that we will consider when developing the interview protocols to digger deeper into why this occurs, łStudents less likely to engage in help-seeking behaviorsž. Again, one researcher classiied the participant responses according to these codes and the second reviewer critically critiqued them and provided detailed feedback. The researchers worked together to address all concerns.
For this instrument, we relied on a combination of external and internal face validity to ensure that we posed meaningful questions that align with the CAPE framework. First, two members of the team created the questionnaire. Then, we solicited feedback from those involved with the creation of the CAPE framework provided detailed feedback on the questions. Once we reviewed these statements and made changes, four members of our internal team then reviewed multiple revisions of the instrumentśthat is, Table 1. Response Classification. The participant feedback category reflects the CAPE component with which the participant was prompted to reflect. The researcher response classification reflects that even though the participant was given only one component, their responses (P=Primary) sometimes belonged in other components. Secondary inferences (S) were coded by the researchers. Comments that reflected an underlying reason for the impacts were tagged (IP) to be considered for Interview Protocol for a planned qualitative study. our team provided feedback, made modiications, then the instrument was sent to the team for further review. We repeated this process three times until all of our collective concerns were addressed.
In total, we coded 35 non-unique primary impacts and secondary inferences for Capacity, 7 for Access, 23 for Participation, and 50 for experience (Table 4). Since the majority (95%) of the responses came from teachers, it makes sense that the impacts are more heavily-oriented towards student experiences. However, we coded only 10 primary impacts for experience and 40 were secondary inferences (e.g., łStudents gradesž). Interestingly, the majority of primary impacts were rooted in Capacity across all of the participants' feedback, despite the component for which the survey asked them to provide feedback. Overall, the number of primary impacts with respect to future implications (35) was much lower than the past and current implications (48). We removed duplicate responses, which reduced the number of items for past and current impacts to 46 (Capacity 23, Access 3, Participation 9, and Experience 11) and future impacts to 52 (Capacity 15, Access 3, Participation 8, and Experience 21) (see Tables 2 and3. Given the rich dataset that these limited number of participants provided and the adequate representation of participants and their schools (though we would like to have seen more schools with Black students represented), we were assured that we would have suicient data to use to build our quantitative survey.
In this section, we describe our methodology for the quantitative study and provide the resultsfoot_6 .
After we extracted the impacts from the irst survey, we examined the responses for overlap, completeness, and novelty. We also considered the list of impacts from our previous research on using CAPE to disaggregate a pilot project to check for omissions from participants, adding 12 items to the list. We then went through each item to produce consistently phrased items that made sense for the category for which it appeared and grouped similar items (Table 4).
Our goal was to keep the survey to a 10-minute response time. We condensed items that were somewhat similar or seemed too narrow to reduce the number of items on the survey. We also eliminated the łfuture impactsž from this survey and added questions about anticipated future impacts to the qualitative protocol that will take place in the future. The impact of COVID-19 on the future CS educational ecosystem would be unquestionably speculative given the timing of those in the U.S. getting vaccines and the potential impact from COVID-19 variants.
Based on the open-ended feedback, we chose to have the scale responses be Increased, Stayed the same, Decreased, Unsure, and Not Applicable. To limit the length of the survey, we set up the survey to select only two components to administer to the participants. Since the participants with roles other than a teacher (e.g., administrator, counselor, etc.) would be most familiar with items related to Capacity and Access, any participant who selected those roles received items from those two components. Since we anticipated that there would be more participants who held the role of a teacher than other roles, those who selected teacher as their primary role would have a 50-50 chance of either receiving the Capacity and Access items or the Participation and Experience items. In this way, we anticipated having thorough coverage, and the survey itself would remain somewhat brief (10-15 minutes to complete). Once the survey was complete, participants could link to a second survey to enter a drawing for one of two $75 gift cards. To prevent bot responses, we used reCaptcha and set up a randomly-generated, simple addition question.
5.1.1 Instrumentation. Similar to the irst survey, we deined participants as high school educators in the United States who were involved in CS education, either as a teacher, administrator, counselor, curriculum designer, or similar role. In 2019, there were 1,050,800 public and private high school teacher positions [44]. Although there are several subject areas that high schools can ofer, we examined this with the assumption that 1% of teachers, or 10,508, taught CS. It is also estimated that 47% of U.S. high schools ofer CS in some form [6], and there are 26,727 high schools as of 2018 [45], leaving 12,561 schools ofering CS. Assuming one CS teacher per high school and averaging the diference of the two, we get 11,534. Using this as a basis and an online sample size calculator, to reach a 95% conidence level with a 5% margin of error, we needed a sample size of 384 for each component or a total of 742 participants.
Our recruitment strategy included the following: recruiting in organizations involved with CS teachers (e.g., CSTA, the RPPforCS community, the ECEP community, the ACM SIGCSE listserv), contacting researchers we knew who were involved in studies involving CS education, and contacting other organizations involved in CS education initiatives.
This strategy does not attempt to query schools that do not have existing CS education in place, nor do we feel as conident about representation from schools where CS educators are not connected with CS networks (like CSTA). We attempt to ascertain the networks of the CS educators in one of the demographic questions.
We opened the survey in February 2021 and held it open for nearly three weeks, receiving 252 responses. We removed 66 incomplete surveys. We removed one response in which the simple math question was incorrect. This brought our inal number of responses to 185. Of these, we coded 98 Capacity and Access from responses and 87 Participation and Experience. This number falls short of the sample size needed for a 5% conidence interval we hoped for. However, it does achieve a 10% conidence interval.
To analyze the data, we used PSPP [51], Excel and CalculatorSoup for conducting descriptive statistics and examined diferences between schools serving students from low-income and higher income families. A participants' answers in all questions fell into the "low-income" if a participant's response indicate that over 50% of students received free or reduced lunch.
We calculated the percentage of responses against the 3-response scale (Increased=1, Stayed the Same=2, and Decreased=3) for schools serving both low-income and non-low (higher) income. This gave us a perspective of the impact of COVID-19 on CS Education across the country. To provide context to the numbers, we ran a one sampleagainst the value "2", which represented the Stayed the Same response. We used p-values to indicate how signiicant the Increase or Decrease was from the Stayed the Same response, while also reporting conidence intervals and resulting values. In other words, we are examining whether the growth or decline is signiicantly diferent from no change.
For comparison across the two types of schools, we placed the descriptive data in a stacked bar chart. We also conducted a chi-square analysis using only the Increased (1), Stayed the Same (2), and Decreased (3) values in the test for signiicance. When conducting the chi-square analysis, we measured the diference between Stayed the Same (2) and Increase (1) across the two types of schools and we also measured the diference between Stayed the Same (2) and Decreased (3), treating these as categorical data.
For this instrument, one of the researchers conducted internal face validity among the four members of the team. The researcher modiied the survey items based on the feedback and sent for a second review and feedback cycle. For reliability, we conducted a Cronbach's alpha test for the groupings presented in Table 4 on all ive scales and with only the three scales of Increased, Stayed the Same, and Decreased. The majority of items are close to or above .70, indicating strong reliability between the items in these groups. In our results, we present items individually.
We present a description of the participant demography followed by the results from each CAPE component. For the comparative analyses in this section, we removed responses of Unsure and Not Applicable. & =86 (100%)). With respect to the time spent in the current positions, the majority of participants have been in their positions between 1-5 years ( =73 (42%); & =41 (43%); & =32 (40%)). With respect to gender, the majority identiied as women ( =99 (57%); & =57 (60%); & =42 (53%)), though not overwhelmingly. With respect to race/ethnicity, the majority identiied as White ( =156 (84%); & =81 (80%); & =75 (88%)). In summary, most participants identiied as women -though not overwhelmingly -and most identiied as White.
We also asked about the CS networks with which participants ailiated to gauge their involvement with organizations that support K-12 CS education. Only a few respondents were not ailiated with a CS network ( =31 (14%); & =23 (19%); & =8 (8%)). The majority of participants were ailiated with the CS Teachers Association (CSTA), ( =123 (54%); & =58 (48%); & =65 (62%)).
The composition of the schools was balanced between boys and girls, though there were several respondents from all-girls or all-boys schools. With respect to rural geographic locations, approximately one-third of respondents indicated that 60% or greater of their students were from rural locations ( =61 (33%); & =29 (30%); & =32 (37%)). With respect to race/ethnicity, the majority of respondents indicated that their school was composed of 81% or higher White students ( =97 (55%); & =54 (57%); & =42 (53%)). We asked participants how their school was ofering their current courses/classes. The majority of respondents reported that they ofered a hybrid model (in-person and online) ( =96 (52%); & =32 (34%); & =26 (33%)).
To identify if a participant's school served students from lower-income families, we asked, łDoes your school historically have students that meet greater than 50% free or reduced lunch guidelines?ž. The responses were nearly evenly split: Yes ( =89 (51%); & =48 (51%); & =41 (51%)). For those that answered Yes, we categorized their remaining responses to relect schools serving students from low-income families. If they selected No, we categorized their remaining responses to relect "higher income" families. Funding, Policy & Curriculum. For the Funding, Policy & Curriculum category, we irst analyzed the combined data from participants from both types of schools (Figure 5) 8 . For statistically signiicant increases and decreases when examining the combined data from all participants (Table 5), we found ive items with signiicant increases: two with extremely statistically signiicant increases (Strategies to make CS curriculum more equitable and Strategies to improve CS curriculum), one with a very statistically signiicant increase (Plans to add additional Fig. 5. Capacity for funding at schools that serve students from low-income and higher income families as measured by participant perceptions.
Next, we conducted a chi-square analysis to determine whether outcomes difered based on school type (lowvs. high-income). The results revealed that there were no statistically signiicant diferences based on school type.
Physical Resources. Combining the data from participants from both types of schools, we found that physical resources decreased extremely statistically signiicantly for Stable learning environments, very signiicantly for Physical tools used to teach CS and signiicantly for Reliable internet for learning CS (see Figure 6 and Table 6). The results of the chi-square revealed that there were no statistically signiicant diferences based on school type.
Human Resources. With respect to human resources, we found ive (and one approaching) statistically signiicant changes when we combined the data from participants from both types of schools (Figure 7, Table 7). Three items (Faculty/staf availability to ofer CS-related extracurricular activities, Faculty/staf availability to discuss taking CS courses with parent/guardian, and Faculty/staf availability to train parents of CS students), all saw extremely statistically signiicant decreases.
One item, Teacher availability to ofer extra instructional help to students, had a very statistically signiicant decrease, and one item, Faculty/staf availability to attend CS professional development had a statistically signiicant decrease. A sixth item, Faculty/staf availability to encourage students to take CS courses, was approaching a statistically signiicant decrease. The results of the chi-square revealed that the decrease in the Number of students who received information about CS courses/CTE pathways was signiicantly higher among schools serving students from low-income families ( 2 (2, N = 60) = 8.61, p = 0.01).
For Access (Figure 8 and Table 8), when combining data from participants from both types of schools, we found statistically signiicant diferences across every item, with a mix of increases and decreases. With respect to enrollment fees, which can be a barrier to access for some students, we asked if the participant's school required students to pay any fees to take CS courses in either 2019 or 2020. Nine schools serving students Fig. 6. Capacity for physical resources at schools that serve students from low-income and higher income families as measured by participant perceptions. from higher income families and four from lower-income families reported charging additional fees, and there was no statistically signiicant diference between low income and higher income schools. For these 13, we found that one school (18%) saw an increase in fees, while all others remained the same.
The results of the chi-square revealed that there were no statistically signiicant diferences based on school type.
With respect to Enrollment (Figure 9 and Table 9) for the participant data from both schools combined, we found no statistically signiicant changes in any of items. We found one item that approached a statistical signiicance decrease, Number of students enrolled in CS A courses.
We found very statistically signiicant decreases across three items (Number of students participating in CSrelated extracurricular activities, Non-CS related extracurricular activities, and Multiple extracurricular activities).
The results of the chi-square revealed that decreases in the Number of girls enrolled in CS classes were signiicantly higher among schools serving students from low-income families ( 2 (2, N = 74) = 6.48, p = 0.04).
We grouped Experience into four categories: Learning, Engagement, Other non-cognitive factors and Taking AP exams.
Experiences in Learning. In Learning (Figure 10 and Table 10), we found several statistically signiicant decreases among four items. Two items, Completion of CS homework assignments and Number of institutional hours in CS students received, indicated extremely statistically signiicant decreases. One item, Content knowledge Fig. 7. Capacity for human resources at schools that serve students from low-income and higher income families as measured by participant perceptions.
students gained in CS classes, indicated a very statistically signiicant decrease and one item Grades given in CS classes indicated a statistically signiicant decrease.
Comparisons between schools serving students from low-vs. higher-income families revealed that decreases in the Number of students receiving College Credit for Dual-Credit CS Courses were signiicantly higher among schools serving students from low-income families ( 2 (2), N = 33) = 13.69, p < .01).
Engagement by Experiences. For Engagement (Figure 11 and Table 11), we found extremely statistically signiicant decreases in engagement activities across the board when combining the data from participants from both types of schools. When comparing diferences between school types, the results of the chi-square revealed statistically signiicant diferences in Attendance in CS Classes. Schools serving students from lowincome families showed signiicant higher decreases in attendance than did those serving high-income families ( 2 (2, N = 72) = 9.32, p < .01).
Engagement Measured by Noncognitive Factors. For the remaining noncognitive factors(Figure 12 and Table 12), when analyzing the data from both types of schools, we found extremely statistically signiicant increases across two items: Understanding of the relevance of technology (39%) and Conidence using technology.
Additionally, the results of the chi-square revealed that schools serving students from lower-income families had signiicantly larger decreases in the Number of students interested in taking additional CS courses, while higher-income serving schools had signiicantly higher increases in this area ( 2 (2, N = 60) = 11.61, p < .01).
Advanced Placement (AP) CS exams. For students taking AP CS exams (Figure 13), again, we found that the item Number of students taking CS AP Exams saw a very statistically signiicant decrease. We also found that the 13).
The results of the chi-square revealed that there were no statistically signiicant diferences based on school type.
The results presented in the previous section provide empirical evidence of teachers' perceptions of the pandemic's impact on student learning, but also the impact on the capacity to ofer computer science education to all students. Overall, the student impact indings match what has been stated in early reportsśthat student learning has indeed been heavily impacted. However, we have learned that students' understanding of the value and relevance of CS has increased. In this section, we take a deeper, relective look at the indings.
Fig. 8. Capacity for access at schools that serve students from low-income and higher income families as measured by participant perceptions.
6.1.1 Capacity. Some districts and schools often provided resources like Wii and hardware devices to students, with one teacher stating that the "COVID pandemic has had no signiicant impact on my school's ability to maintain capacity for CS education. There has been no efect on funding, resources, or access to professional development." We were surprised to learn that the quantitative data supported this, and that there were no signiicant decreases related to Funding, Policy & Curriculum. Remarkably, we found ive of the nine items with statistically signiicant growth, two of which were related to equity (Strategies to make CS curriculum more equitable and recruit more diverse students into CS), while the other four saw no change. One reason for this increase may be related to any urgency administrators may have felt after a year in which racial tensions and unrest were growing within the U.S. after several highly-publicized police killings. Another reason may be that administrators felt an urgency after experiencing the necessity and relevance of computers and technology to teach studentsśa statement of which was made in responses to the irst survey of our study. More states are also requiring schools teach computer science [6], and this could have also been relected in this continued growth. School's capacity to ofer physical resources needed for teaching and learning in a CS course signiicantly decreased in three of the ive categories. These indicated the lack of ability for schools to ofer stable learning environments for CS students prior to the pandemic, notably as it relates to physical tools (e.g., robots) for teaching CS and reliable internet. These results are in line with early research [23] as well as multiple reports from the U.S. media that stressed some of the real-time impacts of remote learning on students [46].
Examining capacity with respect to human resources, we found that schools experienced statistically signiicant decreases across ive items, with a sixth approaching statistical signiicance. Given this, it is safe to state that human resources were signiicantly impacted by the COVID-19 pandemic, which support earlier indings [8].
Some items, like Capacity to ofer extra instructional help to students and Faculty/staf availability to attend CS professional development, are more nuanced issues that have not yet appeared in research, but nevertheless are essential for academic achievement [25] and for recruiting students into elective CS courses.
When comparing outcomes based on school type, we found that practitioners perspectives' indicated that schools serving students from lower-income families showed statistically signiicant decreases in the Number of students who received information about CS courses/CTE pathways compared to their higher-income counterparts.
6.1.2 Access. Variability in Access was noted in the teacher feedback, with one teacher stating that "Majority of our extracurricular clubs have stopped." while another stating that "On a positive note, due to the relative ease of organizing virtual meetings and the lack of competing sports and social activities, we've had far more extracurricular engagement in our after-school CS clubs, including from under-represented populations." We found that the quantitative data also showed a mix of increases and decreases. There was a signiicant increase in the number of CS courses ofered, perhaps relecting on the trend for high schools to add more CS courses to their curriculum [6]. This aligns with the increases in Capacity indings for Funding, Policy, & Curriculum on which Access is entirely dependent and relects on many schools' eforts to continue incorporating CS into their curriculum.
We found that the number of classes conlicting with CS classes increased signiicantly. Decisions made at the Capacity level may have included shifts in schedules to accommodate on-line learning needs, and this may have prohibited students from taking CS courses during the fall and spring semesters [23]. We also found signiicant decreases in the number of CS related and non-CS related extracurricular activities ofered, which aligns with previous indings [4,47]. We note that with the Capacity measures showing an increase across funding, policy and curriculum, the Access measures indicate that these increases were not enough to ensure that CS education oferings were maintained across the board. This may indicate that the intent of growing CS education was faced with signiicant challenges in providing the human and physical resources needed to deliver pre-pandemic levels of instruction.
When comparing the two types of schools, we found no statistically signiicant diferences between low-and higher-income serving schools, indicating Access was impacted similarly in both.
6.1.3 Participation. Among some schools, we found only one item had a near statistically signiicant decrease in the number of students enrolled in CS A courses, which is similar to results found in one earlier study [27]. When considering some of the impact on Access to informal education (Section 6.1.2), we see that student Participation in informal CS education signiicantly decreased. These indings conirm the dependency of Participation on Access the CAPE framework.
During the qualitative portion of this study, one of the teachers remarked that "Most of my diverse students (Asian, Black, Latino and female) are in the virtual school with ChromeBooks, where my counterpart in the virtual school struggles to provide the same quality curricula on ChromeBooks as I do with Desktops. " This aligns with the indings from the IES ongoing data collection in which Black and Hispanic students were enrolled in fully remote learning models at higher rates than average, coupled with the fact that those who are learning remotely face higher obstacles (e.g., stable learning environments, and in the case of CS education, adequate technology to support learning) [21]. For students from low-income families, "low income students were more likely to have to take care of younger siblings during the school day" while facing other obstacles like less likely to have parental support.
Participation outcomes based on school type difered in one area. Practitioners' perspectives indicated that lowincome serving schools showed greater decreases in the Number of girls enrolled in CS classes than higher-income serving schools.
6.1.4 Experience. Engagement was the hardest hit across all of the categories that we measured, with statistically signiicant decreases across all items. We found these decreases among both low-income and higher-income serving schools. This, no doubt, has impacted and inluenced students' experiences learning CS. We found very signiicant decreases in learning, including the number of instructional hours CS students received and their completion of CS homework assignments. We also found signiicant decreases in grades given in CS classes, content knowledge students gained in CS classes, and the number of students receiving college credit for dualcredit CS courses. These changes were comparable across low-income and higher income schools. The reduction in learning and impact on knowledge gains has now been seen across numerous other studies [14,23,37,38] and will certainly impact CS content knowledge gains for these students.
Additionally, when analyzing diferences in outcomes between low-and higher-income serving schools, we found a statistically signiicant diference in the Number of students receiving college credit for dual-credit CS courses and in Attendance in CS classes. Low-income serving schools saw signiicant decreases in both areas, which unfortunately furthers the digital divide [6,24,46,48].
For AP CS exams, again, we found a signiicant decrease in the number of students taking these exams. This is counter to the growth College Board saw for AP exams in 2020, where they found a 1% growth in students taking CS A and a 21% growth in CSP exams [7]. This requires further investigation. Also noteworthy is the applicability of both AP CS measures, including whether low-income serving schools are providing support for students to take AP CS exams. Given the reduction in learning experiences in CS, it will be interesting to see how 2021 AP exams are impactedśboth in terms of students taking the exam and also scores.
For the noncognitive factors, we found statistically signiicant increases in the interest in CS, understanding of the relevance of technology, conidence using technology, and number of students interested in taking additional CS courses. This relected our qualitative feedback from the irst survey in which responses indicated that students became steeped in the technology by attending online classes and this piqued their interest in technology by bringing home its relevance. This is interesting, particularly in light of 1) Mooney and Becker's work investigating belongingness in CS, particularly with the nuances that surrounded some of their indings and 2) our indings that indicate that learning experiences and engagement were so adversely impacted [31].
Additionally, the impact on student experiences were difered in three categories based on the type of school they attended. For schools serving students from low-income families:
• There were greater decreases in the number of students who received college credit for dual-credit CS courses. • There were greater decreases in the attendance in schools serving students from low-income families.
• There were greater decreases in the number of interested in taking additional CS courses in schools serving students from low-income families. • While schools serving students from higher-income families saw larger increases in the Number of Students Interested in Taking Additional CS Courses, schools serving students from lower income families saw signiicantly larger decreases.
These results capture some of the variabilities across the diferent schools. As one teacher put it, "We have actually thrived with our CS program in COVID. Three years ago, 11 kids in the school of 850 took AP CS. This year, when we are all remote, over 25 students are taking CS with around 200 taking AP CS. The online resources have made it thrive in a remote setting. " Other feedback continued to stress the diiculties that students from low income families faced, like caring for siblings or poorer Internet access. As one teacher stated when predicting the future impacts of COVID-19 on CS education, "Really can't predict ś my guess is that it will be a wash overall, but the individual experiences will be all over the map. "
As mentioned earlier, there is a U.S. heightened perceptional awareness of Black students' experiences which may have changed the focus more onto equity over this past year. It is still too early to tell if it will persist or be short-lived. For those conducting studies, we recommend taking this into accountśthat increases or decreases in students' experiences can be impacted by eforts of educators to address inequities. This dual-impact on education makes it harder to tease apart the COVID-19 impacts in the U.S., and we recommend researchers pay attention to both phenomenon when relecting on their own studies.
Based on our results and early indings from others like Kuhfeld et al., there will be a delayed impact, potentially scafolded, with additional future impacts that we need to consider [23]. We recommend that researchers take account that the pandemic's impact on students' learning experiences are signiicant. We can also anticipate that some students will be in a more challenged position than they would be had the pandemic not occurred.
Our study also shows that there is a signiicant reduction in informal learning, and the impacts of this may not be measurable for some time. After school and elective programs decreased across all school types. Some programs, like the JROTC-CS program with which we are involved [11], often falls within the elective or after school program genre. The lack of student access to JROTC instructors may minimize the efect of any training they received in providing encouragement for students to participate in CS education experiences. This may further the ability for students to learn, since, like formal CS curriculum, hands-on learning for authenticity is important to these programs. We recommend researchers consider how informal learning may also be impacting knowledge gains and noncognitive factors like self-eicacy.
We collected this data from practitioners, and it represents their perspectives. Perspectives can provide clues to what is happening across the CAPE components. Corroborating this data with actual data from students (e.g., actual grades, actual homework completion rates, actual number of students taking additional courses) is still needed. Further, the margin of error on responses sits at 10%, which means that the tests for signiicance (greater than 25%) on the percentages could potentially lead to a more likely range of 15-35%. This should be taken into consideration when interpreting the results.
With respect to instrument reliability, although we used Cronbach's alpha as an average measure of internal consistency, we did not conduct a factor analysis on this survey. A factor analysis would provide further information on the validity of the quantitative survey we created for this study [42].
Survey responses from the irst survey ofered robust data, but primarily from people who identiied themselves as White and who were active in CS networks. This instrument, therefore, should be examined more thoroughly to ensure it captures perspectives of other subgroups. The majority (86%) of responses were also ailiated with CS network, meaning that we lack representation from schools that are not part of a network that is designed to grow and strengthen CS education.
The view of schools serving a majority of students from low income households is based upon the use of reported percentages of students participating in the school's free or reduced lunch program. Studies have show that this is an imperfect measure of community wealth. Despite this imperfection, the US Department of Education states that žthe free and reduced price percentage is useful to researchers from an analytic perspective. ž [33] If we found no signiicant diferences for an item between schools serving students from low-income and higher income families, this does not mean that these schools are operating at an equal level. We did not attempt to provide a baseline for comparing their actual capacity prior to COVID-19. Based on prior literature, we know that schools that serve students from lower-income families ofer fewer, if any, CS courses [6].
This study was conducted prior to the receipt of CARES funding, the U.S. government's Elementary and Secondary School Emergency Relief Fund for the pandemic, that was signed into law on March 21, 2021. Therefore, CARES funding did not afect this study; however, it may afect future studies.
Overall, the shifting landscape of what was possible heavily relied on a high school's commitments to continue with the same number of CS courses and extracurricular activities that would have been ofered had the pandemic not occurred. Students and parents made critical choices around course enrollment, perhaps prioritizing "core" academic subjects and family obligations while under the stress of COVID-19. While schools trend to one type of student population, no school is homogeneous. As one participant's mentioned in the irst survey, student experiences were "wildly diferent", and some of our data bears this wildness out across schools. This work will continue to inform the research team as they seek to evaluate school-based interventions to broaden participation in computer science education. While we do not anticipate continuing to probe the indings in this paper speciically, they will inform larger studies and we hope they will be useful to the larger community in reference to any work conducted from March 2020 until school conditions in the US return to a more steady state.
ACM Trans. Comput. Educ. 6 • Monica M. McGill, Eric Snow, Luronne Vaval, Leigh Ann DeLyser, Stephanie Wortel-London, and Angelica Thompson
Indeed, a single teacher's capacity to teach students is a dynamic part of the "systems-level" capacity. One system has many individual parts, and a single teachers' professional development or bandwidth can impact students in the classroom and can be indicative of larger challenges.
Research Practice Partnership for CS community in the United States
Expanding Computing Education Pathways to have a signiicant impact on improving and broadening participation in computing education state by state in the U.S.
Within the U.S., schools with a low-income student population of 40% or greater are categorized as Title I by the federal government.
Free and reduced-lunch is a classiication under the U.S. National School Lunch Program in which students receive either free or reduced lunch costs to their families proximity to the poverty level.
This study was conducted prior to the receipt of CARES funding, the U.S. government's Elementary and Secondary School Emergency Relief Fund for the pandemic, that was signed into law on March 21, 2021. Therefore, CARES funding did not afect this study.
We carefully chose colors from a color-blind friendly palette. Complementary data is also presented in tables and in the text.
ACM Trans. Comput. Educ.