Although computer science (CS) education interventions can be studied in many ways, the recent CAPE framework [1], [2] provides a method for disaggregating various aspects of an intervention's impact across four major components: (C)apacity to offer CS education, student (A)ccess to CS education, student (P)articipation in CS education, and the (E)xperiences of students who participate. This promising framework can be used to study research and used to formally and formatively evaluate interventions.
The intervention that our team evaluated is the JROTC-CS Demonstration Project, a pilot project developed by CSforALL to bring CS and Cybersecurity education to high schools across the United States that offer Air Force Junior Reserves Officer Training Corps (JROTC) [3]. As a pilot project, we wanted to understand the complexities of the program and how the various factors, inputs, and outcomes correlate, particular with the focus on reaching marginalized communities and subgroups of students. This includes the changes in course offerings (access) and participation in courses by students (participation). For our evaluation of this program, we used the CAPE Framework.
Although, to date, we have provided evaluation and research within each of the individual components of CAPE, this paper focuses on the impact of the changes to the course and extracurricular offerings (access) by the schools, and the students enrolled in the courses (participation). Specifically, we wanted to answer the following key evaluation question How do schools' engagement in an intervention designed to build equitable CS and Cybersecurity education capacity impact schools' course offerings and students' participation in these courses?.
According to the CAPE Framework [1], schools that offer equitable capacity to create equitable access to courses can lead to a diverse set of students enrolling (participation). Based on this, we postulate that a school engaged in an equity-focused, multi-school intervention to bring the Advanced Placement (AP) CS Principles (CSP) course to schools, the more likely their schools will: 1) offer AP CSP courses, 2) offer related CS courses, 3) offer Cybersecurity courses and extracurricular activities, and 4) more students who are underrepresented and underserved in the field of computing and in CS education (e.g., students with disabilities, Black and Hispanic students, non-male students) will enroll. Studying the impacts of an intervention on Access and Participation through the CAPE Framework is a new perspective on the topic of increasing equity-focused CS education in PK-12 settings. Access outcomes are focused on the course and extracurricular offerings at the schools, while participation outcomes speak to enrollment in the course and extracurricular offerings. This evaluation is important for two reasons: 1) CAPE is a new, equity-focused framework, and testing the framework across various interventions can provide validation of its usefulness and 2) for this particular intervention, it provides another avenue of understanding the interventions' impacts. Therefore, this work may be of importance for evaluators as well as researchers investigating CS education interventions. It can also be important for understanding more about how a multi-school intervention fared, particularly when schools were facing many challenges during 2020-21 due to the pandemic.
To provide context for this evaluation, we provide a brief background on the JROTC-CS Demonstration Project and the CAPE framework. We follow this with a light overview of key findings within the individual components of capacity, access, and participation.
The JROTC-CS Demonstration Project is a pilot of a multi-year intervention designed to bring CS and cybersecurity education to 30 high schools in the United States [4]. These targeted high schools offer the Junior Reserves Officer Training Corps (JROTC), a program implemented by the Air Force and other Armed Forces to build leadership and citizenship skills among participating high school students. The 30 participating high schools with an Air Force (AF) JROTC program were selected by the organizing organization, CSforALL, to recruit schools in a range of diverse geographic locations, race/ethnicity of the students they serve, and Title I status (i.e., schools that serve students from lower-income families).
Over 500,000 JROTC cadets attend over 3,000 high schools across the US [4], and cadets represent a highly diverse population, with approximately 60% of students from marginalized racial/ethnic background and 40% female students. Cadets are also strongly represented in Title I schools, indicating that the schools serve students from predominantly low-income families, with over 50% of cadets at schools with Title 1 status. The JROTC-CS Demonstration Project is a unique blend of workshops, webinars, and activities that high schools with JROTC programs can engage in to build CS and Cybersecurity education into their offerings (e.g., courses and extracurricular activities). If the Project shows promise in the pilot, it can help address the fact that two-thirds of high schools with JROTC programs do not offer AP CS courses.
For each of the 30 schools, there was a 3-5 member team (hereinafter referred to as a school team) comprised of administrators, JROTC instructors, CS/Cybersecurity teachers, and guidance counselors. These schools worked closely with the JROTC-CS implementation team, attended a two-day kick-off workshop in February 2020 and engaged in various activities promoted by the JROTC-CS implementation team after the kick-off workshop. The workshop was a modified version of the SCRIPT workshop (the Strategic CSforALL Resource & Implementation Planning Tool provided by CSforALL [5]. The tool and the workshop enabled schools to reflect on their own community values, beliefs, resources, and needs to develop a feasible plan for implementing sustainable Computer Science (CS) and Cybersecurity curriculum. Each of the teams left the workshop with a comprehensive 3-month, 6-month, and one-year plan for implementation. The workshop also had sessions for guidance counselors, teachers, JROTC instructors, and administrators to provide guidance and resources for their schools. The workshop was also preceded by one-day congressional visits from teams from 10 schools to inform policymakers about the JROTC-CS initiative.
Post-workshop activities included webinars to support the schools' implementation plans. The webinars facilitated discussions for school teams to share information about the JROTC-CS initiative with school, district, and state officials, reach out to industry to invite guest speakers to their classes or engage in field trips, and share information and resources for CS teachers. CS teacher resources that were shared included teacher professional development opportunities (for which stipends were provided), software and hardware resources for teaching CS and Cybersecurity, and professional learning networks for teachers (e.g., the Computer Science Teachers Association (CSTA)).
The schools involved were at various starting points in their CS and Cybersecurity curriculum. While many schools did not offer any CS or Cybersecurity curriculum, a few offered an Advanced Placement (AP) CS A course, but did not offer the precursor course, AP CS Principles. Likewise, one offered a Cybersecurity course, but did not offer any AP CS courses. Thus, school teams created plans based on their particular needs.
Fletcher and Warner created the CAPE Framework to emphasize the various aspects of CS education that could be impacting its equitable delivery to students [1]. CAPE stands for Capacity of schools to be able to provide equitable CS education, equitable Access through CS education (e.g., courses and extracurricular activities), equitable Participation of students partaking CS education, and equitable Experiences and outcomes of students participating in CS education (Figure 1). CAPE's systems-level approach includes, at a minimum, teacher and student level outcomes, and considers how these outcomes are situated in and impacted by the larger initiative and policy level environment [6].
While the CAPE framework does not include a definition for equity, we have consistently used a definition of equity from the UC Berkeley Initiative for Equity, Inclusion, and Diversity. This definition states:
The guarantee of fair treatment, access, opportunity, and advancement while at the same time striving to identify and eliminate barriers that have prevented the full participation of some groups. The principle of equity acknowledges that there are historically underserved and underrepresented populations and that fairness regarding these unbalanced conditions is needed to assist equality in the provision of effective opportunities to all groups (Definition from [7]).
While we are aware of the differences between equity, diversity, and inclusion, this particular paper focuses on the course offerings (access) and who actually enrolled in these courses (participation). However, equity is embedded in each of the four logic models for each CAPE component and we only present our access and participation logic models in this paper. Further work is in the process of being published on issues related to capacity and the actual experiences of different students who participated in CS and Cybersecurity education at their schools to better understand if student outcomes and experiences are equitable across various groups.
With respect to access, rural schools are often unable to offer CS courses in their high school and can be the last schools in their state to do so [8], [9]. As of 2020, in the U.S. 57% of suburban schools while only 43% of rural schools offer CS in their high schools [10]. Schools with more resources are also more likely to provide CS education [11].
With respect to participation, Torbey et al. found that participation in high school CS courses is diminished by a particular prerequisite math course (Algebra I) [12]. Inequities in participation abound, with previous research showing vast differences in participation among girls and historically marginalized groups McFarlane and Redmiles, [10].
The Workshop was impactful on attendees, from those who already had a CS and Cybersecurity educational program in their schools to those who had yet to start. Although it is beyond the scope of this paper to provide specific details on the intervention, the impacts from the workshop included:
• While many attendees had a moderate understanding of the impact of CS in the lives of their students, they gained measurable knowledge about the importance of CS for all of their students.
• The Workshop had an impact on attendees' understanding of the relationship between Computer Science, Cybersecurity, and JROTC in their schools.
• Attendees had a much better understanding of the roles each person on their team needs to fulfill to make the JROTC-CS initiative a success in their schools, including how to build their own educational offerings, what their role is in this process, and the Project's importance to their schools.
• Attendees left with a workable plan for developing and implementing their schools' goals, knowledge about teacher professional development offered from partners, and knowledge about other programs to provide cadets with more Cyber Security learning opportunities.
• The workshop guided the school teams to clear next steps and energized them to move forward.
3 Methodology To answer the key evaluation question question, How do schools' engagement in an intervention designed to build equitable CS and Cybersecurity education capacity impact schools' course offerings and students' participation in these courses?, we needed to ascertain offerings and enrollment in prior years' at the schools for comparison. We asked for three years of data prior to the course participating in the JROTC-CS program (2017-18, 2018-19, and 2019-20) and then data from the first year after joining the program (the 2020-21 school year).
Based on the CAPE Framework, the summarizing question to ask regarding Access in the context of our evaluation is: Who has access to CS and Cybersecurity education? Therefore, we evaluated Access based on each schools' actual offering of CS and Cybersecurity education (e.g., CS and Cybersecurity courses and extracurricular activities). If schools implemented the plans they created at the JROTC-CS SCRIPT workshop, then there would be an increase in the number of CS and Cybersecurity courses and extracurricular activities at the schools. Given the Project's focus on ensuring that Title I schools (schools serving students from low-income families) and schools with underrepresented students see growth that is on par with schools outside of the project, we would expect to see an equitable increase across all schools. Figure 2 shows our base logic model for Access.
As a unit of measure, the goals of the Project are to have all schools in the cohort, regardless of the students that they serve, offer AP CSP courses and Cybersecurity education, likely in the form of an extracurricular activity such as CyberPatriots which is a cybersecurity extracurricular activity developed by the Air Force. We would also expect to see an increase in the number of schools offering other CS courses, including AP CS A, and this rise would be equitable across schools. Finally, we would also expect to see Cybersecurity Education rise across schools and that this rise would be equitably distributed. Across all measures, we would also expect to see accelerated growth beyond what the schools experienced the prior three years.
Based on the CAPE Framework, the summarizing question to ask regarding Participation is: Who enrolls in CS? Therefore, our logic model (Figure 3)for Participation focuses on enrollment of cadets and other students at high schools in CS and Cybersecurity education that schools offer (i.e., CS and Cybersecurity courses and extracurricular activities). If schools implement their plans created at the JROTC-CS SCRIPT workshop, we would expect to see an increase in the number of cadets and other students in the high school enrolling in CS and Cybersecurity courses and extracurricular activities at the schools. We would also expect to see more students (cadets included) taking AP CS Principles. Given the Project's focus on ensuring that underserved schools and schools with underrepresented students see growth that is on par with the other schools, we would also expect to see that this increase is equitable across all schools.
As part of our analysis, we surveyed the school teams to understand how COVID-19 impacted the school's implementation of their plans to include CS and Cybersecurity education in their schools. Of the 23 schools that responded to the open-ended question about COVID-19's impact in the Capacity survey, only one school reported information about the impact on cadet and student participation (measured by student enrollment), and they noted that their school had an increase in participation.
Since the focus of this paper is the impact of capacity building efforts on access and participation, we spend the bulk of this section on those areas. In this evaluation, a course refers to a subject taught within a curriculum of one teaching term (semester), while classes are offerings of that course. For example, AP CSP may be a course offering, while it may be offered four times in a semester (four classes).
The evidence indicates that there was a significant impact on all school types. CS and cybersecurity courses and classes increased across many schools (see Figure 4). This includes a nearly three-fold increase in the number of AP CSP courses offered and a nearly seven-fold increase in the number of classes offered (see Table 1). Cybersecurity extracurricular activities also increased by 100%, showing that the efforts of working with JROTC instructors at the school also likely had an impact on these offerings.
Across the participating schools, Title I schools saw more growth in the CS and Cybersecurity courses and classes offered when compared to non-Title I schools, indicating that the Project had more of an impact on Title I schools (Table 2). When we examined AP CSP courses in particular, the evidence indicates that Title I schools had a much greater increase in the total number of Figure 4: Changes prior to and after the JROTC-CS intervention, asked as a separate question (n=14) independent of questions aksing actual enrollment numbers, which leads to a slight difference in evidence reported by the schools.
Similar to the Title I findings, the Project had a significant impact on BIPOC+ majority schools versus other schools (Table 3). There was an increase in CS/Cybersecurity courses and classes offered, with an 89% increase in courses and a 106% in classes for schools serving a majority of BIPOC+ students. Schools serving a majority of non-BIPOC+ students saw an 118% increase in the number of CS/Cybersecurity courses offered as well as an 81% increase in the number of classes offered.
For AP CSP courses and classes only, the evidence suggests similarly that slightly fewer CS and Cybersecurity classes were added to schools serving BIPOC+ majority students, but the number of AP CSP classes offered was signifificantly greater (see Table 4).There was a 250% increase in courses and an eight-fold increase in classes offered by schools serving majority BIPOC+ students.
There was a three-fold increase in both the number of courses offered and classes offered in schools serving a majority of non-BIPOC+ students.
The evidence also suggests that the number of extracurricular activities were not quite on par, with Title I schools seeing a significantly greater increase in their offerings of CS-related and Cybersecurity extracurricular activities. For schools serving a majority of BIPOC+ students, there was no change while their counterparts saw a five-fold increase in their offerings from 2019-20 to 2020-21. Overall, the analysis indicates that non-Title I schools were significantly less likely to meet the goal of offering more extracurricular activities.
Since participation is measured by enrollment, we measured the number of students enrolled from participating schools. Of those schools reporting data, we see that the number of students enrolled increased after the Project started.
Between 2019-20 and 2020-21, the number of students enrolled in CS and Cybersecurity courses grew 91% (see Figure 8). The number of students enrolled in AP CSP courses grew five-fold (see Table 5).
While students of diverse race/ethnicities increased, several BIPOC+ groups saw greater enrollment than White, non-Hispanic students (see Table 6). There was a 120% increase in the number of Hispanic students taking CS/Cybersecurity courses, a 33% increase in Black students, a five-fold increase in Asian students, and a ten-fold increase in American Indian students.
Within all courses offered, there was not much data reported for student enrollment by gender, although there was more data reported for gender than race/ethnicity. We only report here the total distribution of students by gender, though the lack of reporting across student gender means this evidence is weak. Given this limitation, caution should be used when interpreting the data.
We note that there was an eight-fold increase of boys taking AP CSP courses, while there was a thirteen-fold increase of girls (see Table 7).
Since enrollment data was not collected for extracurricular activities, we limit this section to student enrollment in formal course curricula across all four categories collected (AP CSP, AP CS A, other CS courses, Cybersecurity). We note across each that there was an increase in student enrollment in both Title I and non-Title I schools, and it appears that there was a larger increase among Title I schools (see Table 8).
Comparing percentage of students at the Title I schools who took CS and Cybersecurity courses versus non-Title I schools who took these courses across 2019-20 and 2020-21, the evidence indicates that there was more than a 200% increase in the percentage of students who took AP CSP courses at Title I schools, while there was a slight decrease at non-Title I schools (see Table 9). This indicates that the Project had a greater impact on Title I schools. Analyzing the percentage of students who took CS/Cybersecurity courses at Title I school compared to non-Title I schools, the evidence indicates that there was an over 200% increase in the percentage of students who took AP CSP courses at Title I schools, while there was a slight decrease at non-Title I schools (see Table 10). This indicates that the Project had a greater impact on Title I schools.
The evidence indicates that there was a greater increase in students taking CS/Cybersecurity courses among BIPOC+ majority serving schools (see Table 11. Comparing the percentage of students at the BIPOC+ majority serving schools who took AP CSP courses compared to non-BIPOC+ schools across 2019-20 and 2020-21, the evidence indicates that there was a three-fold increase in the percentage of students at both types of schools.
5 Discussion and Limitations CS education capacity building efforts can include a wide spectrum of supports that enable states, districts, and schools to employ appropriately trained teachers, as well as funding, policies, and curriculum that enable schools to offer high quality CS education equitably for all of their students. We used the CAPE Framework to evaluate the effectiveness of the JROTC-CS Demonstration Project, exploring the impact of this project's capacity building efforts on access to course offerings and student participation in participating schools. The evidence indicates that there was an impact on all school types as a result of the project. Schools' capacity to build and maintain a CS and cybersecurity education increased across many schools regardless of Title I status or student body diversity.
However, across the board, the Project had a more significant impact on Title I compared to non-Title I schools. Likewise, the Project had a more significant impact on BIPOC+ majority schools compared to other schools. This is significant, since Title I schools are historically under resourced to meet the needs of students at their schools.
The evidence also indicates that there was an impact on enrollment of students taking CS and Cybersecurity courses, with the number of students enrolled in CS and Cybersecurity courses growing 91% and the number of students enrolled in AP CSP courses growing five-fold.
We recognize that there are limits to the data reported by the schools, and in the future, actions are being put into place to ensure broader data collection across all participating schools. It is important to note that the Demonstration Project was piloted in February 2020, just a few weeks prior to the start of the global COVID-19 pandemic. Despite the great strides the implementation team made in bringing CS and Cybersecurity education to participating schools, based on our experiences as evaluators, some of the relationship-building that would have transpired in a program like this were hindered. Therefore, we likely would have seen significantly more positive impact from the program that what actually transpired.
As another limitation, we used actual numbers of students, not percentages of students from the school as well, which would be another interesting analysis to do to further understand what differences, if any, exist across the schools. As additional data is collected over the years, percentage increases from one year to the other will also yield more meaningful results that can be used to inform the intervention as it scales up.
The opportunity to engage in CS or Cybersecurity courses is one more inequitable practice within our K-12 environment. With that being said, this reality presents a unique opportunity for the Demonstration Project as it moves forward to potentially include a control group, which would provide more scholarship regarding the gap in access and enrollment of CS and Cybersecurity courses. This would provide a comparison for school within the Project and outside of the Project, which would ultimately add to the scholarship regarding best practices for expanding Capacity, Access, Participation, and Experience to all students in K-12 settings.
Our evaluation of the JROTC-CS Demonstration Project across Capacity, Access, and Participation has concluded. The results show that this unique intervention to bring CS and cybersecurity education to high schools across the U.S. was successful in increasing the needed Capacity to provide this education. It was also successful in increasing student enrollment in CS and Cybersecurity education. These results were especially true among historically marginalized students at the participating schools.
The phenomenon of inaccessible courses to historically marginalized students in K-12 settings is not uncommon in historical or current practices. With this in mind, however, the participating schools increased their capacity to build and maintain CS and Cybersecurity education, increasing the number of courses and classes offered, especially among Title I and majority-BIPOC schools. Additionally, enrollment among girls and BIPOC+ students increased more than enrollment among boys and White, non-Hispanic students.