More Like this

1. Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review

   https://doi.org/10.1109/FIE43999.2019.9028652  

   Offenberger, Spencer ; Herman, Geoffrey L. ; Peterson, Peter ; Sherman, Alan T ; Golaszewski, Enis ; Scheponik, Travis ; Oliva, Linda ( October 2019 , Proceedings of the 2019 IEEE Frontiers in Education Conference (FIE))

   We analyze expert review and student performance data to evaluate the validity of the cci for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the cci, and 142 students from six different institutions took the cci as a pilot test. The panel reviewed each item of the cci and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the cci to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the cci. This evaluation showed that the cci is sufficiently reliable for measuring student knowledge of cybersecurity and that the cci may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the cci. 

   more » « less
2. Psychometric Evaluation of the Cybersecurity Concept Inventory

   https://doi.org/10.1145/3451346  

   Poulsen, Seth ; Herman, Geoffrey L. ; Peterson, Peter A. ; Golaszewski, Enis ; Gorti, Akshita ; Oliva, Linda ; Scheponik, Travis ; Sherman, Alan T. ( March 2022 , ACM Transactions on Computing Education)

   We present a psychometric evaluation of a revised version of the Cybersecurity Concept Inventory (CCI) , completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned. 

   more » « less
3. Psychometric Evaluation of the Cybersecurity Concept Inventorym

   Poulson, S. ; Herman, G. ; Peterson, P. ; Oliva, L. ; Golaszewski, E. ; Gorti, A. ; Scheponik, T. ; Sherman, A. T. ( January 2021 , The ACM transactions on computing education)

   We present a psychometric evaluation of a revised version of the Cybersecurity Concept Inventory (CCI), completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned. 

   more » « less
4. Psychometric Evaluation of the Cybersecurity Curriculum Assessment

   https://doi.org/10.1145/3545945.3569762  

   Herman, Geoffrey L. ; Huang, Shan ; Peterson, Peter A. ; Oliva, Linda ; Golaszewski, Enis ; Sherman, Alan T. ( March 2023 , SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education)

   We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students. 

   more » « less
5. The CATS Hackathon: Creating and refining test items for the cybersecurity concept inventories

   Sherman, A. T. ; Oliva, L. ; Golaszewski, E. ; Phatak, D. ; Scheponik, T. ; Herman, G. ; Choi, D. S. ; Offenberger, S. ; Peterson, P. A. ; Dykstra, J. ; et al ( January 2019 , IEEE security & privacy)

   For two days in February 2018, 17 cybersecurity ed- ucators and professionals from government and in- dustry met in a “hackathon” to refine existing draft multiple-choice test items, and to create new ones, for a Cybersecurity Concept Inventory (CCI) and Cyber- security Curriculum Assessment (CCA) being devel- oped as part of the Cybersecurity Assessment Tools (CATS) Project. We report on the results of the CATS Hackathon, discussing the methods we used to develop test items, highlighting the evolution of a sample test item through this process, and offer- ing suggestions to others who may wish to organize similar hackathons. Each test item embodies a scenario, question stem, and five answer choices. During the Hackathon, par- ticipants organized into teams to (1) Generate new scenarios and question stems, (2) Extend CCI items into CCA items, and generate new answer choices for new scenarios and stems, and (3) Review and refine draft CCA test items. The CATS Project provides rigorous evidence- based instruments for assessing and evaluating educa- tional practices; these instruments can help identify pedagogies and content that are effective in teach- ing cybersecurity. The CCI measures how well stu- dents understand basic concepts in cybersecurity— especially adversarial thinking—after a first course in the field. The CCA measures how well students understand core concepts after completing a full cy- bersecurity curriculum. 

   more » « less