We present a psychometric evaluation of a revised version of the Cybersecurity Concept Inventory (CCI) , completed by 354 students from 29 colleges and universities. The CCI is a conceptual test of understanding created to enable research on instruction quality in cybersecurity education. This work extends previous expert review and small-scale pilot testing of the CCI. Results show that the CCI aligns with a curriculum many instructors expect from an introductory cybersecurity course, and that it is a valid and reliable tool for assessing what conceptual cybersecurity knowledge students learned.
more »
« less
Psychometric Evaluation of the Cybersecurity Concept Inventorym
We present a psychometric evaluation of a revised version of
the Cybersecurity Concept Inventory (CCI),
completed by 354 students from 29 colleges and universities.
The CCI is a conceptual test of understanding created to enable research on
instruction quality in cybersecurity education.
This work extends previous expert review and small-scale pilot testing of the CCI.
Results show that the CCI aligns with a curriculum many instructors
expect from an introductory cybersecurity course, and that it is a valid and reliable
tool for assessing what conceptual cybersecurity knowledge students learned.
more »
« less
- NSF-PAR ID:
- 10290864
- Date Published:
- Journal Name:
- The ACM transactions on computing education
- ISSN:
- 2308-2003
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We present a psychometric evaluation of the Cybersecurity Curriculum Assessment (CCA), completed by 193 students from seven colleges and universities. The CCA builds on our prior work developing and validating a Cybersecurity Concept Inventory (CCI), which measures students' conceptual understanding of cybersecurity after a first course in the area. The CCA deepens the conceptual complexity and technical depth expectations, assessing conceptual knowledge of students who had completed multiple courses in cybersecurity. We review our development of the CCA and present our evaluation of the instrument using Classical Test Theory and Item-Response Theory. The CCA is a difficult assessment, providing reliable measurements of student knowledge and deeper information about high-performing students.more » « less
-
We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.more » « less
-
We analyze expert review and student performance data to evaluate the validity of the cci for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the cci, and 142 students from six different institutions took the cci as a pilot test. The panel reviewed each item of the cci and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the cci to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the cci. This evaluation showed that the cci is sufficiently reliable for measuring student knowledge of cybersecurity and that the cci may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the cci.more » « less
-
We report on the status of our Cybersecurity Assess- ment Tools (CATS) project that is creating and val- idating a concept inventory for cybersecurity, which assesses the quality of instruction of any first course in cybersecurity. In fall 2014, we carried out a Del- phi process that identified core concepts of cyber- security. In spring 2016, we interviewed twenty-six students to uncover their understandings and mis- conceptions about these concepts. In fall 2016, we generated our first assessment tool–a draft Cyberse- curity Concept Inventory (CCI), comprising approx- imately thirty multiple-choice questions. Each ques- tion targets a concept; incorrect answers are based on observed misconceptions from the interviews. This year we are validating the draft CCI using cognitive interviews, expert reviews, and psychometric testing. In this paper, we highlight our progress to date in developing the CCI. The CATS project provides infrastructure for a rig- orous evidence-based improvement of cybersecurity education. The CCI permits comparisons of different instructional methods by assessing how well students learned the core concepts of the field (especially ad- versarial thinking), where instructional methods re- fer to how material is taught (e.g., lab-based, case- studies, collaborative, competitions, gaming). Specif- ically, the CCI is a tool that will enable researchers to scientifically quantify and measure the effect of their approaches to, and interventions in, cybersecurity ed- ucation.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
