An official website of the United States government
We report on the status of our Cybersecurity Assess- ment Tools (CATS) project that is creating and val- idating a concept inventory for cybersecurity, which assesses the quality of instruction of any first course in cybersecurity. In fall 2014, we carried out a Del- phi process that identified core concepts of cyber- security. In spring 2016, we interviewed twenty-six students to uncover their understandings and mis- conceptions about these concepts. In fall 2016, we generated our first assessment tool–a draft Cyberse- curity Concept Inventory (CCI), comprising approx- imately thirty multiple-choice questions. Each ques- tion targets a concept; incorrect answers are based on observed misconceptions from the interviews. This year we are validating the draft CCI using cognitive interviews, expert reviews, and psychometric testing. In this paper, we highlight our progress to date in developing the CCI. The CATS project provides infrastructure for a rig- orous evidence-based improvement of cybersecurity education. The CCI permits comparisons of different instructional methods by assessing how well students learned the core concepts of the field (especially ad- versarial thinking), where instructional methods re- fer to how material is taught (e.g., lab-based, case- studies, collaborative, competitions, gaming). Specif- ically, the CCI is a tool that will enable researchers to scientifically quantify and measure the effect of their approaches to, and interventions in, cybersecurity ed- ucation.
more »
« less
Validation of a Secure Programming Concept Inventory
Security failures in software arising from failures to practice secure programming are commonplace. Improving this situation requires that practitioners have a clear understanding of the foundational concepts in secure programming to serve as a basis for building new knowledge and responding to new challenges. We developed a Secure Programing Concept Inventory (SPCI) to measure students' understanding of foundational concepts in secure programming. The SPCI consists of thirty-five multiple choice items targeting ten concept areas of secure programming. The SPCI was developed by establishing the content domain of secure programming, developing a pool of test items, multiple rounds of testing and refining the items, and finally testing and inventory reduction to produce the final scale. Scale development began by identifying the core concepts in secure programming. A Delphi study was conducted with thirty practitioners from industry, academia, and government to establish the foundational concepts of secure programming and develop a concept map. To build a set of misconceptions in secure programming, the researchers conducted interviews with students and instructors in the field. These interviews were analyzed using content analysis. This resulted in a taxonomy of misconceptions in secure programming covering ten concept areas. An item pool of multiple-choice questions was developed. The item pool of 225 was administered to a population of 690 students across four institutions. Item discrimination and item difficulty scores were calculated, and the best performing items were mapped to the misconception categories to create subscales for each concept area resulting in a validated 35 item scale.
more »
« less
- Award ID(s):
- 1934279
- PAR ID:
- 10467373
- Publisher / Repository:
- SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education; ACM Digital Library
- Date Published:
- Volume:
- 2
- ISBN:
- 978-1-4503-9433-8
- Page Range / eLocation ID:
- 1423
- Format(s):
- Medium: X
- Location:
- Toronto, ON, Canada
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Concept inventories are standardized assessments that evaluate student understanding of key concepts within academic disciplines. While prevalent across STEM fields, their development lags for advanced computer science topics like dynamic programming (DP)---an algorithmic technique that poses significant conceptual challenges for undergraduates. To fill this gap, we developed and validated a Dynamic Programming Concept Inventory (DPCI). We detail the iterative process used to formulate multiple-choice questions targeting known student misconceptions about DP concepts identified through prior research studies. We discuss key decisions, tradeoffs, and challenges faced in crafting probing questions to subtly reveal these conceptual misunderstandings. We conducted a preliminary psychometric validation by administering the DPCI to 172 undergraduate CS students finding our questions to be of appropriate difficulty and effectively discriminating between differing levels of student understanding. Taken together, our validated DPCI will enable instructors to accurately assess student mastery of DP. Moreover, our approach for devising a concept inventory for an advanced theoretical computer science concept can guide future efforts to create assessments for other under-evaluated areas currently lacking coverage.more » « less
-
Abstract BackgroundChi and colleagues have argued that some of the most challenging engineering concepts exhibit properties of emergent systems. However, students often lack a mental framework, or schema, for understanding emergence. Slotta and Chi posited that helping students develop a schema for emergent systems, referred to as schema training, would increase the understanding of challenging concepts exhibiting emergent properties. PurposeWe tested the effectiveness of schema training and explored the nature of challenging concepts from thermodynamics and heat transfer. We investigated if schema training could (a) repair misconceptions in advanced engineering students and (b) prevent them in beginning engineering students. MethodWe adapted Slotta and Chi's schema training modules and tested their impact in two studies that employed an experimental design. Items from the Thermal and Transport Concept Inventory and expert‐developed multiple‐choice questions were used to evaluate conceptual understanding of the participants. The language used by students in their open‐ended explanations of multiple‐choice questions was also coded. ResultsIn both studies, students in the experimental groups showed larger gains in their understanding of some concepts—specifically in dye diffusion and microfluidics in Study One, and in the final test for thermodynamics in Study Two. But in neither study did students exhibit any gain in conceptual questions about heat transfer. ConclusionOur studies suggest the importance of examining the nature of the phenomena underlying the concepts being taught because the language used in instruction has implications for how students understand them. Therefore, we suggest that instructors reflect on their own understanding of the concepts.more » « less
-
Parks, Samantha T (Ed.)ABSTRACT The Microbiology Concept Inventory is an assessment tool derived from the fundamental statements created by the American Society for Microbiology. This two-tier, multiple-choice question inventory requires students to choose the most correct answer for each question and provide a brief justification of their reasoning. Educators can utilize this tool to identify common misconceptions held by students and adjust curriculum to address and prevent the persistence of student misconceptions. Over the course of 5 years, the Microbiology Concept Inventory was annually administered to undergraduate students enrolled in entry-level, mid-level, and senior capstone microbiology courses at a mid-western rural university. Analysis was completed to compare course, year, majors and minors, gender, ethnicity, and cumulative GPA. Results of this study showed a significant difference in Microbiology Concept Inventory scores between students with high cumulative GPAs (3.5–4.0) and students with comparatively lower cumulative GPAs (2.5–2.99, 3.0–3.49). Results between the other demographic categories revealed statistically different scores in favor of white students, but no differences in scores between genders. The results suggest evidence of ethnic bias, but no gender bias as measured by the Microbiology Concept Inventory. Additionally, significant differences in scores across cohorts are indicative of improvements in the curricula due to prior targeted changes. Analysis of concept inventory results can guide curriculum changes for course instructors. Implementation of curriculum changes can enrich students’ academic success.more » « less
-
For two days in February 2018, 17 cybersecurity ed- ucators and professionals from government and in- dustry met in a “hackathon” to refine existing draft multiple-choice test items, and to create new ones, for a Cybersecurity Concept Inventory (CCI) and Cyber- security Curriculum Assessment (CCA) being devel- oped as part of the Cybersecurity Assessment Tools (CATS) Project. We report on the results of the CATS Hackathon, discussing the methods we used to develop test items, highlighting the evolution of a sample test item through this process, and offer- ing suggestions to others who may wish to organize similar hackathons. Each test item embodies a scenario, question stem, and five answer choices. During the Hackathon, par- ticipants organized into teams to (1) Generate new scenarios and question stems, (2) Extend CCI items into CCA items, and generate new answer choices for new scenarios and stems, and (3) Review and refine draft CCA test items. The CATS Project provides rigorous evidence- based instruments for assessing and evaluating educa- tional practices; these instruments can help identify pedagogies and content that are effective in teach- ing cybersecurity. The CCI measures how well stu- dents understand basic concepts in cybersecurity— especially adversarial thinking—after a first course in the field. The CCA measures how well students understand core concepts after completing a full cy- bersecurity curriculum.more » « less
