Journal ArticleThe CATS Hackathon: Creating and refining test items for the cybersecurity concept inventoriesSherman, A. T.; Oliva, L.; Golaszewski, E.; Phatak, D.; Scheponik, T.; Herman, G.; Choi, D. S.; Offenberger, S.; Peterson, P. A.; Dykstra, J.; Bard, G.; Chattopadhyay, A.; Sharevski, F.; Verma, R.; Vrecenar, R.For two days in February 2018, 17 cybersecurity ed- ucators and professionals from government and in- dustry met in a “hackathon” to refine existing draft multiple-choice test items, and to create new ones, for a Cybersecurity Concept Inventory (CCI) and Cyber- security Curriculum Assessment (CCA) being devel- oped as part of the Cybersecurity Assessment Tools (CATS) Project. We report on the results of the CATS Hackathon, discussing the methods we used to develop test items, highlighting the evolution of a sample test item through this process, and offer- ing suggestions to others who may wish to organize similar hackathons. Each test item embodies a scenario, question stem, and five answer choices. During the Hackathon, par- ticipants organized into teams to (1) Generate new scenarios and question stems, (2) Extend CCI items into CCA items, and generate new answer choices for new scenarios and stems, and (3) Review and refine draft CCA test items. The CATS Project provides rigorous evidence- based instruments for assessing and evaluating educa- tional practices; these instruments can help identify pedagogies and content that are effective in teach- ing cybersecurity. The CCI measures how well stu- dents understand basic concepts in cybersecurity— especially adversarial thinking—after a first course in the field. The CCA measures how well students understand core concepts after completing a full cy- bersecurity curriculum.2019-01-0110104875IEEE security & privacy1540-7993https://doi.org/1753681; 1819521National Science Foundation