An official website of the United States government
We report on the status of our Cybersecurity Assess- ment Tools (CATS) project that is creating and val- idating a concept inventory for cybersecurity, which assesses the quality of instruction of any first course in cybersecurity. In fall 2014, we carried out a Del- phi process that identified core concepts of cyber- security. In spring 2016, we interviewed twenty-six students to uncover their understandings and mis- conceptions about these concepts. In fall 2016, we generated our first assessment tool–a draft Cyberse- curity Concept Inventory (CCI), comprising approx- imately thirty multiple-choice questions. Each ques- tion targets a concept; incorrect answers are based on observed misconceptions from the interviews. This year we are validating the draft CCI using cognitive interviews, expert reviews, and psychometric testing. In this paper, we highlight our progress to date in developing the CCI. The CATS project provides infrastructure for a rig- orous evidence-based improvement of cybersecurity education. The CCI permits comparisons of different instructional methods by assessing how well students learned the core concepts of the field (especially ad- versarial thinking), where instructional methods re- fer to how material is taught (e.g., lab-based, case- studies, collaborative, competitions, gaming). Specif- ically, the CCI is a tool that will enable researchers to scientifically quantify and measure the effect of their approaches to, and interventions in, cybersecurity ed- ucation.
more »
« less
Work in Progress: The Electric Circuit Concepts Diagnostic (ECCD)
Students come to learning in engineering classrooms with misconceptions about the concepts covered in engineering course contents. However, instructional efforts often do not effectively address misconceptions in students’ prior knowledge. Concept inventories (CIs) are often relied upon to identify misconceptions in students’ prior knowledge. However, many instructors never benefit much from using CIs because they lack either the know-how, time commitment, or statistical skills required to use them efficiently and effectively. Furthermore, there sometimes are ambiguities about how to interpret students’ CI scores. The Electric Circuit Concepts Diagnostic (ECCD) project team will address these limitations of CIs by creating a web-based electric circuit concept inventory that: (i.) provides an immediate and multipurpose feedback system for reporting about students’ circuits and electricity prior knowledge; (ii.) differentiates, with a high probability, between a lack of prior knowledge and misconceptions; and (iii.) uses a scheme of multidimensional knowledge profiles to report on students’ prior knowledge and misconceptions. The project will integrate the affordances of cognitive diagnostic modeling (CDM), multitier testing frameworks, and computer-assisted testing to realize these project objectives. This work-in-progress report introduces the objectives of ECCD inventory to the ECE research and teaching community.
more »
« less
- Award ID(s):
- 2021468
- PAR ID:
- 10348376
- Date Published:
- Journal Name:
- Annual Proceeding of the American Society for Engineering Education
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Web-browsing histories, online newspapers, streaming music, and stock prices all show that we live in an age of data. Extracting meaning from data is necessary in many fields to comprehend the information flow. This need has fueled rapid growth in data science education aiming to serve the next generation of policy makers, data science researchers, and global citizens. Initially, teaching practices have been drawn from data science's parent disciplines (e.g., computer science and mathematics). This project addresses the early stages of developing a concept inventory of student difficulty within the newly emerging field of data science. In particular this project will address three primary research objectives: (1) identify student misconceptions in data science courses; (2) document students’ prior knowledge and identify courses that teach early data science concepts; and (3) confirm expert identification of data science concepts, and their importance for introductory-level data science curricula. During the first year of this grant, we have collected approximately 200 responses for a survey to confirm concepts from an existing body of knowledge presented by the Edison Project. Survey respondents are comprised of faculty and industry practitioners within data science and closely related fields. Preliminary analysis of these results will be presented with respect to our third research objective. In addition, we developed and launched a pilot assessment for identifying student difficulties within data science courses. The protocol includes regular responses to reflective questions by faculty, teaching assistants, and students from selected data science courses offered at the three participating institutions. Preliminary analyses will be presented along with implications for future data collection in year two of the project. In addition to the anticipated results, we expect that the data collection and analysis methodologies will be of interest to many scholars who have or will engage in discipline-based educational research.more » « less
-
Concept inventories are standardized assessments that evaluate student understanding of key concepts within academic disciplines. While prevalent across STEM fields, their development lags for advanced computer science topics like dynamic programming (DP)---an algorithmic technique that poses significant conceptual challenges for undergraduates. To fill this gap, we developed and validated a Dynamic Programming Concept Inventory (DPCI). We detail the iterative process used to formulate multiple-choice questions targeting known student misconceptions about DP concepts identified through prior research studies. We discuss key decisions, tradeoffs, and challenges faced in crafting probing questions to subtly reveal these conceptual misunderstandings. We conducted a preliminary psychometric validation by administering the DPCI to 172 undergraduate CS students finding our questions to be of appropriate difficulty and effectively discriminating between differing levels of student understanding. Taken together, our validated DPCI will enable instructors to accurately assess student mastery of DP. Moreover, our approach for devising a concept inventory for an advanced theoretical computer science concept can guide future efforts to create assessments for other under-evaluated areas currently lacking coverage.more » « less
-
Security failures in software arising from failures to practice secure programming are commonplace. Improving this situation requires that practitioners have a clear understanding of the foundational concepts in secure programming to serve as a basis for building new knowledge and responding to new challenges. We developed a Secure Programing Concept Inventory (SPCI) to measure students' understanding of foundational concepts in secure programming. The SPCI consists of thirty-five multiple choice items targeting ten concept areas of secure programming. The SPCI was developed by establishing the content domain of secure programming, developing a pool of test items, multiple rounds of testing and refining the items, and finally testing and inventory reduction to produce the final scale. Scale development began by identifying the core concepts in secure programming. A Delphi study was conducted with thirty practitioners from industry, academia, and government to establish the foundational concepts of secure programming and develop a concept map. To build a set of misconceptions in secure programming, the researchers conducted interviews with students and instructors in the field. These interviews were analyzed using content analysis. This resulted in a taxonomy of misconceptions in secure programming covering ten concept areas. An item pool of multiple-choice questions was developed. The item pool of 225 was administered to a population of 690 students across four institutions. Item discrimination and item difficulty scores were calculated, and the best performing items were mapped to the misconception categories to create subscales for each concept area resulting in a validated 35 item scale.more » « less
-
Eddy, Sarah L (Ed.)Racial biases, which harm marginalized and excluded communities, may be combatted by clarifying misconceptions about race during biology lessons. We developed a human genetics laboratory activity that challenges the misconception that race is biological (biological essentialism). We assessed the relationship between this activity and student outcomes using a survey of students’ attitudes about biological essentialism and color-evasive ideology and a concept inventory about phylogeny and human diversity. Students in the human genetics laboratory activity showed a significant decrease in their acceptance of biological essentialism compared with a control group, but did not show changes in color-evasive ideology. Students in both groups exhibited increased knowledge in both areas of the concept inventory, but the gains were larger in the human genetics laboratory. In the second iteration of this activity, we found that only white students’ decreases in biological essentialist beliefs were significant and the activity failed to decrease color-evasive ideologies for all students. Concept inventory gains were similar and significant for both white and non-white students in this iteration. Our findings underscore the effectiveness of addressing misconceptions about the biological origins of race and encourage more research on ways to effectively change damaging student attitudes about race in undergraduate genetics education.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
