skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: An Item Response Theory Evaluation of a Language-Independent CS1 Knowledge Assessment
Tests serve an important role in computing education, measuring achievement and differentiating between learners with varying knowledge. But tests may have flaws that confuse learners or may be too difficult or easy, making test scores less valid and reliable. We analyzed the Second Computer Science 1 (SCS1) concept inventory, a widely used assessment of introductory computer science (CS1) knowledge, for such flaws. The prior validation study of the SCS1 used Classical Test Theory and was unable to determine whether differences in scores were a result of question properties or learner knowledge. We extended this validation by modeling question difficulty and learner knowledge separately with Item Response Theory (IRT) and performing expert review on problematic questions. We found that three questions measured knowledge that was unrelated to the rest of the SCS1, and four questions were too difficult for our sample of 489 undergrads from two universities.  more » « less
Award ID(s):
1735123
PAR ID:
10107745
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
ACM Technical Symposium on Computer Science Education
Page Range / eLocation ID:
699 to 705
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Langran, E. (Ed.)
    This work discusses ongoing experiences in creating interdisciplinary solutions to the development of cybersecurity/computer science skills for K-12 teachers. Participants for this work were K-12 teachers; the work was conducted during the summer semester of 2020. Nine K-12 teachers enrolled in the pilot course, and five K-12 teachers completed modules 0 & 1 of the cybersecurity course. This work describes a month long, self-paced cybersecurity micro-credential which consisted of an introduction to cybersecurity and the CIA Triad (Confidentiality, Integrity, and Accessibility). The primary research question explores the extent to which the cybersecurity micro-credential was considered helpful in understanding cybersecurity concepts and principles. The authors utilized quantitative and qualitative data collection via four methods: 1) content knowledge quiz (quantitative); 2) focus group (qualitative); 3) five semi-structured interviews (qualitative) and 4) two bi-weekly progress reports (qualitative). Qualitatively, focus group, semi-structured interview questions, and bi-weekly progress reports were coded and grouped into major themes by searching descriptive words. Quantitatively, a moderate positive relationship was found between post-test scores and total time in hours spent on modules 0 & 1. The main limitation is the sample size (n=5), which is too small to produce generalizable results. The work showcases innovative tools for teaching cybersecurity. 
    more » « less
  2. Cognitive tests have been long used as a measure of student knowledge, ability, and as a predictor for success in engineering and computer science. However, these tests are not without their own problems relating to priming, difficulty (resulting in test fatigue) and time on exam. This paper discusses efforts to modify Parker et al.'s Second CS1 aptitude test (SCS1) \citeParker16 to reduce the time spent on the exam, provide greater customization to match concepts taught across three universities, and reduce redundancy of test questions all while maintaining the instrument's reliability. This instrument was modified for use on an ongoing grant investigating whether spatial abilities impact the success of students in introductory CS courses. The instrument developed in this paper is a revised shortened version of Second Computer Science 1 (SCS1) aptitude test, designated as SCS1R. 
    more » « less
  3. Abstract This study contributes to a growing body of scholarship at the intersection of bilingual education and education policy and examines reclassification, or the transition out of formal English language services in schools, as one potential lever in accelerating or decelerating multilingual learners’ science learning. More specifically, it traces multilingual learners’ science academic achievement vis‐à‐vis science test scores over a six‐year period using the nationally‐representative Early Childhood Longitudinal Study of 2010–11 (ECLS‐K:2011) data set. We use regression analyses with panel data to explore the relationship of reclassification with MLs’ science achievement at a national scale, and then, how variation in contextual factors (including family, school, and individual characteristics) shapes this relationship. Results show that, after controlling for covariates and prior test scores, reclassification is not significantly associated with differential science test scores when compared to students that retain their EL status. Results further show that reclassification is associated with higher science achievement for MLs who were previously in a dual‐language program but lower scores for those with higher prior achievement. We conclude with implications for the reclassification process, as well as directions for future research on reclassification, multilingual learners, and academic achievement. 
    more » « less
  4. Abstract Traditional tests of concept knowledge generate scores to assess how well a learner understands a concept. Here, we investigated whether patterns of brain activity collected during a concept knowledge task could be used to compute a neural ‘score’ to complement traditional scores of an individual’s conceptual understanding. Using a novel data-driven multivariate neuroimaging approach—informational network analysis—we successfully derived a neural score from patterns of activity across the brain that predicted individual differences in multiple concept knowledge tasks in the physics and engineering domain. These tasks include an fMRI paradigm, as well as two other previously validated concept inventories. The informational network score outperformed alternative neural scores computed using data-driven neuroimaging methods, including multivariate representational similarity analysis. This technique could be applied to quantify concept knowledge in a wide range of domains, including classroom-based education research, machine learning, and other areas of cognitive science. 
    more » « less
  5. As more people turn to discretionary online tools to learn new skills such as computer programming, exploring how to better support a wide range of learners is becoming increasingly essential to train the next generation of highly skilled technology workers. In our prior work, users with high learner autonomy complained that most online resources they used to learn more programming did not provide them with the flexibility they preferred to navigate through learning materials, locking them into a set sequence of topics/concepts. To explore this, we implemented a level-jumping feature into an online educational programming game. We tested it with 350 new users, tracking their progress through the game for 7 days each. We found that those with high learner autonomy did use the level jumping feature more than those with low learner autonomy. We also found that males were more likely to use this new feature, regardless of learner autonomy level, compared to their female counterparts. Finally, we found that those with low learner autonomy ultimately completed more levels than their high autonomy counterparts, and that this was particularly true of female learners (who completed the most levels overall). Based on these findings, we believe that autonomous-supportive features such as flexible navigation may be beneficial to all users of online educational tools, and that encouraging its use by a wider group of users (particularly females), may increase positive effects. 
    more » « less