Title: Understanding and Measuring Incremental Development in CS1
Incremental development is the process of writing a small snippet of code and testing it before moving on. For students in introductory programming courses, the value of incremental development is especially higher as they may suffer from more syntax errors, lack the proficiency to address complicated bugs, and may be more prone to frustration when struggling to correct code. However, to evaluate the effectiveness of interventions that aim to teach programming processes such as incremental development, we need to develop measures to assess such processes. In this paper, we present a way to measure incremental development. By qualitatively analyzing 15 student coding interviews, we identified common behaviors in the programming process that relate to incremental development. We then leveraged a dataset of over 1000 development sessions -- about 52,000 code snapshots at compilation time -- to automatically detect the common behaviors identified in our qualitative analysis. Finally, we crafted a formal metric, called the ``Measure of Incremental Development’' (MID), to quantify how effectively a student used incremental development during a programming session. The MID detects common non-incremental development patterns such as excessive debugging after large additions of code to automatically assess a sequence of snapshots. The MID aligns with human evaluations of incrementality with over 80% accuracy. Our metric enables new research directions and interventions focused on improving students' development practices. more »« less
Fowler, Megan; Kraemer, Eileen; Sitaraman, Murali; Hollingsworth, Joseph E.
(, ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education)
null
(Ed.)
To develop code that meets its specification and is verifiably correct, such as in a software engineering course, students must be able to understand formal contracts and annotate their code with assertions such as loop invariants. To assist in developing suitable instructor and automated tool interventions, this research aims to go beyond simple pre- and post-conditions and gain insight into student learning of loop invariants involving objects. As students develop suitable loop invariants for given code with the aid of an online system backed by a verification engine, each student attempt, either correct or incorrect, was collected and analyzed automatically, and catalogued using an iterative process to capture common difficulties. Students were also asked to explain their thought process in arriving at their answer for each submission. The collected explanations were analyzed manually and found to be useful to assess their level of understanding as well as to extract actionable information for instructors and automated tutoring systems. Qualitative conclusions include the impact of the medium.
Shah, Anshul; Rexin, Thomas; Alhumrani, Fatimah; Griswold, William G; Porter, Leo; Soosai_Raj, Gerald
(, ACM Transactions on Computing Education)
Objectives The traditional, instructor-led form of live coding has been extensively studied, with findings showing that this form of live coding imparts similar learning to static-code examples. However, a concern with Traditional Live Coding is that it can turn into a passive learning activity for students as they simply observe the instructor program. Therefore, this study compares Active Live Coding—a form of live coding that leverages in-class coding activities and peer discussion—to Traditional Live Coding on three outcomes: 1) students’ adherence to effective programming processes, 2) students’ performance on exams and in-lecture questions, and 3) students’ lecture experience. Participants Roughly 530 students were enrolled in an advanced, CS1 course taught in Java at a large, public university in North America. The students were primarily first- and second-year undergraduate students with some prior programming experience. The student population was spread across two lecture sections—348 students in the Active Live Coding (ALC) lecture and 185 students in the Traditional Live Coding (TLC) lecture. Study Methods We used a mixed-methods approach to answer our‘ research questions. To compare students’ programming processes, we applied process-oriented metrics related to incremental development and error frequencies. To measure students’ learning outcomes, we compared students’ performance on major course components and used pre- and post-lecture questionnaires to compare students’ learning gain during lectures. Finally, to understand students’ lecture experience, we used a classroom observation protocol to measure and compare students’ behavioral engagement during the two lectures. We also inductively coded open-ended survey questions to understand students’ perceptions of live coding. Findings We did not find a statistically significant effect of ALC on students’ programming processes or learning outcomes. It seems that both ALC and TLC impart similar programming processes and result in similar student learning. However, our findings related to students’ lecture experience shows a persistent engagement effect of ALC, where students’ behavioral engagement peaks andremains elevatedafter the in-class coding activity and peer discussion. Finally, we discuss the unique affordances and drawbacks of the lecture technique as well as students’ perceptions of ALC. Conclusions Despite being motivated by well-established learning theories, Active Live Coding did not result in improved student learning or programming processes. This study is preceded by several prior works that showed that Traditional Live Coding imparts similar student learning and programming skills as static-code examples. Though potential reasons for the lack of observed learning benefits are discussed in this work, multiple future analyses to further investigate Active Live Coding may help the community understand the impacts (or lack thereof) of the instructional technique.
Long, Xingyu; Techapalokul, Peeratham; Tilevich, Eli
(, Proceedings of ACM SIGPLAN SPLASH Educational Symposium 2021 (SPLASH-E 2021))
null
(Ed.)
As Scratch has become one of the most popular educational programming languages, understanding its common programming idioms can benefit both computing educators and learners. This understanding can fine-tune the curricular development to help learners master the fundamentals of writing idiomatic code in their programming pursuits. Unfortunately, the research community’s understanding of what constitutes idiomatic Scratch code has been limited. To help bridge this knowledge gap, we systematically identified idioms as based on canonical source code, presented in widely available educational materials. We implemented a tool that automatically detects these idioms to assess their prevalence within a large dataset of over 70K Scratch projects in different experience backgrounds and project categories. Since communal learning and the practice of remixing are one of the cornerstones of the Scratch programming community, we studied the relationship between common programming idioms and remixes. Having analyzed the original projects and their remixes, we observed that different idioms may associate with dissimilar types of code changes. Code changes in remixes are desirable, as they require a meaningful programming effort that spurs the learning process. The ability to substantially change a project in its remixes hinges on the project’s code being easy to understand and modify. Our findings suggest that the presence of certain common idioms can indeed positively impact the degree of code changes in remixes. Our findings can help form a foundation of what comprises common Scratch programming idioms, thus benefiting both introductory computing education and Scratch programming tools.
Hundhausen, Christopher; Carter, Adam; Conrad, Phillip; Tariq, Ahsun; Adesope, Olusola
(, SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education)
null
(Ed.)
Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.
Wang, Wengran; Zhang, Chenhao; Stahlbauer, Andreas; Fraser, Gordon; Price, Thomas W.
(, ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education)
null
(Ed.)
Programming environments such as Snap, Scratch, and Processing engage learners by allowing them to create programming artifacts such as apps and games, with visual and interactive output. Learning programming with such a media-focused context has been shown to increase retention and success rate. However, assessing these visual, interactive projects requires time and laborious manual effort, and it is therefore difficult to offer automated or real-time feedback to students as they work. In this paper, we introduce SnapCheck, a dynamic testing framework for Snap that enables instructors to author test cases with Condition-Action templates. The goal of SnapCheck is to allow instructors or researchers to author property-based test cases that can automatically assess students' interactive programs with high accuracy. Our evaluation of SnapCheck on 162 code snapshots from a Pong game assignment in an introductory programming course shows that our automated testing framework achieves at least 98% accuracy over all rubric items, showing potentials to use SnapCheck for auto-grading and providing formative feedback to students.
Shah, Anshul, Granado, Michael, Sharma, Mrinal, Driscoll, John, Porter, Leo, Griswold, William G., and Soosai Raj, Adalbert Gerald.
"Understanding and Measuring Incremental Development in CS1". ACM Technical Symposium on Computer Science Education (SIGCSE) (). Country unknown/Code not available. https://par.nsf.gov/biblio/10398969.
@article{osti_10398969,
place = {Country unknown/Code not available},
title = {Understanding and Measuring Incremental Development in CS1},
url = {https://par.nsf.gov/biblio/10398969},
abstractNote = {Incremental development is the process of writing a small snippet of code and testing it before moving on. For students in introductory programming courses, the value of incremental development is especially higher as they may suffer from more syntax errors, lack the proficiency to address complicated bugs, and may be more prone to frustration when struggling to correct code. However, to evaluate the effectiveness of interventions that aim to teach programming processes such as incremental development, we need to develop measures to assess such processes. In this paper, we present a way to measure incremental development. By qualitatively analyzing 15 student coding interviews, we identified common behaviors in the programming process that relate to incremental development. We then leveraged a dataset of over 1000 development sessions -- about 52,000 code snapshots at compilation time -- to automatically detect the common behaviors identified in our qualitative analysis. Finally, we crafted a formal metric, called the ``Measure of Incremental Development’' (MID), to quantify how effectively a student used incremental development during a programming session. The MID detects common non-incremental development patterns such as excessive debugging after large additions of code to automatically assess a sequence of snapshots. The MID aligns with human evaluations of incrementality with over 80% accuracy. Our metric enables new research directions and interventions focused on improving students' development practices.},
journal = {ACM Technical Symposium on Computer Science Education (SIGCSE)},
author = {Shah, Anshul and Granado, Michael and Sharma, Mrinal and Driscoll, John and Porter, Leo and Griswold, William G. and Soosai Raj, Adalbert Gerald},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
An official website of the United States government
