An official website of the United States government
Abstract: How well do code-writing tasks measure students’ knowledge of programming patterns and anti-patterns? How can we assess this knowledge more accurately? To explore these questions, we surveyed 328 intermediate CS students and measured their performance on different types of tasks, including writing code, editing someone else’s code, and, if applicable, revising their own alternatively-structured code. Our tasks targeted returning a Boolean expression and using unique code within an if and else.We found that code writing sometimes under-estimated student knowledge. For tasks targeting returning a Boolean expression, over 55% of students who initially wrote with non-expert structure successfully revised to expert structure when prompted - even though the prompt did not include guidance on how to improve their code. Further, over 25% of students who initially wrote non-expert code could properly edit someone else’s non-expert code to expert structure. These results show that non-expert code is not a reliable indicator of deep misconceptions about the structure of expert code. Finally, although code writing is correlated with code editing, the relationship is weak: a model with code writing as the sole predictor of code editing explains less than 15% of the variance. Model accuracy improves when we include additional predictors that reflect other facets of knowledge, namely the identification of expert code and selection of expert code as more readable than non-expert code. Together, these results indicate that a combination of code writing, revising, editing, and identification tasks can provide a more accurate assessment of student knowledge of programming patterns than code writing alone.
more »
« less
Readable vs. Writable Code: A Survey of Intermediate Students' Structure Choices
Since intermediate CS students can use a variety of control structures, why do their choices often not match experts' Students may not realize what choices expert prefer, find non-expert choices easier to read, or simply forget to write with expert structure. To disentangle these explanations, we surveyed 328 2nd and 3rd semester undergraduates, with tasks including writing short functions, selecting which structure was most readable or best styled, and comprehension questions. Questions focused on seven control structure topics that were important to instructors (e.g., factoring out repeated code between an if-block and its else). Students frequently wrote with non-expert structure, and, for five topics, at least 1/3 of students (48% - 71%) thought a non-expert structure was more readable than the expert one. However, students often made one choice when writing code, but preferred a different choice when reading it. Additionally, for more complex topics, students often failed to notice (or understand) differences in execution caused by changes in structure. Together, these results suggest that instruction and practice for choosing control structures should be context-specific, and that assessment focused only on code writing may miss underlying misunderstandings.
more »
« less
- Award ID(s):
- 1948519
- PAR ID:
- 10325821
- Date Published:
- Journal Name:
- SIGCSE 2022: Proceedings of the 53rd ACM Technical Symposium on Computer Science Education
- Volume:
- 1
- Page Range / eLocation ID:
- 321 to 327
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Experts often use particular control flow structures to make their code easier to read and modify, such as using the logical operator AND to conjoin conditions rather than nesting separate if statements. Within Boolean expressions, experts take advantage of short-circuit evaluation by ordering their conditions to avoid errors (such as checking that an index is within the bounds of an array before examining the value at that index). How well do students understand these structures? We investigate students' use and understanding of conjoined versus separate conditions within a larger assessment of 125 undergraduate students at the end of their second- and third-semester CS courses (in algorithms & data structures and introductory software engineering). The assessment asked students to: write code where an edge case error could be avoided with short-circuit evaluation, revise their code with nudges towards expert structure, and answer comprehension questions involving code tracing. When writing, students frequently forgot to check for a key edge case. When that case was included, the check was often separated in its own if-statement rather than conjoined with the other conditions. This could indicate a stylistic choice or a belief that the check had to be separated for functionality. Notably, students who included all necessary conditions rarely exhibited the error of ordering them incorrectly. However, with code comprehension, students demonstrated significant misunderstandings about the effects of condition ordering. Students were more accurate on comprehension tasks with nested ifs than conjoined conditions, and this effect was most pronounced when the ordering of the conditions would lead to errors. When conditions were conjoined in a single expression, only 35% of students recognized that checking a value at an index before checking that the index was in bounds would lead to an error. However, 54% of students recognized the problem when the conditions were separated into individual if-statements. This demonstrates a subtlety in code execution that intermediate students may not have mastered and emphasizes the challenges in assessing students' understanding solely via the way they write code.more » « less
-
How can we teach students to use more readable code structures? How common is it for students to choose less readable (but still functional) alternatives? We explore these questions for a specific anti-pattern: using sequential if statements when conditions are exclusive (rather than using else-if or else). We created and validated an automated detector to identify this anti-pattern in student's code. Running the detector on 1,764 homework submissions (from 270 students in a CS2 class on data structures and algorithms) showed that this anti-pattern was common and varied by assignment: across 12 assignments, 3% to 50% of submissions used sequential ifs for exclusive cases. However, using this anti-pattern did not preclude using else-ifs: across assignments, up to 34% of the submissions used both forms. Further, students used sequential if statements in surprising ways, such as checking a condition and then the negation of that condition, indicating a more novice level of understanding than expected for an intermediate course. Hand-inspection of the detector-flagged cases suggests that sequential ifs for exclusive cases may be a code smell that can indicate larger problems with logic and abstraction.more » « less
-
This investigation is the first of four investigations funded by the NSF (DUE award 2215807) to develop and then field test on open educational browser-based writing-to-learn tool called GIKS. The underlying theory is that writing-to-learn with immediate formative feedback presented as concept networks is engaging and effective for learning concepts covered in lectures. This work was studied in a second year architectural engineering course focusing on building materials, processes and modeling. Participants (n=84) completed a lesson (readings, lecture, and labs) then followed by writing prompts centered on the following topics: Building with Concrete and Wood Construction (3 weeks later). Participants were assigned to one of two counterbalanced groups, group A used GIKS software to write a 300-word summary of the first lesson but did not write in the second lesson, while group B did not write in the first lesson but used GIKS in the second lesson, so that each group served as a control treatment for the other group. All students completed a concept structure survey at the end of each lesson that contained 20 key concepts from that lesson, the two concept structure surveys’ data were transformed into concept networks and then these networks were compared to an expert network benchmark referent, as well as to networks of the textbook chapter and the PowerPoint slides of the related lesson. Then a week after the second lesson students completed the standing end-of-module multiple-choice posttest that included items from these lesson as well as from other lessons in the module. Results to date highlight that for both lessons, the group using GIKS scored higher on the concept structure survey (more like the expert network) BUT lower on the multiple-choice test, the difference was significant for the Building with Concrete lesson (p < .05) but not for the Wood Construction lesson. This interaction has been reported before by Ntshalintshali & Clariana (2020), that improving conceptual knowledge sometimes decreases memory of lesson details. Descriptive analysis of the group-average networks derived from the concept structure surveys for Building with Concrete show that the group-averaged network of those using GIKS compared to the control was more like the expert network (54% vs. 36%), the network of the textbook Chapter (32% vs. 29%), the network of the PowerPoint (PP) (46% vs. 41%), and especially like peers in the other group (67%). For Wood Construction the difference between the groups was less, the group-averaged network of those using GIKS compared to the control was more like the expert (40% vs. 39%), like the light-framed construction PP (28% vs. 24%), and especially like peers in the other group (72%). These findings show that writing-to-learn with GIKS with immediate network feedback improves conceptual knowledge as expected but at the cost of details. Peers conceptual structure of the lesson materials were very similar (peer-peer mental model convergence) and were more like others than like the expert, or the book chapters, or the PowerPoint slides; in addition, the PowerPoint slides appear to influence conceptual structure more than the textbook chapters. Investigation 2 will consider writing-to-learn with or without immediate network feedback in order to isolate the effects of immediate network feedback.more » « less
-
Engineering undergraduates’ academic writing experiences prior to entry-level engineering lab courses can be classified into three different groups: a group with both rhetorically-focused writing (e.g., first-year-composition) and technical writing courses; a group with only rhetorically-focused writing courses; and a group with no rhetorically-focused writing or technical writing courses. Using a lens of transfer theories that explain how much knowledge from one context is used or adapted in new contexts, these three groups can be called concurrent, vertical, and absent transfer groups respectively. This study, which is part of a larger project developing and implementing writing-focused modules in engineering labs, aims to investigate undergraduates’ perspectives on readiness, writing transfer, and effectiveness of writing instructions in engineering lab report writing through a student survey. End-of-term online surveys (n = 40) of undergraduates in entry-level engineering lab courses were collected from three distinctive universities: an urban, commuter, public research university; an urban, private, teaching-focused university; and a rural, public, teaching-focused university. The survey questions have three parts: 1) student perspectives in writing in engineering disciplines; 2) how students use prior writing knowledge when writing lab reports in engineering lab courses; and 3) how engineering lab course writing instructions impact students’ engineering lab report writing. Findings suggest that the three transfer groups present statistical distinctions on the readiness of writing engineering lab reports (concurrent group as the highest and absent group as the lowest). The three groups also show different perspectives on how their freshmen writing courses contributed their engineering lab report writing. The concurrent transfer group believed freshmen writing instruction regarding “focus on purpose” contributed most when they write lab reports, while the greatest number of vertical transfer group students mentioned “knowledge about format and structure” was most helpful. Many absent transfer students valued “identifying problems or questions” instructed from their freshmen writing-intensive philosophy course as the content they used most when writing lab reports. Ultimately, the analysis of the data suggested that despite their perceived preparedness for writing lab reports, most of the students felt their skills improved as a result of engaging in lab report writing activities.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3478431.3499413
