Search for: All records

We conducted a study to see if using Bayesian Knowledge Tracing (BKT) models would save time and problems in programming tutors. We used legacy data collected by two programming tutors to compute BKT models for every concept covered by each tutor. The novelty of our model was that slip and guess parameters were computed for every problem presented by each tutor. Next, we used cross-validation to evaluate whether the resulting BKT model would have reduced the number of practice problems solved and time spent by the students represented in the legacy data. We found that in 64.23% of the concepts, students would have saved time with the BKT model. The savings varied among concepts. Overall, students would have saved a mean of 1.28 minutes and 1.23 problems per concept. We also found that BKT models were more effective at saving time and problems on harder concepts. 

We developed a tutor for imperative programming in C++. It covers algorithm formulation, program design and coding – all three stages involved in writing a program to solve a problem. The design of the tutor is epistemic, i.e., true to real-life programming practice. The student works through all the three stages of programming in interleaved fashion, and within the context of a single code canvas. The student has the sole agency to compose the program and write the code. The tutor uses goals and plans as prompts to scaffold the student through the programming process designed by an expert. It provides drill-down immediate feed-back at the abstract, concrete and bottom-out levels at each step. So, by the end of the session, the student is guaranteed to write the complete and correct program for a given problem. We used model-based architecture to implement the tutor be-cause of the ease with which it facilitates adding problems to the tutor. In a preliminary study, we found that practicing with the tutor helped students solve problems with fewer erroneous actions and less time. 

We studied long-term retention of the concepts that introductory programming students learned using two software tutors on tracing the behavior of functions and debugging functions. Whereas the concepts covered by the tutor on the behavior of functions were interdependent, the concepts covered by debugging tutor were independent. We analyzed the data of the students who had used the tutors more than once, hours to weeks apart. Our objective was to find whether students retained what they had learned during the first session till the second session. We found that the more the problems students solved during the first session, the greater the retention. Knowledge and retention varied between debugging and behavior tutors, even though they both dealt with functions, possibly because de-bugging tutor covered independent concepts whereas behavior tutor covered interdependent concepts. 

Parsons puzzles are jigsaw puzzles wherein students are given a program in scrambled order and tasked with reassembling the program in its correct order. Do students use program semantics when solving Parsons puzzles? The answer to this question has implications for the use of Parsons puzzles as a pedagogic tool. In order to answer this question, we considered semantics at the level of statements and control-flow. We analyzed the data collected by a Parsons puzzle tutor over 5 semesters and measured the extent to which students’ puzzle-solving behavior conformed with the use of statement-level and control-flow semantics. We found that students used statement-level semantics to assemble up to 73% of the lines in a puzzle and control-flow semantics to assemble up to 47% of the lines. They used statement-level semantics more than control-flow semantics and more on some puzzles than others. Whenever we found a significant difference between C++ and Java students, C++ students used semantics more than Java students. Finally, we did not find an increase in the use of semantics with in-creased practice. 

For decades computing educators have been trying to identify and predict at-risk students, particularly early in the first programming course. These efforts range from the analyzing demographic data that pre-exists undergraduate entrance to using instruments such as concept inventories, to the analysis of data arising during education. Such efforts have had varying degrees of success, have not seen widespread adoption, and have left room for improvement. We analyse results from a two-year study with several hundred students in the first year of programming, comprising majors and non-majors. We find evidence supporting a hypothesis that engagement with extra credit assessment provides an effective method of differentiating students who are not at risk from those who may be. Further, this method can be used to predict risk early in the semester, as any engagement – not necessarily completion – is enough to make this differentiation. Additionally, we show that this approach is not dependent on any one programming language. In fact, the extra credit opportunities need not even involve programming. Our results may be of interest to educators, as well as researchers who may want to replicate these results in other settings. 

In this study, we studied whether the number of revisions allowed per problem when error-flagging feedback is provided has a significant effect on learning. We used a partial cross-over study and analyzed the data collected by two adaptive tutors on while loops and for loops over six semesters. We found that when students were unfamiliar with the concepts, they solved fewer problems and therefore, learned significantly less when they were provided more opportunities for revision with error-flagging feedback. But, once they became more familiar with the concepts, allowing for more revisions had no deleterious effect on learning. 

Do students retain the programming concepts they have learned using software tutors over the long term? In order to answer this question, we analyzed the data collected by a software tutor on selection statements. We used the data of the students who used the tutor more than once to see whether they had retained for the second session what they had learned during the first session. We found that students retained over 71% of selection concepts that they had learned during the first session. The more problems students solved during the first session, the greater the percentage of retention. Even when students already knew a concept and did not benefit from using the tutor, a small percentage of concepts were for-gotten from the first session to the next, corresponding to transience of learning. Transience of learning varied with concepts. We list confounding factors of the study. 

We conducted a controlled study to investigate whether having students choose the concept on which to solve each practice problem in an adaptive tutor helped improve learning. We analyzed data from an adaptive tutor used by introductory programming students over three semesters. The tutor presented code-tracing problems, used pretest-practice-post-test protocol, and presented line-by-line explanation of the correct solution as feedback. We found that choice did not in-crease the amount of learning or pace of learning. But, it resulted in greater improvement in score on learned concepts, and the effect size was medium. 

Would providing choice lead to improved learning with a tutor? We had conducted and reported a controlled study earlier, wherein, introductory programing students were given the choice of skipping the line-by-line feedback provided after each incorrect answer in a tutor on if/if-else statements. Contrary to expectations, the study found that the choice to skip feedback did not lead to greater learning. We tried to reproduce these results using two tutors on if/if-else and switch statements, and with a larger subject pool. We found that whereas choice did not lead to greater learning on if/if-else tutor in this reproducibility study either, it resulted in decreased learning on switch tutor. We hypothesize that skipping feedback is indeed detrimental to learning. But, inter-relationships among the concepts covered by a tutor and the transfer of learning facilitated by these relationships compensate for the negative effect of skipping line-by-line feedback. We also found contradictory results between the two studies which highlight the need for reproducibility studies in empirical research. 

Parsons puzzles are popular for programming education. Identifying the strategies used by students solving Parsons puzzles is of interest because they can be used to determine to what extent students use the strategies typically associated with programming expertise, and to provide feedback and monitor the progress of students in a tutor. We propose solution sequence as an approximation of the student’s strategy for solving Parsons puzzles. It is scalable in terms of both the size of the puzzle and the number of students solving the puzzle. We propose BNF grammar to represent desirable puzzle-solving strategies associated with expert programmers. This representation is extensible and agnostic to the puzzle-solving strategies themselves. Finally, we propose a best match parser that matches a student’s solution sequence against the BNF grammar of a desirable strategy and quantifies the degree to which the student’s solution conforms to the desirable strategy. As a proof of concept, we used the parser to analyze the data collected by a Parsons puzzle tutor on if-else statements over five semesters and found a significant difference between C++ and Java puzzle-solvers in terms of their conformance to one desirable puzzle-solving strategy. Being able to tease out the effects of individual components of a strategy is one benefit of our approach: we found that relaxing shell-first constraint in the strategy resulted in significant improvement in the conformance of both C++ and Java students.