skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Providing the Option to Skip Feedback – A Reproducibility Study
Would providing choice lead to improved learning with a tutor? We had conducted and reported a controlled study earlier, wherein, introductory programing students were given the choice of skipping the line-by-line feedback provided after each incorrect answer in a tutor on if/if-else statements. Contrary to expectations, the study found that the choice to skip feedback did not lead to greater learning. We tried to reproduce these results using two tutors on if/if-else and switch statements, and with a larger subject pool. We found that whereas choice did not lead to greater learning on if/if-else tutor in this reproducibility study either, it resulted in decreased learning on switch tutor. We hypothesize that skipping feedback is indeed detrimental to learning. But, inter-relationships among the concepts covered by a tutor and the transfer of learning facilitated by these relationships compensate for the negative effect of skipping line-by-line feedback. We also found contradictory results between the two studies which highlight the need for reproducibility studies in empirical research.  more » « less
Award ID(s):
1432190
PAR ID:
10097974
Author(s) / Creator(s):
Date Published:
Journal Name:
Intelligent Tutoring Systems
Page Range / eLocation ID:
180-185
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. (, Proceedings of AI-ED 2019)
    We conducted a controlled study to investigate whether having students choose the concept on which to solve each practice problem in an adaptive tutor helped improve learning. We analyzed data from an adaptive tutor used by introductory programming students over three semesters. The tutor presented code-tracing problems, used pretest-practice-post-test protocol, and presented line-by-line explanation of the correct solution as feedback. We found that choice did not in-crease the amount of learning or pace of learning. But, it resulted in greater improvement in score on learned concepts, and the effect size was medium. 
    more » « less
  2. (, Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education)
    A study was conducted to reproduce the results of an earlier study on the effectiveness of visualization for learning expression evaluation in a problem-solving software tutor on arithmetic expressions. In the current reproducibility study, data was collected from a software tutor on assignment expressions over six semesters. ANOVA analysis of the amount and speed of learning was conducted with treatment, sex and racial groups as fixed factors. Results include that visualization helped the students learn significantly more concepts, whether the students needed to use the tutor or benefited from using the tutor. However, it only benefited the less-prepared students. It did not help the students learn faster. It benefited both the sexes and traditionally represented as well as underrepresented groups. The current study confirmed almost all the results from the previous study, albeit for a harder topic. One reason why visualization was found to be effective in both these studies may be that the same visualization scheme was used by the students to both view feedback and construct their answers. 
    more » « less
  3. (, Intelligent Tutoring Systems 2020)
    Do students retain the programming concepts they have learned using software tutors over the long term? In order to answer this question, we analyzed the data collected by a software tutor on selection statements. We used the data of the students who used the tutor more than once to see whether they had retained for the second session what they had learned during the first session. We found that students retained over 71% of selection concepts that they had learned during the first session. The more problems students solved during the first session, the greater the percentage of retention. Even when students already knew a concept and did not benefit from using the tutor, a small percentage of concepts were for-gotten from the first session to the next, corresponding to transience of learning. Transience of learning varied with concepts. We list confounding factors of the study. 
    more » « less
  4. ; (, Proceedings of the 21stInternational Conference on Artificial Intelligence in Education)
    Reported here are the findings of a comparative study on the effects of using a Socratic Intelligent Tutoring System for source code comprehension and learning computer programming. The result shows there are significant differences between the two groups where students who used Socratic Tutor ITS improved their knowledge by 45% in term of learning gain, developed a better understanding of concepts such as nested if-else and for loop, and improved their confidence level by 13%. Furthermore, the result of the Pearson product-moment correlation coefficient shows a positive correlation (r = 0.68) between feedback from the ITS and learning gain. 
    more » « less
  5. (, Intelligent Tutoring Systems)
    Parsons puzzles are popular for programming education. Identifying the strategies used by students solving Parsons puzzles is of interest because they can be used to determine to what extent students use the strategies typically associated with programming expertise, and to provide feedback and monitor the progress of students in a tutor. We propose solution sequence as an approximation of the student’s strategy for solving Parsons puzzles. It is scalable in terms of both the size of the puzzle and the number of students solving the puzzle. We propose BNF grammar to represent desirable puzzle-solving strategies associated with expert programmers. This representation is extensible and agnostic to the puzzle-solving strategies themselves. Finally, we propose a best match parser that matches a student’s solution sequence against the BNF grammar of a desirable strategy and quantifies the degree to which the student’s solution conforms to the desirable strategy. As a proof of concept, we used the parser to analyze the data collected by a Parsons puzzle tutor on if-else statements over five semesters and found a significant difference between C++ and Java puzzle-solvers in terms of their conformance to one desirable puzzle-solving strategy. Being able to tease out the effects of individual components of a strategy is one benefit of our approach: we found that relaxing shell-first constraint in the strategy resulted in significant improvement in the conformance of both C++ and Java students. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email