Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance
Title: Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance
Computer science pedagogy, especially in the higher education and vocational training context, has long-favored the hands-on practice provided by programming tasks due to the belief that this leads to better performance on hands-on tasks at work. This assumption, however, has not been experimentally tested against other modes of engagement such as worked example-based reflection. While theory suggests that example-based reflection could be better for conceptual learning, the concern is that the lack of practice will leave students unable to implement the learned concepts in practice, thus leaving them unprepared for work. In this paper, therefore, we experimentally contrast programming practice with example-based reflection to observe their differential impact on conceptual learning and performance on a hands-on task in the context of a collaborative programming project. The industry paradigm of Mob Programming, adapted for use in an online and instructional context, is used to structure the collaboration. Keeping with the prevailing view held in pedagogy, we hypothesize that example-based reflection will lead to better conceptual learning but will be detrimental to hands-on task performance. Results support that reflection leads to conceptual learning. Additionally, however, reflection does not pose an impediment to hands-on task performance. We discuss possible explanations for this effect, thus providing an improved understanding of prior theory in this new computer science education context. We also discuss implications for the pedagogy of software engineering education, in light of this new evidence, that impacts student learning as well as work performance in the future. more »« less
Sankaranarayanan, S.(
, General Proceedings of the 2nd Annual Meeting of the International Society of the Learning Sciences 2022)
Jun Oshima, Toshio Mochizuki
(Ed.)
Designing activities for maximizing collaborative learning in advanced computer
science contexts is of broad interest. While programming exercises remain the dominant form
of pedagogy here, prior work showed that collaborative reflection over worked examples is as
good or even better for conceptual learning and future programming. This work used a “phased”
design, with separate collaborative reflection and programming phases, and varied the time
boundary between the two to determine their differential impact. A more effective design,
however, could involve collaborative reflection prompted “in the flow” of programming, with
benefits similar to self-explanation prompts interleaved into individual problem-solving. While
total time-on-task is the same, this “interleaved” design might allow learners to spend a larger
proportion of this time on reflection. Thus, this paper compares this novel interleaved approach
to the phased design. We determine that interleaving increases the proportion of time available
for reflection resulting in performance improvements on future programming.
A solid understanding of electromagnetic (E&M) theory is key to the education of electrical engineering students. However, these concepts are notoriously challenging for students to learn, due to the difficulty in grasping abstract concepts such as the electric force as an invisible force that is acting at a distance, or how electromagnetic radiation is permeating and propagating in space. Building physical intuition to manipulate these abstractions requires means to visualize them in a three-dimensional space. This project involves the development of 3D visualizations of abstract E&M concepts in Virtual Reality (VR), in an immersive, exploratory, and engaging environment.
VR provides the means of exploration, to construct visuals and manipulable objects to represent knowledge. This leads to a constructivist way of learning, in the sense that students are allowed to build their own knowledge from meaningful experiences. In addition, the VR labs replace the cost of hands-on labs, by recreating the experiments and experiences on Virtual Reality platforms.
The development of the VR labs for E&M courses involves four distinct phases: (I) Lab Design, (II) Experience Design, (III) Software Development, and (IV) User Testing. During phase I, the learning goals and possible outcomes are clearly defined, to provide context for the VR laboratory experience, and to identify possible technical constraints pertaining to the specific laboratory exercise. During stage II, the environment (the world) the player (user) will experience is designed, along with the foundational elements, such as ways of navigation, key actions, and immersion elements. During stage III, the software is generated as part of the course projects for the Virtual Reality course taught in the Computer Science Department at the same university, or as part of independent research projects involving engineering students. This reflects the strong educational impact of this project, as it allows students to contribute to the educational experiences of their peers. During phase IV, the VR experiences are played by different types of audiences that fit the player type. The team collects feedback and if needed, implements changes.
The pilot VR Lab, introduced as an additional instructional tool for the E&M course during the Fall 2019, engaged over 100 students in the program, where in addition to the regular lectures, students attended one hour per week in the E&M VR lab. Student competencies around conceptual understanding of electromagnetism topics are measured via formative and summative assessments. To evaluate the effectiveness of VR learning, each lab is followed by a 10-minute multiple-choice test, designed to measure conceptual understanding of the various topics, rather than the ability to simply manipulate equations.
This paper discusses the implementation and the pedagogy of the Virtual Reality laboratory experiences to visualize concepts in E&M, with examples for specific labs, as well as challenges, and student feedback with the new approach. We will also discuss the integration of the 3D visualizations into lab exercises, and the design of the student assessment tools used to assess the knowledge gain when the VR technology is employed.
This Innovate Practice full paper presents a cloud-based personalized learning lab platform. Personalized learning is gaining popularity in online computer science education due to its characteristics of pacing the learning progress and adapting the instructional approach to each individual learner from a diverse background. Among various instructional methods in computer science education, hands-on labs have unique requirements of understanding learner's behavior and assessing learner's performance for personalization. However, it is rarely addressed in existing research. In this paper, we propose a personalized learning platform called ThoTh Lab specifically designed for computer science hands-on labs in a cloud environment. ThoTh Lab can identify the learning style from student activities and adapt learning material accordingly. With the awareness of student learning styles, instructors are able to use techniques more suitable for the specific student, and hence, improve the speed and quality of the learning process. With that in mind, ThoTh Lab also provides student performance prediction, which allows the instructors to change the learning progress and take other measurements to help the students timely. For example, instructors may provide more detailed instructions to help slow starters, while assigning more challenging labs to those quick learners in the same class. To evaluate ThoTh Lab, we conducted an experiment and collected data from an upper-division cybersecurity class for undergraduate students at Arizona State University in the US. The results show that ThoTh Lab can identify learning style with reasonable accuracy. By leveraging the personalized lab platform for a senior level cybersecurity course, our lab-use study also shows that the presented solution improves students engagement with better understanding of lab assignments, spending more effort on hands-on projects, and thus greatly enhancing learning outcomes.
Taking part in creating location-based augmented reality (LBAR) experiences that focus on communication, art and design could serve as an entry point for art-oriented girls and young women towards career pathways in computer science and information communication technology. This conceptual paper presents our theory-based approach and subsequent application, as well as lessons learned informed by team discussions and reflections. We built an LBAR program entitled AR Girls on four foundational principles: stealth science (embedding science in familiar appealing experiences), place-based education (situating learning in one’s own community), non-hierarchical design (collaborations where both adults and youth generate content), and learning through design (engaging in design, not just play). To translate these principles into practice, we centered the program around the theme of art by forming partnerships with small community art organizations and positioning LBAR as an art-based communication medium. We found that LBAR lends itself to an interdisciplinary approach that blends technology, art, science and communication. We believe our approach helped girls make connections to their existing interests and build soft skills such as leadership and interpersonal communication as they designed local environmentally-focused LBAR walking tours. Our “use-modify-create” approach provided first-hand experiences with the AR software early on, and thus supported the girls and their art educators in designing and showcasing their walking tours. Unfortunately, the four foundational principles introduced considerable complexity to AR Girls, which impacted recruitment and retention, and at times overwhelmed the art educators who co-led the program. To position AR Girls for long-term success, we simplified the program approach and implementation, including switching to a more user-friendly AR software; reducing logistical challenges of location-based design and play; narrowing the topic addressed by the girls design; and making the involvement of community partners optional. Overall, our initial work was instrumental in understanding how to translate theoretical considerations for learning in out-of-school settings into an LBAR program aimed at achieving multiple complementary outcomes for participating girls. Ultimately, we achieved better scalability by simplifying AR Girls both conceptually and practically. The lessons learned from AR Girls can inform others using LBAR for education and youth development programming.
The subgoal learning framework has improved performance for novice programmers in higher education, but it has only started to be applied and studied in K-12 (primary/secondary). Programming education in K-12 is growing, and many international initiatives are attempting to increase participation, including curricular initiatives like Computer Science Principles and non-profit organizations like Code.org. Given that subgoal learning is designed to help students with no prior knowledge, we designed and implemented subgoals in the introduction to programming unit in Code.org's Computer Science Principles course. The redesigned unit includes subgoal-oriented instruction and subgoal-themed pre-written comments that students could add to their programming activities. To evaluate efficacy, we compared behaviors and performance of students who received the redesigned subgoal unit to those receiving the original unit. We found that students who learned with subgoals performed better on problem-solving questions but not knowledge-based questions and wrote more in open-ended response questions, including a practice Performance Task for the AP exam. Moreover, at least one-third of subgoal students continued to use the subgoal comments after the subgoal-oriented instruction had been faded, suggesting that they found them useful. Survey data from the teachers suggested that students who struggled with the concepts found the subgoals most useful. Implications for future designs are discussed.
Sankaranarayanan, Sreecharan, Kandimalla, Siddharth Reddy, Bogart, Christopher, Murray, R. Charles, Hilton, Michael, Sakr, Majd F, and Rose, Carolyn Penstein. Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance. Retrieved from https://par.nsf.gov/biblio/10356430. IEEE Transactions on Learning Technologies . Web. doi:10.1109/TLT.2022.3169121.
Sankaranarayanan, Sreecharan, Kandimalla, Siddharth Reddy, Bogart, Christopher, Murray, R. Charles, Hilton, Michael, Sakr, Majd F, & Rose, Carolyn Penstein. Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance. IEEE Transactions on Learning Technologies, (). Retrieved from https://par.nsf.gov/biblio/10356430. https://doi.org/10.1109/TLT.2022.3169121
Sankaranarayanan, Sreecharan, Kandimalla, Siddharth Reddy, Bogart, Christopher, Murray, R. Charles, Hilton, Michael, Sakr, Majd F, and Rose, Carolyn Penstein.
"Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance". IEEE Transactions on Learning Technologies (). Country unknown/Code not available. https://doi.org/10.1109/TLT.2022.3169121.https://par.nsf.gov/biblio/10356430.
@article{osti_10356430,
place = {Country unknown/Code not available},
title = {Collaborative Programming for Work-Relevant Learning: Comparing Programming Practice with Example-Based Reflection for Student Learning and Transfer Task Performance},
url = {https://par.nsf.gov/biblio/10356430},
DOI = {10.1109/TLT.2022.3169121},
abstractNote = {Computer science pedagogy, especially in the higher education and vocational training context, has long-favored the hands-on practice provided by programming tasks due to the belief that this leads to better performance on hands-on tasks at work. This assumption, however, has not been experimentally tested against other modes of engagement such as worked example-based reflection. While theory suggests that example-based reflection could be better for conceptual learning, the concern is that the lack of practice will leave students unable to implement the learned concepts in practice, thus leaving them unprepared for work. In this paper, therefore, we experimentally contrast programming practice with example-based reflection to observe their differential impact on conceptual learning and performance on a hands-on task in the context of a collaborative programming project. The industry paradigm of Mob Programming, adapted for use in an online and instructional context, is used to structure the collaboration. Keeping with the prevailing view held in pedagogy, we hypothesize that example-based reflection will lead to better conceptual learning but will be detrimental to hands-on task performance. Results support that reflection leads to conceptual learning. Additionally, however, reflection does not pose an impediment to hands-on task performance. We discuss possible explanations for this effect, thus providing an improved understanding of prior theory in this new computer science education context. We also discuss implications for the pedagogy of software engineering education, in light of this new evidence, that impacts student learning as well as work performance in the future.},
journal = {IEEE Transactions on Learning Technologies},
author = {Sankaranarayanan, Sreecharan and Kandimalla, Siddharth Reddy and Bogart, Christopher and Murray, R. Charles and Hilton, Michael and Sakr, Majd F and Rose, Carolyn Penstein},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
