skip to main content


Title: Assessing Bootstrap: Algebra Students on Scaffolded and Unscaffolded Word Problems
Bootstrap:Algebra is a curricular module designed to integrate introductory computing into an algebra class; the module aims to help students improve on various essential learning outcomes from state and national algebra standards. In prior work, we published initial findings about student performance gains on algebra problems after taking Bootstrap. While the results were promising, the dataset was not large, and had students working on algebra problems that had been scaffolded with Bootstrap's pedagogy. This paper reports on a more detailed study with (a) data from more than three times as many students, (b) analysis of performance changes in incorrect answers, (c) some problems in which the Bootstrap scaffolds have been removed, and (d) an IRT analysis across the elements of Bootstrap's program-design pedagogy. Our results confirm that students improve on algebraic word problems after completing the module, even on unscaffolded problems. The nature of incorrect answers to symbolic-form questions also appears to improve after Bootstrap.  more » « less
Award ID(s):
1535276
NSF-PAR ID:
10072911
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Proceedings of the 49th ACM Technical Symposium on Computer Science Education
Page Range / eLocation ID:
8 to 13
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Sacristán, A. I. ; Cortés-Zavala, J. C. ; Ruiz-Arias, P. M. (Ed.)
    What impact, if any, do interesting lessons have on the types of questions students ask? To explore this question, we used lesson observations of six teachers from three high schools in the Northeast who were part of a larger study. Lessons come from a range of courses, spanning Algebra through Calculus. After each lesson, students reported interest via lesson experience surveys (Author, 2019). These interest measures were then used to identify each teachers’ highest and lowest interest lessons. The two lessons per teacher allows us to compare across the same set of students per teacher. We compiled 145 student questions and identified whether questions were asked within a group work setting or part of a whole class discussion. Two coders coded 10% of data to improve the rubric for type of students’ questions (what, why, how, and if) and perceived intent (factual, procedural, reasoning, and exploratory). Factual questions asked for definitions or explicit answers. Procedural questions were raised when students looked for algorithms or a solving process. Reasoning questions asked about why procedures worked, or facts were true. Exploratory questions expanded beyond the topic of focus, such as asking about changing the parameters to make sense of a problem. The remaining 90% of data were coded independently to determine interrater reliability (see Landis & Koch, 1977). A Cohen’s Kappa statistic (K=0.87, p<0.001) indicates excellent reliability. Furthermore, both coders reconciled codes before continuing with data analysis. Initial results showed differences between high- and low-interest lessons. Although students raised fewer mathematical questions in high-interest lessons (59) when compared with low-interest lessons (86), high-interest lessons contained more “exploratory” questions (10 versus 6). A chi-square test of independence shows a significant difference, χ2 (3, N = 145) = 12.99, p = .005 for types of students’ questions asked in high- and low-interest lessons. The high-interest lessons had more student questions arise during whole class discussions, whereas low-interest lessons had more student questions during group work. By partitioning each lesson into acts at points where the mathematical content shifted, we were able to examine through how many acts questions remained open. The average number of acts the students’ questions remained unanswered for high-interest lessons (2.66) was higher than that of low-interest lessons (1.68). Paired samples t-tests suggest that this difference is significant t(5)=2.58, p = 0.049. Therefore, student interest in the lesson did appear to impact the type of questions students ask. One possible reason for the differences in student questions is the nature of the lessons students found interesting, which may allow for student freedom to wonder and chase their mathematical ideas. There may be more overall student questions in low-interest lessons because of confusion, but more research is needed to unpack the reasoning behind student questions. 
    more » « less
  2. The process of synthesizing solutions for mathematical problems is cognitively complex. Students formulate and implement strate- gies to solve mathematical problems, develop solutions, and make connections between their learned concepts as they apply their reasoning skills to solve such problems. The gaps in student knowl- edge or shallowly-learned concepts may cause students to guess at answers or otherwise apply the wrong approach, resulting in errors in their solutions. Despite the complexity of the synthesis process in mathematics learning, teachers’ knowledge and ability to anticipate areas of potential difficulty is essential and correlated with student learning outcomes. Preemptively identifying the common miscon- ceptions in students that result in subsequent incorrect attempts can be arduous and unreliable, even for experienced teachers. This pa- per aims to help teachers identify the subsequent incorrect attempts that commonly occur when students are working on math problems such that they can address the underlying gaps in knowledge and common misconceptions through feedback. We report on a longi- tudinal analysis of historical data, from a computer-based learning platform, exploring the incorrect answers in the prior school years (’15-’20) that establish the commonality of wrong answers on two Open Educational Resources (OER) curricula–Illustrative Math (IM) and EngageNY (ENY) for grades 6, 7, and 8. We observe that incor- rect answers are pervasive across 5 academic years despite changes in underlying student and teacher population. Building on our find- ings regarding the Common Wrong Answers (CWAs), we report on goals and task analysis that we leveraged in designing and develop- ing a crowdsourcing platform for teachers to write Common Wrong Answer Feedback (CWAF) aimed are remediating the underlying cause of the CWAs. Finally, we report on an in vivo study by analyz- ing the effectiveness of CWAFs using two approaches; first, we use next-problem-correctness as a dependent measure after receiving CWAF in an intent-to-treat second, using next-attempt correctness as a dependent measure after receiving CWAF in a treated analysis. With the rise in popularity and usage of computer-based learning platforms, this paper explores the potential benefits of scalability in identifying CWAs and the subsequent usage of crowd-sourced CWAFs in enhancing the student learning experience through re- mediation. 
    more » « less
  3. The Hispanic Serving Institution Advanced Technological Education Hub 2 (HSI ATE Hub 2) is a three-year collaborative research project funded by the National Science Foundation (NSF) that builds upon the successful outcomes of two mentoring and professional development (PD) programs in a pilot that translates foundational theory related to culturally responsive pedagogy into practice using a 3-tier scaffolded faculty PD model. The goal of HSI ATE Hub 2 is to improve outcomes for Latinx students in technician education programs through design, development, pilot, optimization, and dissemination of this model at 2-year Hispanic Serving Institutions (HSIs). The tiered PD model has been tested by two faculty cohorts at Westchester Community College (WCC), an HSI in the State University of New York (SUNY) system. In year one, Cohort A piloted the PD modules in Tier 1 which featured reflective exercises and small culturally responsive activities to try with their STEM students. In year two, Cohort A piloted the PD modules in Tier 2 and peer-mentored Cohort B as they piloted optimizations introduced to Tier 1 from Cohort A feedback. Three types of optimizations came from faculty feedback. The first considered feedback regarding delivery and/or nature of the content that influenced a subsequent module. The second involved making changes to a particular module before it was delivered to another faculty cohort. The third takes into account what worked and what didn’t to decide which content to bring into virtual webinars for the broader advanced technician education community. Dissemination of the tiered PD model has been achieved in annual webinars with the broader ATE community and at conferences for advanced technician educators to achieve broader impacts in the ATE Community. Longer term, providing professional development in culturally responsive pedagogy and practices can help existing and future faculty learn to productively engage their students in more inclusive ways. As faculty mindsets shift to asset-based thinking and a climate of mutual respect is developed, the learning environment for all students in technician education programs will improve. When students learn in a supportive environment, their chances for success increase. The professional development provided in the HSI ATE Hub 2 project will lead to longer term improvements in four ways: 1) Retainment of Culturally responsive practices by those directly engaged after the project ends; 2) Inserting top activities from the PD into national webinars to extend the reach of the training; 3) Strengthening grant proposals as faculty integrate culturally responsive strategies, knowledge and experience within their ATE proposals to the NSF; and 4) Meeting industry demand for a diverse technician workforce. This second paper in a three-part series describes ongoing progress and lessons learned in developing and piloting the 3-Tier PD model with two Cohorts of STEM faculty at a 2-year HSI. 
    more » « less
  4. The Hispanic Serving Institution Advanced Technological Education Hub 2 (HSI ATE Hub 2) is a three-year collaborative research project funded by the National Science Foundation (NSF) that builds upon the successful outcomes of two mentoring and professional development (PD) programs in a pilot that translates foundational theory related to culturally responsive pedagogy into practice using a 3-tier scaffolded faculty PD model. The goal of HSI ATE Hub 2 is to improve outcomes for Latinx students in technician education programs through design, development, pilot, optimization, and dissemination of this model at 2-year Hispanic Serving Institutions (HSIs). The tiered PD model has been tested by two faculty cohorts at Westchester Community College (WCC), an HSI in the State University of New York (SUNY) system. In year one, Cohort A piloted the PD modules in Tier 1 which featured reflective exercises and small culturally responsive activities to try with their STEM students. In year two, Cohort A piloted the PD modules in Tier 2 and peer-mentored Cohort B as they piloted optimizations introduced to Tier 1 from Cohort A feedback. Three types of optimizations came from faculty feedback. The first considered feedback regarding delivery and/or nature of the content that influenced a subsequent module. The second involved making changes to a particular module before it was delivered to another faculty cohort. The third takes into account what worked and what didn’t to decide which content to bring into virtual webinars for the broader advanced technician education community. Dissemination of the tiered PD model has been achieved in annual webinars with the broader ATE community and at conferences for advanced technician educators to achieve broader impacts in the ATE Community. Longer term, providing professional development in culturally responsive pedagogy and practices can help existing and future faculty learn to productively engage their students in more inclusive ways. As faculty mindsets shift to asset-based thinking and a climate of mutual respect is developed, the learning environment for all students in technician education programs will improve. When students learn in a supportive environment, their chances for success increase. The professional development provided in the HSI ATE Hub 2 project will lead to longer term improvements in four ways: 1) Retainment of Culturally responsive practices by those directly engaged after the project ends; 2) Inserting top activities from the PD into national webinars to extend the reach of the training; 3) Strengthening grant proposals as faculty integrate culturally responsive strategies, knowledge and experience within their ATE proposals to the NSF; and 4) Meeting industry demand for a diverse technician workforce. This second paper in a three-part series describes ongoing progress and lessons learned in developing and piloting the 3-Tier PD model with two Cohorts of STEM faculty at a 2-year HSI. 
    more » « less
  5. Practice plays a critical role in learning engineering dynamics. Typical practice in a dynamics course involves solving textbook problems. These problems can impose great cognitive load on underprepared students because they have not mastered constituent knowledge and skills required for solving whole problems. For these students, learning can be improved by being engaged in deliberate practice. Deliberate practice refers to a type of practice aimed at improving specific constituent knowledge or skills. Compared to solving whole problems requiring the simultaneous use of multiple constituent skills, deliberate practice is usually focused on one component skill at a time, which results in less cognitive load and more specificity. Contemporary theories of expertise development have highlighted the influence of deliberate practice (DP) on achieving exceptional performance in sports, music, and various professional fields. Concurrently, there is an emerging method for improving learning efficiency of novices by combining deliberate practice with cognitive load theory (CLT), a cognitive-architecture-based theory for instructional design. Mechanics is a foundation for most branches of engineering. It serves to develop problem-solving skills and consolidate understanding of other subjects, such as applied mathematics and physics. Mechanics has been a challenging subject. Students need to understand governing principles to gain conceptual knowledge and acquire procedural knowledge to apply these principles to solve problems. Due to the difficulty in developing conceptual and procedural knowledge, mechanics courses are among those that receive high DFW rates (percentage of students receiving a grade of D or F or Withdrawing from a course), and students are more likely to leave engineering after taking mechanics courses. Deliberate practice can help novices develop good representations of the knowledge needed to produce superior problem solving performance. The goal of the present study is to develop deliberate practice techniques to improve learning effectiveness and to reduce cognitive load. Our pilot study results revealed that the student mental effort scores were negatively correlated with their knowledge test scores with r = -.29 (p < .05) after using deliberate practice strategies. This supports the claim that deliberate practice can improve student learning while reducing cognitive load. In addition, the higher the students’ knowledge test scores, the lower their mental effort was when taking the tests. In other words, the students who used deliberate practice strategies had better learning results with less cognitive load. To design deliberate practice, we often need to analyze students’ persistent problems caused by faulty mental models, also referred to as an intuitive mental model, and misconceptions. In this study, we continue to conduct an in-depth diagnostic process to identify students’ common mistakes and associated intuitive mental models. We then use the results to develop deliberate practice problems aimed at changing students’ cognitive strategies and mental models. 
    more » « less