More Like this

1. Understanding Context: Propagation and Effectiveness of the Concept Warehouse in Mechanical Engineering at Five Diverse Institutions and Beyond – Results from Year 2

   https://doi.org/10.18260/1-2--35419  

   Self, B. P.; Koretsky, M. D.; Papadopoulos, C.; Widmann, J. M. (July 2021, ASEE annual conference exposition)

   It has been well-established that concept-based active learning strategies increase student retention, improve engagement and student achievement, and reduce the performance gap of underrepresented students. Despite the evidence supporting concept-based instruction, many faculty continue to stress algorithmic problem solving. In fact, the biggest challenge to improving STEM education is not the need to develop more effective instructional practices, but to find ways to get faculty to adopt the evidence-based pedagogies that already exist. Our project aims to propagate the Concept Warehouse (CW), an online innovation tool that was developed in the Chemical Engineering community, into Mechanical Engineering (ME). A portion of our work focuses on content development in mechanics, and includes statics, dynamics, and to a lesser extent strength of materials. Our content development teams had created 170 statics and 253 dynamics questions. Additionally, we have developed four different simulations to be embedded in online Instructional Tools – these are interactive modules that provided different physical scenarios to help students understand important concepts in mechanics. During initial interviews, we found that potential adopters needed coaching on the benefits of concept-based instruction, training on how to use the CW, and support on how to best implement the different affordances offered by the CW. This caused a slight shift in our initial research plans, and much of our recent work has concentrated on using faculty development activities to help us advertise the CW and encourage evidence-based practices. From these activities, we are recruiting participants for surveys and interviews to help us investigate how different contexts affect the adoption of educational innovations. A set of two summer workshops attracted over 270 applicants, and over 60 participants attended each synchronous offering. Other applicants were provided links to recordings of the workshop. From these participants, we recruited 20 participants to join our Community of Practice (CoP). These members are sharing how they use the CW in their classes, especially in the virtual environment. Community members discuss using evidence-based practices, different things that the CW can do, and suggest potential improvements to the tool. They will also be interviewed to help us determine barriers to adoption, how their institutional contexts and individual epistemologies affect adoption, and how they have used the CW in their classes. Our research will help us formulate strategies that others can use when attempting to propagate pedagogical innovations. 

   more » « less
2. Understanding Context: Propagation and Effectiveness of the Concept Warehouse in Mechanical Engineering at Five Diverse Institutions and Beyond – Results from Year 1.

   Self, B. P.; Dal Bello, D. J.; Koretsky, M.; Nolen, S. B.; Papadopoulos, C.; Prince, M. J.; Widmann, J. M. (June 2020, ASEE Annual Conference proceedings)

   Several consensus reports cite a critical need to dramatically increase the number and diversity of STEM graduates over the next decade. They conclude that a change to evidence-based instructional practices, such as concept-based active learning, is needed. Concept-based active learning involves the use of activity-based pedagogies whose primary objectives are to make students value deep conceptual understanding (instead of only factual knowledge) and then to facilitate their development of that understanding. Concept-based active learning has been shown to increase academic engagement and student achievement, to significantly improve student retention in academic programs, and to reduce the performance gap of underrepresented students. Fostering students' mastery of fundamental concepts is central to real world problem solving, including several elements of engineering practice. Unfortunately, simply proving that these instructional practices are more effective than traditional methods is not enough to ensure widespread pedagogical change. In fact, the biggest challenge to improving STEM education is not the need to develop more effective instructional practices, but to find ways to get faculty to adopt the evidence-based pedagogies that already exist. 

   more » « less
3. WIP: Instances of Dynamic Pedagogical Decision Making in the Uptake of a Technology Tool

   Harpreet Auby; John Galisky; Susan Nolen; Milo Koretsky (June 2022, ASEE Annual Conference proceedings)

   In this work-in-progress paper, we continue investigation into the propagation of the Concept Warehouse within mechanical engineering (Friedrichsen et al., 2017; Koretsky et al., 2019a). Even before the pandemic forced most instruction online, educational technology was a growing element in classroom culture (Koretsky & Magana, 2019b). However, adoption of technology tools for widespread use is often conceived from a turn-key lens, with professional development focused on procedural competencies and fidelity of implementation as the goal (Mills & Ragan, 2000; O’Donnell, 2008). Educators are given the tool with initial operating instructions, then left on their own to implement it in particular instructional contexts. There is little emphasis on the inevitable instructional decisions around incorporating the tool (Hodge, 2019) or on sustainable incorporation of technologies into existing instructional practice (Forkosh-Baruch et al., 2021). We consider the take-up of a technology tool as an emergent, rather than a prescribed process (Henderson et al., 2011). In this WIP paper, we examine how two instructors who we call Al and Joe reason through their adoption of a technology tool, focusing on interactions among instructors, tool, and students within and across contexts. The Concept Warehouse (CW) is a widely-available, web-based, open educational technology tool used to facilitate concept-based active learning in different contexts (Friedrichsen et al., 2017; Koretsky et al., 2014). Development of the CW is ongoing and collaboration-driven, where user-instructors from different institutions and disciplines can develop conceptual questions (called ConcepTests) and other learning and assessment tools that can be shared with other users. Currently there are around 3,500 ConcepTests, 1,500 faculty users, and 36,000 student users. About 700 ConcepTests have been developed for mechanics (statics and dynamics). The tool’s spectrum of affordances allows different entry points for instructor engagement, but also allows their use to grow and change as they become familiar with the tool and take up ideas from the contexts around them. Part of a larger study of propagation and use across five diverse institutions (Nolen & Koretsky, 2020), instructors were introduced to the tool, offered an introductory workshop and opportunity to participate in a community of practice (CoP), then interviewed early and later in their adoption. For this paper, we explore a bounded case study of the two instructors, Al and Joe, who took up the CW to teach Introductory Statics. Al and Joe were experienced instructors, committed to active learning, who presented examples from their ongoing adaptation of the tool for discussion in the community of practice. However, their decisions about how to integrate the tool fundamentally differed, including the aspects of the tool they took up and the ways they made sense of their use. In analyzing these two cases, we begin to uncover how these instructors navigated the dynamic nature of pedagogical decision making in and across contexts. 

   more » « less
4. WIP: Using Machine Learning to Automate Coding of Student Explanations to Challenging Mechanics Concept Questions

   Harpreet Auby; Namrata Shivagunde; Anna Rumshisky; Milo Koretsky (June 2022, ASEE Annual Conference proceedings)

   This work-in-progress paper describes a collaborative effort between engineering education and machine learning researchers to automate analysis of written responses to conceptually challenging questions in mechanics. These qualitative questions are often used in large STEM classes to support active learning pedagogies; they require minimum calculations and focus on the application of underlying physical phenomena to various situations. Active learning pedagogies using this type of questions has been demonstrated to increase student achievement (Freeman et al., 2014; Hake, 1998) and engagement (Deslauriers, et al., 2011) of all students (Haak et al., 2011). To emphasize reasoning and sense-making, we use the Concept Warehouse (Koretsky et al., 2014), an audience response system where students provide written justifications to concept questions. Written justifications better prepare students for discussions with peers and in the whole class and can also improve students’ answer choices (Koretsky et al., 2016a, 2016b). In addition to their use as a tool to foster learning, written explanations can also provide valuable information to concurrently assess that learning (Koretsky and Magana, 2019). However, in practice, there has been limited deployment of written justifications with concept questions, in part, because they provide a daunting amount of information for instructors to process and for researchers to analyze. In this study, we describe the initial evaluation of large pre-trained generative sequence-to-sequence language models (Raffel et al., 2019; Brown et al., 2020) to automate the laborious coding process of student written responses. Adaptation of machine learning algorithms in this context is challenging since each question targets specific concepts which elicit their own unique reasoning processes. This exploratory project seeks to utilize responses collected through the Concept Warehouse to identify viable strategies for adapting machine learning to support instructors and researchers in identifying salient aspects of student thinking and understanding with these conceptually challenging questions. 

   more » « less
5. Toward Benchmarking Student Progress in Mechanics: Assessing Learning Cycles through Mastery Learning and Concept Questions

   Papadopoulos, C.; Santiago-Román, A. I.; Hillman, E. F.; Figueroa, G. L.; Vega Morales, I. (July 2021, ASEE Virtual Annual Conference)

   null (Ed.)

   In mechanics, the standard 3-credit, 45-hour course is sufficient to deliver standard lectures with prepared examples and questions. Moreover, it is not only feasible, but preferable, to employ any of a variety of active learning and teaching techniques. Nevertheless, even when active learning is strategically used, students and instructors alike experience pressure to accomplish their respective learning and teaching goals under the constraints of the academic calendar, raising questions as to whether the allocated time is sufficient to enable authentic learning. One way to assess learning progress is to examine the learning cycles through which students attempt, re-think, and re-attempt their work. This article provides data to benchmark the time required to learn key Statics concepts based on results of instruction of approximately 50 students in a Statics class at a public research university during the Fall 2020 semester. Two parallel techniques are employed to foster and understand student learning cycles. • Through a Mastery Based Learning model, 15 weekly pass/fail “Mastery Tests” are given. Students who do not pass may re-test with a different but similar test on the same topic each week until the semester’s conclusion. The tests are highly structured in that they are well posed and highly focused. For example, some tests focus only on drawing Free Body Diagrams, with no equations or calculations. Other tests focus on writing equilibrium equations from a given Free Body Diagram. Passing the first six tests is required to earn the grade of D; passing the next three for C; the next three for B; and the final three for A. Evaluations include coding of student responses to infer student reasoning. Learning cycles occur as students repeat the same topics, and their progress is assessed by passing rates and by comparing evolving responses to the same test topics. • Concept Questions that elicit qualitative responses and written explanations are deployed at least weekly. The learning cycle here consists of students answering a question, seeing the overall class results (but without the correct answer), having a chance to explore the question with other students and the instructor, and finally an opportunity to re-answer the same question, perhaps a few minutes or up to a couple days later. Sometimes, that same question is given a third time to encourage further effort or progress. To date, results from both cycles appear to agree on one important conclusion: the rate of demonstrated learning is quite low. For example, each Mastery Test has a passing rate of 20%-30%, including for students with several repeats. With the Concept Questions, typically no more than half of the students who answered incorrectly change to the correct answer by the time of the final poll. The final article will provide quantitative and qualitative results from each type of cycle, including tracking coded responses on Mastery Tests, written responses on Concept Questions, and cross-comparisons thereof. Additional results will be presented from student surveys. Since the Mastery Tests and Concept Questions follow typical Statics topics, this work has potential to lead to a standardized set of benchmarks and standards for measuring student learning – and its rate – in Statics. 

   more » « less