More Like this

1. Insights from the First Year of Project # 2044472 “Improving the Conceptual Mastery of Engineering Students in High Enrollment Engineering Courses through Oral Exams

   Qi, H. ; Lubarda M. ; Schurgers C. ; Sandoval C. ; Klement, L. ; Ghazinejad M. ; Relaford-Doyle, J. ; Kim, M. ; Minnes, M. ; Baghdadchi S. ; et al ( June 2022 , ASEE Annual Conference proceedings)

   This project aims to enhance students’ learning in foundational engineering courses through oral exams based on the research conducted at the University of California San Diego. The adaptive dialogic nature of oral exams provides instructors an opportunity to better understand students’ thought processes, thus holding promise for improving both assessments of conceptual mastery and students’ learning attitudes and strategies. However, the issues of oral exam reliability, validity, and scalability have not been fully addressed. As with any assessment format, careful design is needed to maximize the benefits of oral exams to student learning and minimize the potential concerns. Compared to traditional written exams, oral exams have a unique design space, which involves a large range of parameters, including the type of oral assessment questions, grading criteria, how oral exams are administered, how questions are communicated and presented to the students, how feedback were provided, and other logistical perspectives such as weight of oral exam in overall course grade, frequency of oral assessment, etc. In order to address the scalability for high enrollment classes, key elements of the project are the involvement of the entire instructional team (instructors and teaching assistants). Thus the project will create a new training program to prepare faculty and teaching assistants to administer oral exams that include considerations of issues such as bias and students with disabilities. The purpose of this study is to create a framework to integrate oral exams in core undergraduate engineering courses, complementing existing assessment strategies by (1) creating a guideline to optimize the oral exam design parameters for the best students learning outcomes; and (2) Create a new training program to prepare faculty and teaching assistants to administer oral exams. The project will implement an iterative design strategy using an evidence-based approach of evaluation. The effectiveness of the oral exams will be evaluated by tracking student improvements on conceptual questions across consecutive oral exams in a single course, as well as across other courses. Since its start in January 2021, the project is well underway. In this poster, we will present a summary of the results from year 1: (1) exploration of the oral exam design parameters, and its impact in students’ engagement and perception of oral exams towards learning; (2) the effectiveness of the newly developed instructor and teaching assistants training programs (3) The development of the evaluation instruments to gauge the project success; (4) instructors and teaching assistants experience and perceptions. 

   more » « less
2. Development, Implementation, Refining and Revising of Adaptive Platform Lessons for an Engineering Course

   Kaw, A. ; Yalcin, A. ; Gomes, R. ; Serrano, L. ; Scott, A. ; Lou, Y. ; Clark, R. ( July 2022 , Proceedings of ASEE Annual Conference and Exposition)

   Since the 2014 high-profile meta-analysis of undergraduate STEM courses, active learning has become a standard in higher education pedagogy. One way to provide active learning is through the flipped classroom. However, finding suitable pre-class learning activities to improve student preparation and the subsequent classroom environment, including student engagement, can present a challenge in the flipped modality. To address this challenge, adaptive learning lessons were developed for pre-class learning for a course in Numerical Methods. The lessons would then be used as part of a study to determine their cognitive and affective impacts. Before the study could be started, it involved constructing well-thought-out adaptive lessons. This paper discusses developing, refining, and revising the adaptive learning platform (ALP) lessons for pre-class learning in a Numerical Methods flipped course. In a prior pilot study at a large public southeastern university, the first author had developed ALP lessons for the pre-class learning for four (Nonlinear Equations, Matrix Algebra, Regression, Integration) of the eight topics covered in a Numerical Methods course. In the current follow-on study, the first author and two other instructors who teach Numerical Methods, one from a large southwestern urban university and another from an HBCU, collaborated on developing the adaptive lessons for the whole course. The work began in Fall 2020 by enumerating the various chapters and breaking each one into individual lessons. Each lesson would include five sections (introduction, learning objectives, video lectures, textbook content, assessment). The three instructors met semi-monthly to discuss the content that would form each lesson. The main discussion of the meetings centered on what a student would be expected to learn before coming to class, choosing appropriate content, agreeing on prerequisites, and choosing and making new assessment questions. Lessons were then created by the first author and his student team using a commercially available platform called RealizeIT. The content was tested by learning assistants and instructors. It is important to note that significant, if not all, parts of the content, such as videos and textbook material, were available through previously done work. The new adaptive lessons and the revised existing ones were completed in December 2020. The adaptive lessons were tested for implementation in Spring 2021 at the first author's university and made 15% of the students' grade calculation. Questions asked by students during office hours, on the LMS discussion board, and via emails while doing the lessons were used to update content, clarify questions, and revise hints offered by the platform. For example, all videos in the ALP lessons were updated to HD quality based on student feedback. In addition, comments from the end-of-semester surveys conducted by an independent assessment analyst were collated to revise the adaptive lessons further. Examples include changing the textbook content format from an embedded PDF file to HTML to improve quality and meet web accessibility standards. The paper walks the reader through the content of a typical lesson. It also shows the type of data collected by the adaptive learning platform via three examples of student interactions with a single lesson. 

   more » « less
3. Development, Implementation, Refining and Revising of Adaptive Platform Lessons for an Engineering Course

   Kaw, A. ( June 2023 , Proceedings of the 2022 ASEE Annual Conference and Exposition)

   Since the 2014 high-profile meta-analysis of undergraduate STEM courses, active learning has become a standard in higher education pedagogy. One way to provide active learning is through the flipped classroom. However, finding suitable pre-class learning activities to improve student preparation and the subsequent classroom environment, including student engagement, can present a challenge in the flipped modality. To address this challenge, adaptive learning lessons were developed for pre-class learning for a course in Numerical Methods. The lessons would then be used as part of a study to determine their cognitive and affective impacts. Before the study could be started, it involved constructing well-thought-out adaptive lessons. This paper discusses developing, refining, and revising the adaptive learning platform (ALP) lessons for pre-class learning in a Numerical Methods flipped course. In a prior pilot study at a large public southeastern university, the first author had developed ALP lessons for the pre-class learning for four (Nonlinear Equations, Matrix Algebra, Regression, Integration) of the eight topics covered in a Numerical Methods course. In the current follow-on study, the first author and two other instructors who teach Numerical Methods, one from a large southwestern urban university and another from an HBCU, collaborated on developing the adaptive lessons for the whole course. The work began in Fall 2020 by enumerating the various chapters and breaking each one into individual lessons. Each lesson would include five sections (introduction, learning objectives, video lectures, textbook content, assessment). The three instructors met semi-monthly to discuss the content that would form each lesson. The main discussion of the meetings centered on what a student would be expected to learn before coming to class, choosing appropriate content, agreeing on prerequisites, and choosing and making new assessment questions. Lessons were then created by the first author and his student team using a commercially available platform called RealizeIT. The content was tested by learning assistants and instructors. It is important to note that significant, if not all, parts of the content, such as videos and textbook material, were available through previously done work. The new adaptive lessons and the revised existing ones were completed in December 2020. The adaptive lessons were tested for implementation in Spring 2021 at the first author's university and made 15% of the students' grade calculation. Questions asked by students during office hours, on the LMS discussion board, and via emails while doing the lessons were used to update content, clarify questions, and revise hints offered by the platform. For example, all videos in the ALP lessons were updated to HD quality based on student feedback. In addition, comments from the end-of-semester surveys conducted by an independent assessment analyst were collated to revise the adaptive lessons further. Examples include changing the textbook content format from an embedded PDF file to HTML to improve quality and meet web accessibility standards. The paper walks the reader through the content of a typical lesson. It also shows the type of data collected by the adaptive learning platform via three examples of student interactions with a single lesson. 

   more » « less
4. Mastery Learning in Undergraduate Engineering Courses: A Systematic Review

   Perez Leon, C. ; Verdin, D. ( August 2022 , Zone 1 Conference of the American Society for Engineering Education)

   This theory paper focuses on understanding how mastery learning has been implemented in undergraduate engineering courses through a systematic review. Academic environments that promote learning, mastery, and continuous improvement rather than inherent ability can promote performance and persistence. Scholarship has argued that students could achieve mastery of the course material when the time available to master concepts and the quality of instruction was made appropriate to each learner. Increasing time to demonstrate mastery involves a course structure that allows for repeated attempts on learning assessments (i.e., homework, quizzes, projects, exams). Students are not penalized for failed attempts but are rewarded for achieving eventual mastery. The mastery learning approach recognizes that mastery is not always achieved on first attempts and learning from mistakes and persisting is fundamental to how we learn. This singular concept has potentially the greatest impact on students’ mindset in terms of their belief they can be successful in learning the course material. A significant amount of attention has been given to mastery learning courses in secondary education and mastery learning has shown an exceptionally positive effect on student achievement. However, implementing mastery learning in an undergraduate course can be a cumbersome process as it requires instructors to significantly restructure their assignments and exams, evaluation process, and grading practices. In light of these challenges, it is unclear the extent to which mastery learning has been implemented in undergraduate engineering courses or if similar positive effects can be found. Therefore, we conducted a systematic review to elucidate, how in the U.S., (1) has mastery learning been implemented in undergraduate engineering courses from 1990 to the present time and (2) the student outcomes that have been reported for these implementations. Using the systematic process outlined by Borrego et al. (2014), we surveyed seven databases and a total of 584 articles consisting of engineering and non-engineering courses were identified. We focused our review on studies that were centered on applying the mastery learning pedagogical method in undergraduate engineering courses. All peer-reviewed and practitioner articles and conference proceedings that were within our scope were included in the synthetization phase of the review. Most articles were excluded based on our inclusion and exclusion criteria. Twelve studies focused on applying mastery learning to undergraduate engineering courses. The mastery learning method was mainly applied on midterm exams, few studies used the method on homework assignments, and no study applied the method to the final exam. Students reported an increase in learning as a result of applying mastery learning. Several studies reported that students’ grades in a traditional final exam were not affected by mastery learning. Students’ self-reported evaluation of the course suggests that students prefer the mastery learning approach over traditional methods. Although a clear consensus on the effect of the mastery learning approach could not be achieved as each article applied different survey instruments to capture students’ perspectives. Responses to open-ended questions have mixed results. Two studies report more positive student comments on opened-ended questions, while one study report receiving more negative comments regarding the implementation of the mastery learning method. In the full paper we more thoroughly describe the ways in which mastery learning was implemented along with clear examples of common and divergent student outcomes across the twelve studies. 

   more » « less
5. Board 305: HSI Implementation and Evaluation Project: Commitment to Learning Instilled by Mastery-Based Undergraduate Program (CLIMB-UP)

   Verdin, D. ; Krinsky, S. ; Raymond, D. ; Schiorring, E. ; Allen, E. L. ; Perez, C. L. ( June 2023 , 2023 ASEE Annual Conference & Exposition)

   This is a board presentation at the 2023 ASEE Annual Conference describing the HSI Implementation and Evaluation Project: Commitment to Learning Instilled by Mastery-Based Undergraduate Program (CLIMB-UP). Commitment to Learning Instilled by a Mastery-Based Undergraduate Program (CLIMBUP) is an NSF IUSE:HSI project centered on re-designing courses with high non-completion rates (C- or lower) that have implications towards students’ graduation, transfer ability and retention. Despite decades of effort to create active, inquiry-based learning practices in classrooms, our institution continues to see equity gaps and many required courses with noncompletion rates exceeding 50%. Grading practices have been identified as one of the main culprits in the persistence of equity gaps. As a Hispanic Serving Institution, we recognize and value the diversity of experience that our students bring to our campuses and are committed to utilizing their strengths by creating datadriven, equitable grading practices that give students space to take risks and bring alternative viewpoints to our classrooms and be rewarded. We believe a Mastery-Based Grading (MBG) approach can address problems that a traditional grading approach has caused. The CLIMB-UP project is building the infrastructure to support and train STEM faculty (both tenure-line and adjuncts) to redesign and teach a Mastery-Based Graded (MGB) course, and is conducting research on faculty experiences and on the change in student attitudes, mindsets, and outcomes. 

   more » « less