The 2021 return to face-to-face teaching and proctored exams revealed significant gaps in student learning during remote instruction. The challenge of supporting underperforming students is not expected to abate in the next 5-10 years as COVID-19-related learning losses compound structural inequalities in K-12 education. More recently, anecdotal evidence across courses shows declines in classroom attendance and student engagement. Lack of engagement indicates emotional barriers rather than intellectual deficiencies, and its growth coincides with the ongoing mental health epidemic. Regardless of the underlying reasons, professors are now faced with the unappealing choice of awarding failing grades to an uncomfortably large fraction of classes or awarding passing grades to students who do not seem prepared for the workforce or adult life in general. Faculty training, if it exists, addresses neither the scale of this situation nor the emotional/identity aspects of the problem. There is an urgent need for pedagogical remediation tools that can be applied without additional TA or staff resources, without training in psychiatry, and with only five or eight weeks remaining in the semester.
This work presents two work-in-progress interventions for engineering faculty who face the challenges described above. In the first intervention, students can improve their exam score by submitting videos of reworked exams. The requirement of voiceover forces students to understand the thought process behind problems, even if they have copied the answers from a friend. Incorporating peer review into the assignment reduces the workload for instructor grading. This intervention has been successfully implemented in sophomore- and senior-level courses with positive feedback from both faculty and students. In the second intervention, students who fail the midterm are offered an automatic passing exam grade (typically 51%) in exchange for submitting a knowledge inventory and remediation plan. Students create a glossary of terms and concepts from the class and rank them by their level of understanding. Recent iterations of the remediation plan also include reflections on emotions and support networks.
In February 2023, the project team will scale the interventions to freshman-level Introductory Programming, which has 400 students and the highest fail/withdrawal rate in the college. The large sample size will enable more robust statistics to correlate exam scores, intervention rubric items, and surveys on assignment effectiveness. Piloting interventions in a variety of environments and classes will establish best pedagogical practices that minimize instructors’ workload and decision fatigue. The ultimate goal of this project is to benefit students and faculty through well-defined and systematic interventions across the curriculum.
more »
« less
Insights from the First Year of Project # 2044472 “Improving the Conceptual Mastery of Engineering Students in High Enrollment Engineering Courses through Oral Exams
This project aims to enhance students’ learning in foundational engineering courses through oral exams based on the research conducted at the University of California San Diego. The adaptive dialogic nature of oral exams provides instructors an opportunity to better understand students’ thought processes, thus holding promise for improving both assessments of conceptual mastery and students’ learning attitudes and strategies. However, the issues of oral exam reliability, validity, and scalability have not been fully addressed. As with any assessment format, careful design is needed to maximize the benefits of oral exams to student learning and minimize the potential concerns. Compared to traditional written exams, oral exams have a unique design space, which involves a large range of parameters, including the type of oral assessment questions, grading criteria, how oral exams are administered, how questions are communicated and presented to the students, how feedback were provided, and other logistical perspectives such as weight of oral exam in overall course grade, frequency of oral assessment, etc.
In order to address the scalability for high enrollment classes, key elements of the project are the involvement of the entire instructional team (instructors and teaching assistants). Thus the project will create a new training program to prepare faculty and teaching assistants to administer oral exams that include considerations of issues such as bias and students with disabilities.
The purpose of this study is to create a framework to integrate oral exams in core undergraduate engineering courses, complementing existing assessment strategies by (1) creating a guideline to optimize the oral exam design parameters for the best students learning outcomes; and (2) Create a new training program to prepare faculty and teaching assistants to administer oral exams. The project will implement an iterative design strategy using an evidence-based approach of evaluation. The effectiveness of the oral exams will be evaluated by tracking student improvements on conceptual questions across consecutive oral exams in a single course, as well as across other courses.
Since its start in January 2021, the project is well underway. In this poster, we will present a summary of the results from year 1: (1) exploration of the oral exam design parameters, and its impact in students’ engagement and perception of oral exams towards learning; (2) the effectiveness of the newly developed instructor and teaching assistants training programs (3) The development of the evaluation instruments to gauge the project success; (4) instructors and teaching assistants experience and perceptions.
more »
« less
- Award ID(s):
- 2044472
- NSF-PAR ID:
- 10387432
- Date Published:
- Journal Name:
- ASEE Annual Conference proceedings
- ISSN:
- 1524-4644
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This work-in-progress paper presents an innovative practice of using oral exams to maintain academic integrity and promote student engagement in large-enrollment engineering courses during remote instruction. With the abrupt and widespread transition to distance learning and assessment brought on by the COVID-19 pandemic, there has been a registered upsurge in academic integrity violations globally. To address the challenge of compromised integrity, in the winter quarter of 2021 we have implemented oral exams across six mostly high-enrollment mechanical and electrical engineering undergraduate courses. We present our oral exam design parameters in each of the courses and discuss how oral exams relate to academic integrity, student engagement, stress, and implicit bias. We also address the challenge of scalability, as most of our oral exams were implemented in large classes, where academic integrity and student-instructor disconnection have generally gotten disproportionately worse during remote learning. Our survey results indicate that oral exams have positively contributed to academic integrity in our courses. Based on our preliminary study and experiences, we expect oral exams can be effectively leveraged to hinder cheating and foster academic honesty in students, even when in-person instruction and assessment resumes.more » « less
-
This is a research study that investigates the range of conceptions of prototyping in engineering design courses through exploring the conceptions and implementations from the instructors’ perspective. Prototyping is certainly an activity central to engineering design. The context of prototyping to support engineering education and practice has a range of implementations in an undergraduate engineering curriculum, from first-year engineering to capstone engineering design experiences. Understanding faculty conceptions’ of the reason, purpose, and place of prototyping can help illustrate how teaching and learning of the engineering design process is realistically implemented across a curriculum and how students are prepared for work practice. We seek to understand, and consequently improve, engineering design teaching and learning, through transformations of practice that are based on engineering education research. In this exploratory study, we interviewed three faculty members who teach engineering design in project-based learning courses across the curriculum of an undergraduate engineering program. This builds on related work done by the authors that previously investigated undergraduate engineering students’ conceptions of prototyping activities and process. With our instructor participants, a similar interview protocol was followed through semi-structured qualitative interviews. Data analysis has been undertaken through an emerging thematic analysis of these interview transcripts. Early findings characterize the focus on teaching the design process; the kind of feedback that the educators provide on students’ prototypes; students’ behavior while working on design projects; and educators’ perspectives on the design course. Understanding faculty conceptions with students’ conceptions of prototyping can shed light on the efficacy of using prototyping as an authentic experience in design teaching and learning. In project-based learning courses, particular issues of authenticity and assessment are under consideration, especially across the curriculum. More specifically, “proportions of problems” inform “problem solving” as one of the key characteristics in design thinking, teaching and learning. More attention to prototyping as part of the study of problem-solving processes can be useful to enhance understanding of the impact of instructional design. Challenges for teaching engineering design exist, and may be due to difficulties in framing design problems, recognizing what expertise students possess, and assessing their expertise to help them reach their goals, all at an appropriate place and ambiguity with student learning goals. Initial findings show that prototyping activities can help students become more reflective on their design. Scaffolded activities in prototyping can support self-regulated learning by students. The range of support and facilities, such as campus makerspaces, may also help students and instructors alike develop industry-ready engineering students.more » « less
-
null (Ed.)This research paper studies the challenges that mathematics faculty and graduate teaching assistants (GTAs) faced when moving active and collaborative calculus courses from in-person to virtual instruction. As part of a larger pedagogical change project (described below), the math department at a public Research-1 university began transitioning pre-calculus and calculus courses to an active and collaborative learning (ACL) format in Fall 2019. The change began with the introduction of collaborative worksheets in recitations which were led by GTAs and supported by undergraduate learning assistants (LAs). Students recitation periods collaboratively solving the worksheet problems on whiteboards. When COVID-19 forced the rapid transition to online teaching, these ACL efforts faced an array of challenges. Faculty and GTA reflections on the changes to teaching and learning provide insight into how instructional staff can be supported in implementing ACL across various modes of instruction. The calculus teaching change efforts discussed in this paper are part of an NSF-supported project that aims to make ACL the default method of instruction in highly enrolled gateway STEM courses across the institution. The theoretical framework for the project builds on existing work on grassroots change in higher education (Kezar and Lester, 2011) to study the effect of communities of practice on changing teaching culture. The project uses course-based communities of practice (Wenger, 1999) that include instructors, GTAs, and LAs working together to design and enact teaching change in the targeted courses alongside ongoing professional development for GTAs and LAs. Six faculty and five GTAs involved in the teaching change effort in mathematics were interviewed after the Spring 2020 semester ended. Interview questions focused on faculty and GTA experiences implementing active learning after the rapid transition to online teaching. A grounded coding scheme was used to identify common themes in the challenges faced by instructors and GTAs as they moved online and in the impacts of technology, LA support, and the department community of practice on the move to online teaching. Technology, including both access and capabilities, emerged as a common barrier to student engagement. A particular barrier was students’ reluctance to share video or participate orally in sessions that were being recorded, making group work more difficult than it had been in a physical classroom. In addition, most students lacked access to a tablet for freehand writing, presenting a significant hurdle for sharing mathematical notation when physical whiteboards were no longer an option. These challenges point to the importance of incorporating flexibility in active learning implementation and in the professional development that supports teaching changes toward active learning, since what is conceived for a collaborative physical classroom may be implemented in a much different environment. The full paper will present a detailed analysis of the data to better understand how faculty and GTA experiences in the transition to online delivery can inform planning and professional development as the larger institutional change effort moves forward both in mathematics and in other STEM fields.more » « less
-
Carvalho, Paulo F. (Ed.)Evidence-based teaching practices are associated with improved student academic performance. However, these practices encompass a wide range of activities and determining which type, intensity or duration of activity is effective at improving student exam performance has been elusive. To address this shortcoming, we used a previously validated classroom observation tool, Practical Observation Rubric to Assess Active Learning (PORTAAL) to measure the presence, intensity, and duration of evidence-based teaching practices in a retrospective study of upper and lower division biology courses. We determined the cognitive challenge of exams by categorizing all exam questions obtained from the courses using Bloom’s Taxonomy of Cognitive Domains. We used structural equation modeling to correlate the PORTAAL practices with exam performance while controlling for cognitive challenge of exams, students’ GPA at start of the term, and students’ demographic factors. Small group activities, randomly calling on students or groups to answer questions, explaining alternative answers, and total time students were thinking, working with others or answering questions had positive correlations with exam performance. On exams at higher Bloom’s levels, students explaining the reasoning underlying their answers, students working alone, and receiving positive feedback from the instructor also correlated with increased exam performance. Our study is the first to demonstrate a correlation between the intensity or duration of evidence-based PORTAAL practices and student exam performance while controlling for Bloom’s level of exams, as well as looking more specifically at which practices correlate with performance on exams at low and high Bloom’s levels. This level of detail will provide valuable insights for faculty as they prioritize changes to their teaching. As we found that multiple PORTAAL practices had a positive association with exam performance, it may be encouraging for instructors to realize that there are many ways to benefit students’ learning by incorporating these evidence-based teaching practices.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
