This paper describes an evidence based-practice paper to a formative response to the engineering faculty and students’ needs at Anonymous University.
Within two weeks, the pandemic forced the vast majority of the 1.5 million faculty and 20 million students nationwide to transition all courses from face-to-face to entirely online. Never in the history of higher education has there been a concerted effort to adapt so quickly and radically, nor have we had the technology to facilitate such a rapid and massive change. At Anonymous University, over 700 engineering educators were racing to transition their courses. Many of those faculty had never experienced online course preparation, much less taught one synchronously or asynchronously. Faculty development centers and technology specialists across the university made a great effort to aid educators in this transition. These educators had questions about the best practices for moving online, how their students were affected, and the best ways to engage their students. However, these faculty’s detailed questions were answerable only by faculty peers’ experience, students’ feedback, and advice from experts in relevant engineering education research-based practices. This paper describes rapid, continuous, and formative feedback provided by the Engineering Education Faculty Group (EEFG) to provide an immediate response for peer faculty guidance during the pandemic, creating a community of practice. The faculty membership spans multiple colleges in the university, including engineering, education, and liberal arts.
The EEFG transitioned immediately to weekly meetings focused on the rapidly changing needs of their colleagues. Two surveys were generated rapidly by Hammond et al. to characterize student and faculty concerns and needs in March of 2020 and were distributed through various means and media. Survey 1 and 2 had 3381 and 1506 respondents respectively with most being students, with 113 faculty respondents in survey 1, the focus of this piece of work. The first survey was disseminated as aggregated data to the College of Engineering faculty with suggested modifications to course structures based on these findings. The EEFG continued to meet and collaborate during the remainder of the Spring 2020 semester and has continued through to this day. This group has acted as a hub for teaching innovation in remote online pedagogy and techniques, while also operating as a support structure for members of the group, aiding those members with training in teaching tools, discussion difficult current events, and various challenges they are facing in their professional teaching lives.
While the aggregated data gathered from the surveys developed by Hammond et al. was useful beyond measure in the early weeks of the pandemic, little attention at the time was given to the responses of faculty to that survey. The focus of this work has been to characterize faculty perceptions at the beginning of the pandemic and compare those responses between engineering and non-engineering faculty respondents, while also comparing reported perceptions of pre- and post-transition to remote online teaching. Interviews were conducted between 4 members of the EEFG with the goal of characterizing some of the experiences they have had while being members of the group during the time of the pandemic utilizing Grounded theory qualitative analysis.
more »
« less
When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.
more »
« less
- NSF-PAR ID:
- 10283746
- Date Published:
- Journal Name:
- American Journal of Evaluation
- ISSN:
- 1098-2140
- Page Range / eLocation ID:
- 109821402093194
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Background Numerous studies show that active and engaging classrooms help students learn and persist in college, but adoption of new teaching practices has been slow. Professional development programs encourage instructors to implement new teaching methods and change the status quo in STEM undergraduate teaching, and structured observations of classrooms can be used in multiple ways to describe and assess this instruction. We addressed the challenge of measuring instructional change with observational protocols, data that often do not lend themselves easily to statistical comparisons. Challenges using observational data in comparative research designs include lack of descriptive utility for holistic measures and problems related to construct representation, non-normal distributions and Type-I error inflation for segmented measures.
Results We grouped 790 mathematics classes from 74 instructors using Latent Profile Analysis (a statistical clustering technique) and found four reliable categories of classes. Based on this grouping we proposed a simple proportional measure we called Proportion Non-Didactic Lecture (PND). The measure aggregated the proportions of interactive to lecture classes for each instructor. We tested the PND and a measure derived from the Reformed Teaching Observation Protocol (RTOP) with data from a professional development study. The PND worked in simple hypothesis tests but lacked some statistical power due to possible ceiling effects. However, the PND provided effective descriptions of changes in instructional approaches from pre to post. In tandem with examining the proportional measure, we also examined the RTOP-Sum, an existing outcome measure used in comparison studies. The measure is based on the aggregated items in a holistic observational protocol. As an aggregate measure we found it to be highly reliable, correlated highly with the PND, and had more statistical power than the PND. However, the RTOP measure did not provide the thick descriptions of teaching afforded by the PND.
Conclusions Findings suggest that useful dependent measures can be derived from both segmented and holistic observational measures. Both have strengths and weaknesses: measures from segmented data are best at describing changes in teaching, while measures derived from the RTOP have more statistical power. Determining the validity of these measures is important for future use of observational data in comparative studies.
-
Seagroves, Scott ; Barnes, Austin ; Metevier, Anne ; Porter, Jason ; Hunter, Lisa (Ed.)We designed, facilitated, and re-designed an inquiry activity in an introductory undergraduate astronomy research methods course at the University of Texas at Austin over two different semesters. The teaching venue for this inquiry activity took place in the course “AST 376R: A Practical Introduction to Research Methods”, the inquiry activity was inserted into an existing course structure, taking place over multiple class periods. We discuss how we were able to leverage the Professional Development Program (PDP) inquiry themes and introduce students to specific STEM practices, using this experience as a primer or mini version of a larger research activity and research experience that they would determine and lead themselves later on in the semester. In this paper we describe the benefits for students in this course and the lessons learned by the instructors.more » « less
-
Flexible classroom spaces, which have movable tables and chairs that can be easily rearranged into different layouts, make it easier for instructors to effectively implement active learning than a traditional lecture hall. Instructors can move throughout the room to interact with students during active learning, and they can rearrange the tables into small groups to facilitate conversation between students. Classroom technology, such as wall-mounted monitors and movable whiteboards, also facilitates active learning by allowing students to collaborate. In addition to enabling active learning, the flexible classroom can still be arranged in front-facing rows that support traditional lecture-based pedagogies. As a result, instructors do not have to make time- and effort-intensive changes to the way their courses are taught in order to use the flexible classroom. Instead, they can make small changes to add active learning. We are in the second year of a study of flexible classroom spaces funded by the National Science Foundation’s Division of Undergraduate Education. This project asks four research questions that investigate the relationships between the instructor, the students, and the classroom: 1) What pedagogy do instructors use in a flexible classroom space? 2) How do instructors take advantage of the instructional affordances (including the movable furniture, movable whiteboards, wall-mounted whiteboards, and wall-mounted monitors) of a flexible classroom? 3) What is the impact of faculty professional development on instructors’ use of flexible classroom spaces? and 4) How does the classroom influence the ways students interpret and engage in group learning activities? In the first year of our study we have developed five research instruments to answer these questions: a three-part classroom observation protocol, an instructor interview protocol, two instructor surveys, and a student survey. We have collected data from nine courses taught in one of ten flexible classrooms at the University of Michigan during the Fall 2018 semester. Two of these courses were first-year introduction to engineering courses co-taught by two instructors, and the other seven courses were sophomore- and junior-level core technical courses taught by one instructor. Five instructors participated in a faculty learning community that met three times during the semester to discuss active learning, to learn how to make the best use of the flexible classroom affordances, and to plan activities to implement in their courses. In each course we gathered data from the perspective of the instructor (through pre- and post-semester interviews), the researcher (through observations of three class meetings with our observation protocol), and the students (through conducting a student survey at the end of the semester). This poster presents qualitative and qualitative analyses of these data to answer our research questions, along with evidence based best practices for effectively using a flexible classroom.more » « less
-
The instructional practices used in introductory college courses often differ dramatically from those used in high school courses, and dissatisfaction with these practices is cited by students as a prominent reason for leaving science, technology, engineering, and mathematics (STEM) majors. To better characterize the transition to college course work, we investigated the extent to which incoming expectations of course activities differ based on student demographic characteristics, as well as how these expectations align with what students will experience. We surveyed more than 1500 undergraduate students in large introductory STEM courses at three research-intensive institutions during the first week of classes about their expectations regarding how class time would be spent in their courses. We found that first-generation and first-semester students predict less lecture than their peers and that class size had the largest effect on student predictions. We also collected classroom observation data from the courses and found that students generally underpredicted the amount of lecture observed in class. This misalignment between student predictions and experiences, especially for first-generation and first-semester college students and students enrolled in large- and medium-size classes, has implications for instructors and universities as they design curricula for introductory STEM courses with explicit retention goals.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1177/1098214020931941
