Search for: All records

Abstract BackgroundNumerous studies show that active and engaging classrooms help students learn and persist in college, but adoption of new teaching practices has been slow. Professional development programs encourage instructors to implement new teaching methods and change the status quo in STEM undergraduate teaching, and structured observations of classrooms can be used in multiple ways to describe and assess this instruction. We addressed the challenge of measuring instructional change with observational protocols, data that often do not lend themselves easily to statistical comparisons. Challenges using observational data in comparative research designs include lack of descriptive utility for holistic measures and problems related to construct representation, non-normal distributions and Type-I error inflation for segmented measures. ResultsWe grouped 790 mathematics classes from 74 instructors using Latent Profile Analysis (a statistical clustering technique) and found four reliable categories of classes. Based on this grouping we proposed a simple proportional measure we called Proportion Non-Didactic Lecture (PND). The measure aggregated the proportions of interactive to lecture classes for each instructor. We tested the PND and a measure derived from the Reformed Teaching Observation Protocol (RTOP) with data from a professional development study. The PND worked in simple hypothesis tests but lacked some statistical power due to possible ceiling effects. However, the PND provided effective descriptions of changes in instructional approaches from pre to post. In tandem with examining the proportional measure, we also examined the RTOP-Sum, an existing outcome measure used in comparison studies. The measure is based on the aggregated items in a holistic observational protocol. As an aggregate measure we found it to be highly reliable, correlated highly with the PND, and had more statistical power than the PND. However, the RTOP measure did not provide the thick descriptions of teaching afforded by the PND. ConclusionsFindings suggest that useful dependent measures can be derived from both segmented and holistic observational measures. Both have strengths and weaknesses: measures from segmented data are best at describing changes in teaching, while measures derived from the RTOP have more statistical power. Determining the validity of these measures is important for future use of observational data in comparative studies. 

Projects to improve teaching as a means to improve learning need good tools to measure shifts in the use of effective research-based instructional strategies. We describe a survey tool to assess active and collaborative learning practices in college STEM by asking students to report what practices are used commonly in their course. Student reports correlate with their experience of the learning environment, and with their self-reported learning outcomes. Student means by section are corroborated by instructor-reported use of these practices and external observations. 

Studies show that Research-Based Instructional Strategies (RBIS) help students learn, however their adoption has been slow. The Teacher Centered Systematic Reform Model (TCRM) is a general model for organizing enablers and barriers to adoption of new teaching methods that includes departmental, personal and teacher thinking factors. We used the TCRM model as a framework to assess the amount of formal lecture reported by 634 mathematics instructors in their undergraduate courses. Regression analyses found that instructors who participated in Project NExT (a professional development workshop) during their early careers were less likely to use lecture than non-participants. Other significant predictors of lecture less included evaluation expectations emphasizing active teaching methods, involvement in equity and diversity efforts, and prior experience with RBIS. Factors with a positive correlational association with lecture included evaluation efforts by departments where lecture was expected. Results confirmed some prior models in different disciplines. 

Teaching observations can be used in multiple ways to describe and assess instruction. We addressed the challenge of measuring instructional change with observational protocols, data that often do not lend themselves easily to statistical comparisons. We first grouped 790 mathematics classes using Latent Profile Analysis and found four reliable categories of classes. Based on the grouping we proposed a proportional measure called Proportion Non-Didactic Lecture (PND). The measure is the proportion of interactive to lecture classes for each instructor. The PND worked in simple hypothesis tests but lacked some statistical power due to possible scaler ceiling effects. The measure correlated highly with a dependent measure derived from the Reformed Teaching Observation Protocol (RTOP), a holistic observational measure. The PND also provided effective descriptions and visualizations of instructional approaches and how these changed from pre to post. 

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.