skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
Title: When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching
Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.  more » « less
Award ID(s):
1821704 1525077 1245436
PAR ID:
10283746
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
American Journal of Evaluation
ISSN:
1098-2140
Page Range / eLocation ID:
109821402093194
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; (, Numeracy)
    Grawe, Nathan D (Ed.)
    Many educators and professional organizations recommend Quantitative Reasoning as the best entry-level postsecondary mathematics course for non-STEM majors. However, novice and veteran instructors who have no prior experience in teaching a QR course often express their ignorance of the content to choose for this course, the instruction to offer students, and the assessments to measure student learning. We conducted a case study to investigate the initial implementation of an entry-level university quantitative reasoning course during fall semester, 2018. The participants were the course instructor and students. We examined the instructor’s motives and actions and the students’ responses to the course. The instructor had no prior experience teaching a QR course but did have 15 years of experience teaching student-centered mathematics. Data included course artifacts, class observations, an instructor interview, and students’ written reflections. Because this was a new course—and to adapt to student needs—the instructor employed his instructional autonomy and remained flexible in designing and enacting the course content, instruction, and assessment. His instructional decision making and flexible approach helped the instructor tailor the learning activities and teaching practices to the needs and interests of the students. The students generally appreciated and benefited from this approach, enjoyed the course, and provided positive remarks about the instructors’ practices. 
    more » « less
  2. ; ; ; ; ; ; ; ; ; ; et al (, Journal of Microbiology & Biology Education)
    Santillan, Karen A (Ed.)
    ABSTRACT Graduate teaching assistants (GTAs) teach undergraduates directly, and many are future faculty, making it particularly important for them to be trained in using evidence-based instructional practices. We implemented and assessed a teaching professional development program for 19 biology GTAs aimed to help them develop their teaching perspectives and improve their teaching self-efficacy. The program consisted of a pre-semester bootcamp of pedagogical workshops, mentoring sessions throughout the semester, crafting a teaching philosophy statement, and reflective peer teaching observation. We surveyed and interviewed students throughout the program to assess their growth and identify elements of the program that supported their growth. We found that participants’ self-efficacy improved immediately following the pre-semester bootcamp, but these improvements did not persist throughout the semester. At the end of the semester, participants’ teaching self-efficacy did not differ from GTAs in the department who did not participate in our program. Throughout the semester, our participants shifted toward the social reform perspective of teaching, which views good teaching as encouraging students to critically evaluate information and give them power to take social action to improve their lives. At the end of the semester, our participants more strongly endorsed the social reform perspective of teaching than GTAs who did not participate in our program. Our results suggest that pre-semester workshops supported novice GTAs before their class, but more sustained interactions may be needed for these boosts to persist. Our results also suggest that encouraging GTAs to reflect on what makes good teaching can help them solidify a coherent teaching perspective. 
    more » « less
  3. ; ; (, Institute for Scientist & Engineer Educators (ISEE))
    Seagroves, Scott; Barnes, Austin; Metevier, Anne; Porter, Jason; Hunter, Lisa (Ed.)
    We designed, facilitated, and re-designed an inquiry activity in an introductory undergraduate astronomy research methods course at the University of Texas at Austin over two different semesters. The teaching venue for this inquiry activity took place in the course “AST 376R: A Practical Introduction to Research Methods”, the inquiry activity was inserted into an existing course structure, taking place over multiple class periods. We discuss how we were able to leverage the Professional Development Program (PDP) inquiry themes and introduce students to specific STEM practices, using this experience as a primer or mini version of a larger research activity and research experience that they would determine and lead themselves later on in the semester. In this paper we describe the benefits for students in this course and the lessons learned by the instructors. 
    more » « less
  4. ; ; ; ; ; (, BioScience)
    null (Ed.)
    Abstract Students are more likely to learn in college science, technology, engineering, and math (STEM) classrooms when instructors use teacher discourse moves (TDMs) that encourage student engagement and learning. However, although teaching practices are well studied, TDMs are not well understood in college STEM classrooms. In STEM courses at a minority-serving institution (MSI; n = 74), we used two classroom observation protocols to investigate teaching practices and TDMs across disciplines, instructor types, years of teaching experience, and class size. We found that instructors guide students in active learning activities, but they use authoritative discourse approaches. In addition, chemistry instructors presented more than biology instructors. Also, teaching faculty had relatively high dialogic, interactive discourse, and neither years of faculty teaching experience nor class size had an impact on teaching practices or TDMs. Our results have implications for targeted teaching professional development efforts across instructor and course characteristics to improve STEM education at MSIs. 
    more » « less
  5. ; ; ; ; ; ; ; ; ; (, CBE—Life Sciences Education)
    The instructional practices used in introductory college courses often differ dramatically from those used in high school courses, and dissatisfaction with these practices is cited by students as a prominent reason for leaving science, technology, engineering, and mathematics (STEM) majors. To better characterize the transition to college course work, we investigated the extent to which incoming expectations of course activities differ based on student demographic characteristics, as well as how these expectations align with what students will experience. We surveyed more than 1500 undergraduate students in large introductory STEM courses at three research-intensive institutions during the first week of classes about their expectations regarding how class time would be spent in their courses. We found that first-generation and first-semester students predict less lecture than their peers and that class size had the largest effect on student predictions. We also collected classroom observation data from the courses and found that students generally underpredicted the amount of lecture observed in class. This misalignment between student predictions and experiences, especially for first-generation and first-semester college students and students enrolled in large- and medium-size classes, has implications for instructors and universities as they design curricula for introductory STEM courses with explicit retention goals. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email