skip to main content


Title: Guides to Advance Teaching Evaluation (GATEs): A Resource for STEM Departments Planning Robust and Equitable Evaluation Practices
Most science, technology, engineering, and mathematics (STEM) departments inadequately evaluate teaching, which means they are not equipped to recognize or reward effective teaching. As part of a project at one institution, we observed that departmental chairs needed help recognizing the decisions they would need to make to improve teaching evaluation practices. To meet this need, we developed the Guides to Advance Teaching Evaluation (GATEs), using an iterative development process. The GATEs are designed to be a planning tool that outlines concrete goals to guide reform in teaching evaluation practices in STEM departments at research-intensive institutions. The GATEs are grounded in the available scholarly literature and guided by existing reform efforts and have been vetted with STEM departmental chairs. The GATEs steer departments to draw on three voices to evaluate teaching: trained peers, students, and the instructor. This research-based resource includes three components for each voice: 1) a list of departmental target practices to serve as goals; 2) a characterization of common starting places to prompt reflection; and 3) ideas for getting started. We provide anecdotal examples of potential uses of the GATEs for reform efforts in STEM departments and as a research tool to document departmental practices at different time points.  more » « less
Award ID(s):
1821023
NSF-PAR ID:
10366123
Author(s) / Creator(s):
; ; ; ; ;
Editor(s):
Bauerle, Cynthia
Date Published:
Journal Name:
CBE—Life Sciences Education
Volume:
21
Issue:
3
ISSN:
1931-7913
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. A 2019 report from the National Academies on Minority Serving Institutions (MSIs) concluded that MSIs need to change their culture to successfully serve students with marginalized racial and/or ethnic identities. The report recommends institutional responsiveness to meet students “where they are,” metaphorically, creating supportive campus environments and providing tailored academic and social support structures. In recent years, the faculty, staff, and administrators at California State University, Los Angeles have made significant efforts to enhance student success through multiple initiatives including a summer bridge program, first-year in engineering program, etc. However, it has become clear that more profound changes are needed to create a culture that meets students “where they are.” In 2020, we were awarded NSF support for Eco-STEM, an initiative designed to change a system that demands "college-ready" students into one that is "student-ready." Aimed at shifting the deficit mindset prevailing in engineering education, the Eco-STEM project embraces an asset-based ecosystem model that thinks of education as cultivation, and ideas as seeds we are planting, rather than a system of standards and quality checks. This significant paradigm and culture transformation is accomplished through: 1) The Eco-STEM Faculty Fellows’ Community of Practice (CoP), which employs critically reflective dialogue[ ][ ] to enhance the learning environment using asset-based learner-centered instructional approaches; 2) A Leadership CoP with department chairs and program directors that guides cultural change at the department/program level; 3) A Facilitators’ CoP that prepares facilitators to lead, sustain, update, and expand the Faculty and Leadership CoPs; 4) Reform of the teaching evaluation system to sustain the cultural changes. This paper presents the progress and preliminary findings of the Eco-STEM project. During the first project year, the project team formulated the curriculum for the Faculty CoP with a focus on inclusive pedagogy, community cultural wealth, and community building, developed a classroom peer observation tool to provide formative data for teaching reflection, and designed research inquiry tools. The latter investigates the following research questions: 1) To what extent do the Eco-STEM CoPs effectively shift the mental models of participants from a factory-like model to an ecosystem model of education? 2) To what extent does this shift support an emphasis on the assets of our students, faculty, and staff members and, in turn, allow for enhanced motivation, excellence and success? 3) To what extent do new faculty assessment tools designed to provide feedback that reflects ecosystem-centric principles and values allow for individuals within the system to thrive? In Fall 2021, the first cohort of Eco-STEM Faculty Fellows were recruited, and rich conversations and in-depth reflections in our CoP meetings indicated Fellows’ positive responses to both the CoP curriculum and facilitation practices. This paper offers a work-in-progress introduction to the Eco-STEM project, including the Faculty CoP, the classroom peer observation tool, and the proposed research instruments. We hope this work will cultivate broader conversations within the engineering education research community about cultural change in engineering education and methods towards its implementation. 
    more » « less
  2. The Department of Electrical and Computer Engineering at a large Midwestern University is seeking to enhance undergraduate engineering education through a combination of programmatic efforts to create departmental change. Three distinct programs aim to transform ECE education through collaborative course design, enhancements to the department climate, and increases in the opportunities for underrepresented undergraduate engineering students. Due to the integrative and corresponding programmatic goals, it was vital to develop a unified evaluation in line with the program evaluation standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011). Further, the interaction of multiple programs necessitated evaluating goal attainment at both the programmatic and departmental levels to determine not only the effects of individual programs but also to examine the broader effect of the interaction of multiple ongoing programmatic efforts to enhance engineering education. To facilitate this process, program team members developed comprehensive lists of ongoing activities designed to create change in the department within each program. Evaluators worked with the program teams to theme and cluster activities into similar groups. To understand how each cluster of activities was positioned to create departmental change and revolutionize engineering education, the evaluators and team members then attempted to identify how each cluster of activities worked as change strategies within the model by Henderson, Beach, and Finkelstein (2011). Thus, evaluators were able to identify over twenty distinct clusters of change activities working as change strategies within the four pillars of the change model: Curriculum and pedagogy, reflective teachers, policy, and shared vision. Positioning activities within this model allowed the evaluators and team members to 1) Better understand the broad scope of departmental activities and change strategies, 2) Identify strengths and challenges associated with their current efforts to transform engineering education within the department, and 3) Develop and integrate ongoing evaluation efforts to further understand both the programmatic and interactive effects of having multiple programs designed at facilitating departmental change and enhancing engineering education. The model for understanding department change and the approaches within that model that are being used to transform ECE education will be presented. We will further explain how the change model approach facilitated evaluating each program and the interactive effects of the combined programmatic efforts within the program evaluation standards of utility, feasibility, propriety, and accuracy (Yarbrough et al., 2011). Specific programmatic and interactive evaluation approaches will be discussed. References Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48(8), 952-984. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage. 
    more » « less
  3. Abstract

    Instructional reform in STEM aims for the widespread adoption of evidence based instructional practices (EBIPS), practices that implement active learning. Research recognizes that faculty social networks regarding discussion or advice about teaching may matter to such efforts. But teaching is not the only priority for university faculty – meeting research expectations is at least as important and, often, more consequential for tenure and promotion decisions. We see value in understanding how research networks, based on discussion and advice about research matters, relate to teaching networks to see if and how such networks could advance instructional reform efforts. Our research examines data from three departments (biology, chemistry, and geosciences) at three universities that had recently received funding to enhance adoption of EBIPs in STEM fields. We evaluate exponential random graph models of the teaching network and find that (a) the existence of a research tie from one faculty member$$i$$ito another$$j$$jenhances the prospects of a teaching tie from$$i$$ito$$j$$j, but (b) even though faculty highly placed in the teaching network are more likely to be extensive EBIP users, faculty highly placed in the research network are not, dimming prospects for leveraging research networks to advance STEM instructional reforms.

     
    more » « less
  4. Changing Electrical and Computer Engineering Department Culture from the Bottom Up: Action Plans Generated from Faculty Interviews We prefer a Lessons Learned Paper. In a collaborative effort between a RED: Revolutionizing Engineering and Computer Science Departments (RED) National Science Foundation grant awarded to an electrical and computer engineering department (ECpE) and a broader, university-wide ADVANCE program, ECpE faculty were invited to participate in focus groups to evaluate the culture of their department, to further department goals, and to facilitate long-term planning. Forty-four ECpE faculty members from a large Midwestern university participated in these interviews, which were specifically focused on departmental support and challenges, distribution of resources, faculty workload, career/family balance, mentoring, faculty professional development, productivity, recruitment, and diversity. Faculty were interviewed in groups according to rank, and issues important to particular subcategories of faculty (e.g., rank, gender, etc.) were noted. Data were analyzed by a social scientist using the full transcript of each interview/focus group and the NVivo 12 Qualitative Research Software Program. She presented the written report to the entire faculty. Based on the results of the focus groups, the ECpE department developed an action plan with six main thrusts for improving departmental culture and encouraging departmental change and transformation. 1. Department Interactions – Encourage open dialogue and consider department retreats. Academic areas should be held accountable for the working environment and encouraged to discuss department-related issues. 2. Mentoring, Promotion, and Evaluation – Continue mentoring junior faculty. Improve the clarity of P&T operational documents and seek faculty input on the evaluation system. 3. Teaching Loads – Investigate teaching assistant (TA) allocation models and explore models for teaching loads. Develop a TA performance evaluation system and return TA support to levels seen in the 2010 timeframe. Improvements to teaching evaluations should consider differential workloads, clarifying expectations for senior advising, and hiring more faculty for undergraduate-heavy areas. 4. Diversity, Equity, and Inclusion – Enact an explicit focus on diversity in hiring. Review departmental policies on inclusive teaching and learning environments. 5. Building – Communicate with upper administration about the need for a new building. Explore possibilities for collaborations with Computer Science on a joint building. 6. Support Staff – Increase communication with the department regarding new service delivery models. Request additional support for Human Resources, communications, and finance. Recognize staff excellence at the annual department banquet and through college/university awards. 
    more » « less
  5. Student success in educational ecosystems is a primary goal of leadership efforts. Yet, power and privilege affect the racial, classist, and gendered implications of STEM education work in K-12 education as well as higher education. Interventions have been done at various levels, but despite the hard work of implementation, this has not resulted in dramatic improvements to STEM educational ecosystems or student engagement within them. Often, these implementations are done at the faculty/student level or institutional level but not at the departmental leadership level. The NSF-supported Eco-STEM Project proposes to establish a healthy educational ecosystem that supports all individuals (students, faculty, and staff) to thrive. Project activities are guided by ecosystem paradigm measures that support a culturally responsive learning/working environment; make teaching and learning rewarding and fulfilling; and emphasize community assets to enhance motivation, excellence, and success. For this work-in-progress paper, we describe the development of a leadership community of practice, comprised of department chairs of science and engineering departments, at [university name redacted], a large state-funded comprehensive majority minority master’s granting institution in the Southwest United States. In the year-long Leadership Community of Practice (L-CoP), the Fellows work on unpacking issues of power and privilege in their roles as STEM leaders and educators. During the Fall semester of 2022, the Fellows participated in four sessions. They engaged in readings, videos, active-learning activities, and critically reflective dialogues to facilitate discussion and reflection on identity, agency, the culture of power in STEM, and interventions and change in higher education. The L-CoP starts with Fellows reflecting on their social and professional identities and how their identities influence their teaching and leadership philosophies. Then Fellows are introduced to the framework of the culture of power in science--where they explore the social, cultural, and political impacts of preparing for a STEM college education. Finally, they explore theories and models of change for STEM higher education spaces. Through this curriculum, we aim to examine mental models to deconstruct notions that uphold the culture of power in science by instead building counternarratives with faculty and students in their departments. Through dialogues within the L-CoP, leaders discuss classroom/program climate, structure, and vibrancy to better support healthy educational ecosystems, as well as their participation in these systems. We are currently in the middle of our first implementation of the L-CoP. The first cohort consists of six L-CoP Fellows with highly diverse positionalities; there is racial, ethnic, and gender diversity, and all Fellows are full professors in the tenure line and chairs of their respective departments. We present details of the L-CoP, including the formation of the Fellow cohort, training of the facilitators, structure of the sessions, and initial results of our mid-program survey. The survey results provide insights into potential improvements to our tools and program. We also share some of the Fellows’ and facilitators’ reflections demonstrating a shift toward an ecosystem mindset. We prefer to present this work as a poster at the 2023 ASEE Annual Conference. 
    more » « less