skip to main content

Title: Evaluating the extent of a large-scale transformation in gateway science courses
We evaluate the impact of an institutional effort to transform undergraduate science courses using an approach based on course assessments. The approach is guided by A Framework for K-12 Science Education and focuses on scientific and engineering practices, crosscutting concepts, and core ideas, together called three-dimensional learning. To evaluate the extent of change, we applied the Three-dimensional Learning Assessment Protocol to 4 years of chemistry, physics, and biology course exams. Changes in exams differed by discipline and even by course, apparently depending on an interplay between departmental culture, course organization, and perceived course ownership, demonstrating the complex nature of transformation in higher education. We conclude that while transformation must be supported at all organizational levels, ultimately, change is controlled by factors at the course and departmental levels.
; ; ; ; ; ; ; ; ; ;
Award ID(s):
1725520 1725395 1725609 1726360
Publication Date:
Journal Name:
Science Advances
Page Range or eLocation-ID:
Sponsoring Org:
National Science Foundation
More Like this
  1. The Department of Electrical and Computer Engineering at a large Midwestern University is seeking to enhance undergraduate engineering education through a combination of programmatic efforts to create departmental change. Three distinct programs aim to transform ECE education through collaborative course design, enhancements to the department climate, and increases in the opportunities for underrepresented undergraduate engineering students. Due to the integrative and corresponding programmatic goals, it was vital to develop a unified evaluation in line with the program evaluation standards (Yarbrough, Shulha, Hopson, & Caruthers, 2011). Further, the interaction of multiple programs necessitated evaluating goal attainment at both the programmatic and departmental levels to determine not only the effects of individual programs but also to examine the broader effect of the interaction of multiple ongoing programmatic efforts to enhance engineering education. To facilitate this process, program team members developed comprehensive lists of ongoing activities designed to create change in the department within each program. Evaluators worked with the program teams to theme and cluster activities into similar groups. To understand how each cluster of activities was positioned to create departmental change and revolutionize engineering education, the evaluators and team members then attempted to identify how each cluster of activities worked asmore »change strategies within the model by Henderson, Beach, and Finkelstein (2011). Thus, evaluators were able to identify over twenty distinct clusters of change activities working as change strategies within the four pillars of the change model: Curriculum and pedagogy, reflective teachers, policy, and shared vision. Positioning activities within this model allowed the evaluators and team members to 1) Better understand the broad scope of departmental activities and change strategies, 2) Identify strengths and challenges associated with their current efforts to transform engineering education within the department, and 3) Develop and integrate ongoing evaluation efforts to further understand both the programmatic and interactive effects of having multiple programs designed at facilitating departmental change and enhancing engineering education. The model for understanding department change and the approaches within that model that are being used to transform ECE education will be presented. We will further explain how the change model approach facilitated evaluating each program and the interactive effects of the combined programmatic efforts within the program evaluation standards of utility, feasibility, propriety, and accuracy (Yarbrough et al., 2011). Specific programmatic and interactive evaluation approaches will be discussed. References Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48(8), 952-984. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.« less
  2. Changing Electrical and Computer Engineering Department Culture from the Bottom Up: Action Plans Generated from Faculty Interviews We prefer a Lessons Learned Paper. In a collaborative effort between a RED: Revolutionizing Engineering and Computer Science Departments (RED) National Science Foundation grant awarded to an electrical and computer engineering department (ECpE) and a broader, university-wide ADVANCE program, ECpE faculty were invited to participate in focus groups to evaluate the culture of their department, to further department goals, and to facilitate long-term planning. Forty-four ECpE faculty members from a large Midwestern university participated in these interviews, which were specifically focused on departmental support and challenges, distribution of resources, faculty workload, career/family balance, mentoring, faculty professional development, productivity, recruitment, and diversity. Faculty were interviewed in groups according to rank, and issues important to particular subcategories of faculty (e.g., rank, gender, etc.) were noted. Data were analyzed by a social scientist using the full transcript of each interview/focus group and the NVivo 12 Qualitative Research Software Program. She presented the written report to the entire faculty. Based on the results of the focus groups, the ECpE department developed an action plan with six main thrusts for improving departmental culture and encouraging departmental change andmore »transformation. 1. Department Interactions – Encourage open dialogue and consider department retreats. Academic areas should be held accountable for the working environment and encouraged to discuss department-related issues. 2. Mentoring, Promotion, and Evaluation – Continue mentoring junior faculty. Improve the clarity of P&T operational documents and seek faculty input on the evaluation system. 3. Teaching Loads – Investigate teaching assistant (TA) allocation models and explore models for teaching loads. Develop a TA performance evaluation system and return TA support to levels seen in the 2010 timeframe. Improvements to teaching evaluations should consider differential workloads, clarifying expectations for senior advising, and hiring more faculty for undergraduate-heavy areas. 4. Diversity, Equity, and Inclusion – Enact an explicit focus on diversity in hiring. Review departmental policies on inclusive teaching and learning environments. 5. Building – Communicate with upper administration about the need for a new building. Explore possibilities for collaborations with Computer Science on a joint building. 6. Support Staff – Increase communication with the department regarding new service delivery models. Request additional support for Human Resources, communications, and finance. Recognize staff excellence at the annual department banquet and through college/university awards.« less
  3. Many university engineering programs require their students to complete a senior capstone experience to equip them with the knowledge and skills they need to succeed after graduation. Such capstone experiences typically integrate knowledge and skills learned cumulatively in the degree program, often engaging students in projects outside of the classroom. As part of an initiative to completely transform the civil engineering undergraduate program at Clemson University, a capstone-like course sequence is being incorporated into the curriculum during the sophomore year. Funded by a grant from the National Science Foundation’s Revolutionizing Engineering Departments (RED) program, this departmental transformation (referred to as the Arch initiative) is aiming to develop a culture of adaptation and a curriculum support for inclusive excellence and innovation to address the complex challenges faced by our society. Just as springers serve as the foundation stones of an arch, the new courses are called “Springers” because they serve as the foundations of the transformed curriculum. The goal of the Springer course sequence is to expose students to the “big picture” of civil engineering while developing student skills in professionalism, communication, and teamwork through real-world projects and hands-on activities. The expectation is that the Springer course sequence will allow facultymore »to better engage students at the beginning of their studies and help them understand how future courses contribute to the overall learning outcomes of a degree in civil engineering. The Springer course sequence is team-taught by faculty from both civil engineering and communication, and exposes students to all of the civil engineering subdisciplines. Through a project-based learning approach, Springer courses mimic capstone in that students work on a practical application of civil engineering concepts throughout the semester in a way that challenges students to incorporate tools that they will build on and use during their junior and senior years. In the 2019 spring semester, a pilot of the first of the Springer courses (Springer 1; n=11) introduced students to three civil engineering subdisciplines: construction management, hydrology, and transportation. The remaining subdisciplines will be covered in a follow-on Springer 2 pilot.. The project for Springer 1 involved designing a small parking lot for a church located adjacent to campus. Following initial instruction in civil engineering topics related to the project, students worked in teams to develop conceptual project designs. A design charrette allowed students to interact with different stakeholders to assess their conceptual designs and incorporate stakeholder input into their final designs. The purpose of this paper is to describe all aspects of the Springer 1 course, including course content, teaching methods, faculty resources, and the design and results of a Student Assessment of Learning Gains (SALG) survey to assess students’ learning outcomes. An overview of the Springer 2 course is also provided. The feedback from the SALG indicated positive attitudes towards course activities and content, and that students found interaction with project stakeholders during the design charrette especially beneficial. Challenges for full scale implementation of the Springer course sequence as a requirement in the transformed curriculum are also discussed.« less
  4. We are seeking to incorporate authentic inquiry into an undergraduate biochemistry lab course. Students on six campuses are combining computational (“in silico”) and wet lab (“in vitro”) techniques as they characterize proteins whose three dimensional structures are known but to which functions have not been previously ascribed. The in silico modules include protein visualization with PyMOL, structural alignment using Dali and ProMOL, sequence exploration with BLAST and Pfam, and ligand docking with PyRX and Autodock Vina. The goal is to predict the function of the protein and to identify the most promising substrates for the active sites. In the wet lab, students express and purify their target proteins, then conduct enzyme kinetics with substrates selected from their docking studies. Their learning as students and their growth as scientists is being assessed in terms of research methods, visualization, biological context, and mechanism of protein function. The lab course is an extension of successful undergraduate research efforts at RIT and Dowling College. The modules that are developed will be disseminated to the scientific community via a web site (, including both protocols and captioned video instruction in the techniques involved. Over the course of the project, we will also be following changesmore »in faculty and teaching assistant competence in two areas: effective teaching with structural biology tools and the development of skills in the area of measuring learning gains by students. As we conduct the lab on these different campuses, we will also focus on advantages of our approach and barriers to implementation that exist on each campus, from the level of student acceptance and faculty training, to resources that are needed to changes in the culture at the departmental and institutional levels. As we analyze the feasibility of this approach on other campuses, we will seek input from other potential adopters about their level of interest and the barriers that they anticipate on their campuses.« less
  5. A solid understanding of electromagnetic (E&M) theory is key to the education of electrical engineering students. However, these concepts are notoriously challenging for students to learn, due to the difficulty in grasping abstract concepts such as the electric force as an invisible force that is acting at a distance, or how electromagnetic radiation is permeating and propagating in space. Building physical intuition to manipulate these abstractions requires means to visualize them in a three-dimensional space. This project involves the development of 3D visualizations of abstract E&M concepts in Virtual Reality (VR), in an immersive, exploratory, and engaging environment. VR provides the means of exploration, to construct visuals and manipulable objects to represent knowledge. This leads to a constructivist way of learning, in the sense that students are allowed to build their own knowledge from meaningful experiences. In addition, the VR labs replace the cost of hands-on labs, by recreating the experiments and experiences on Virtual Reality platforms. The development of the VR labs for E&M courses involves four distinct phases: (I) Lab Design, (II) Experience Design, (III) Software Development, and (IV) User Testing. During phase I, the learning goals and possible outcomes are clearly defined, to provide context for themore »VR laboratory experience, and to identify possible technical constraints pertaining to the specific laboratory exercise. During stage II, the environment (the world) the player (user) will experience is designed, along with the foundational elements, such as ways of navigation, key actions, and immersion elements. During stage III, the software is generated as part of the course projects for the Virtual Reality course taught in the Computer Science Department at the same university, or as part of independent research projects involving engineering students. This reflects the strong educational impact of this project, as it allows students to contribute to the educational experiences of their peers. During phase IV, the VR experiences are played by different types of audiences that fit the player type. The team collects feedback and if needed, implements changes. The pilot VR Lab, introduced as an additional instructional tool for the E&M course during the Fall 2019, engaged over 100 students in the program, where in addition to the regular lectures, students attended one hour per week in the E&M VR lab. Student competencies around conceptual understanding of electromagnetism topics are measured via formative and summative assessments. To evaluate the effectiveness of VR learning, each lab is followed by a 10-minute multiple-choice test, designed to measure conceptual understanding of the various topics, rather than the ability to simply manipulate equations. This paper discusses the implementation and the pedagogy of the Virtual Reality laboratory experiences to visualize concepts in E&M, with examples for specific labs, as well as challenges, and student feedback with the new approach. We will also discuss the integration of the 3D visualizations into lab exercises, and the design of the student assessment tools used to assess the knowledge gain when the VR technology is employed.« less