skip to main content


Title: Changing the Paradigm: Developing a Framework for Secondary Analysis of EER Qualitative Datasets
This paper reports on a project funded through the Engineering Education and Centers (EEC) Division of the National Science Foundation. Since 2010, EEC has funded more than 500 proposals totaling over $150 million through engineering education research (EER) programs such as Research in Engineering Education (REE) and Research in the Formation of Engineers (RFE), to enhance understanding and improve practice. The resulting archive of robust qualitative and quantitative data represents a vast untapped potential to exponentially increase the impact of EEC funding and transform engineering education. But tapping this potential has thus far been an intractable problem, despite ongoing calls for data sharing by public funders of research. Changing the paradigm of single-use data collection requires actionable, proven practices for effective, ethical data sharing, coupled with sufficient incentives to both share and use existing data. To that end, this project draws together a team of experts to overcome substantial obstacles in qualitative data sharing by building a framework to guide secondary analysis in engineering education research (EER), and to test this framework using pioneering data sets. Herein, we report on accomplishments within the first year of the project during which time we gathered a group of 13 expert qualitative researchers to engage in the first of a series of working meetings intended to meet our project goals. We came into this first workshop with a potentially limiting definition of secondary data analysis and the idea that people would want to share existing datasets if we could find ways around anticipated hurdles. However, the workshop yielded a broader definition of secondary data analysis and revealed a stronger interest in creating new datasets designed for sharing rather than sharing existing datasets. Thus, we have reconceived our second phase as one that is a cohesive effort based on an inclusive “open cohort model” to pilot projects related to secondary data analysis.  more » « less
Award ID(s):
2039871
NSF-PAR ID:
10392042
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
2022 ASEE Annual Conference & Exposition
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. In this proposal, we will share some initial findings about how teacher and student engagement in cogenerative dialogues influenced the development of the Culturally Relevant Pedagogical Guidelines for Computational Thinking and Computer Science (CRPG-CSCT). The CRPG-CSCT’s purpose is to provide computer science teachers with tools to enhance their instruction by accurately reflecting students’ diverse cultural resources in the classroom. Additionally, the CRPG-CSCT will provide guidance to non-computer science teachers on how to facilitate the integration of computational thinking skills to a broad spectrum of classes in the arts, humanities, sciences, social sciences, and mathematics. Our initial findings shared here are part of a larger NSF-funded research project (Award No. 2122367) which aims to better understand the barriers to entry and challenges for success faced by underrepresented secondary school students in computer science, through direct engagement with the students themselves. Throughout the 2022-23 academic year, the researchers have been working with a small team of secondary school teachers, students, and instructional designers, as well as university faculty in computer science, secondary education, and sociology to develop the CRPG-CSCT. The CRPG-CSCT is rooted in the tenets of culturally relevant pedagogy (Ladson-Billings, 1995) and borrows from Muhammad’s (2020) work in Cultivating Genius: An Equity Framework for Culturally and Historically Responsive Literacy. The CRPG-CCT is being developed over six day-long workshops held throughout the academic year. At the time of this submission, five of the six workshops had been completed. Each workshop utilized cogenerative dialogues (cogens) as the primary tool for organizing and sustaining participants’ engagement. Through cogens, participants more deeply learn about students’ cultural capital and the value of utilizing that capital within the classroom (Roth, Lawless, & Tobin, 2000). The success of cogens relies on following specific protocols (Emdin, 2016), such as listening attentively, ensuring there are equal opportunities for all participants to share, and affirming the experiences of other participants. The goal of a cogen is to reach a collective decision, based on the dialogue, that will positively impact students by explicitly addressing barriers to their engagement in the classroom. During each workshop, one member of the research team and one undergraduate research assistant observed the interactions among cogen participants and documented these in the form of ethnographic field notes. Another undergraduate research assistant took detailed notes during the workshop to record the content of small and large group discussions, presentations, and questions/responses throughout the workshops. A grounded theory approach was used to analyze the field notes. Additionally, at the conclusion of each workshop, participants completed a Cogen Feedback Survey (CFS) to gather additional information. The CFS were analyzed through open thematic coding, memos, and code frequencies. Our preliminary results demonstrate high levels of engagement from teacher and student participants during the workshops. Students identified that the cogen structure allowed them to participate comfortably, openly, and honestly. Further, students described feeling valued and heard. Students’ ideas and experiences were frequently affirmed, which served as an important step toward dismantling traditional teacher-student boundaries that might otherwise prevent them from sharing freely. Another result from the use of cogens was the shared experience of participants comprehending views from the other group’s perspective in the classroom. Students appreciated the opportunity to learn from teachers about their struggles in keeping students engaged. Teachers appreciated the opportunity to better understand students’ schooling experiences and how these may affirm or deny aspects of their identity. Finally, all participants shared meaningful suggestions and strategies for future workshops and for the collective betterment of the group. Initial findings shared here are important for several reasons. First, our findings suggest that cogens are an effective approach for fostering participants’ commitment to creating the conditions for students’ success in the classroom. Within the context of the workshops, cogens provided teachers, students, and faculty with opportunities to engage in authentic conversations for addressing the recruitment and retention problems in computer science for underrepresented students. These conversations often resulted in the development of tangible pedagogical approaches, examples, metaphors, and other strategies to directly address the recruitment and retention of underrepresented students in computer science. Finally, while we are still developing the CRPG-CSCT, cogens provided us with the opportunity to ensure the voices of teachers and students are well represented in and central to the document. 
    more » « less
  2. Previous studies have convincingly shown that traditional, content-centered, and didactic teaching methods are not effective for developing a deep understanding and knowledge transfer. Nor does it adequately address the development of critical problem-solving skills. Active and collaborative instruction, coupled with effective means to encourage student engagement, invariably leads to better student learning outcomes irrespective of academic discipline. Despite these findings, the existing construction engineering programs, for the most part, consist of a series of fragmented courses that mainly focus on procedural skills rather than on the fundamental and conceptual knowledge that helps students become innovative problem-solvers. In addition, these courses are heavily dependent on traditional lecture-based teaching methods focused on well-structured and closed-ended problems that prepare students to plug variables into equations to get the answer. Existing programs rarely offer a systematic approach to allow students to develop a deep understanding of the engineering core concepts and discover systematic solutions for fundamental problems. Without properly understanding these core concepts, contextualized in domain-specific settings, students are not able to develop a holistic view that will help them to recognize the big picture and think outside the box to come up with creative solutions for arising problems. The long history of empirical learning in the field of construction engineering shows the significant potential of cognitive development through direct experience and reflection on what works in particular situations. Of course, the complex nature of the construction industry in the twenty-first century cannot afford an education through trial and error in the real environment. However, recent advances in computer science can help educators develop virtual environments and gamification platforms that allow students to explore various scenarios and learn from their experiences. This study aims to address this need by assessing the effectiveness of guided active exploration in a digital game environment on students’ ability to discover systematic solutions for fundamental problems in construction engineering. To address this objective, through a research project funded by the NSF Division of Engineering Education and Centers (EEC), we designed and developed a scenario-based interactive digital game, called Zebel, to guide students solve fundamental problems in construction scheduling. The proposed gamified pedagogical approach was designed based on the Constructivism learning theory and a framework that consists of six essential elements: (1) modeling; (2) reflection; (3) strategy formation; (4) scaffolded exploration; (5) debriefing; and (6) articulation. We also designed a series of pre- and post-assessment instruments for empirical data collection to assess the effectiveness of the proposed approach. The proposed gamified method was implemented in a graduate-level construction planning and scheduling course. The outcomes indicated that students with no prior knowledge of construction scheduling methods were able to discover systematic solutions for fundamental scheduling problems through their experience with the proposed gamified learning method. 
    more » « less
  3. null (Ed.)
    Our work with teams funded through the National Science Foundation REvolutionizing Engineering and Computer Science Departments (RED) program began in 2015. Our project—funded first by a NSF EAGER grant, and then by a NSF RFE grant—focuses on understanding how the RED teams make change on their campuses and how this information about change can be captured and communicated to other STEM programs that seek to make change happen. Because our RED Participatory Action Research (REDPAR) Project is a collaboration between researchers (Center for Evaluation & Research for STEM Equity at the University of Washington) and practitioners (Making Academic Change Happen Workshop at Rose-Hulman Institute of Technology), we have challenged ourselves to develop means of communication that allow for both aspects of the work—both research and practice—to be treated equitably. As a result, we have created a new dissemination channel—the RED Participatory Action Project Tipsheet. The tipsheet format accomplishes several important goals. First, the content is drawn from both the research conducted with the RED teams and the practitioners’ work with the teams. Each tipsheet takes up a single theme and grounds the theme in the research literature while offering practical tips for applying the information. Second, the format is accessible to a wide spectrum of potential users, remaining free of jargon and applicable to multiple program and departmental contexts. Third, by publishing the tipsheets ourselves, rather than submitting them to an engineering education research journal, we make the information timely and freely available. We can make a tipsheet as soon as a theme emerges from the intersection of research data and observations of practice. During the poster session at ASEE 2019, we will share the three REDPAR Tipsheets that have been produced thus far: Creating Strategic Partnerships, Communicating Change, and Shared Vision. We will also work with attendees to demonstrate how the tipsheet content is adaptable to the attendees’ specific academic context. Our goal for the poster session is to provide attendees with tipsheet resources that are useful to their specific change project. 
    more » « less
  4. Our work with teams funded through the National Science Foundation REvolutionizing Engineering and Computer Science Departments (RED) program began in 2015. Our project—funded first by a NSF EAGER grant, and then by a NSF RFE grant—focuses on understanding how the RED teams make change on their campuses and how this information about change can be captured and communicated to other STEM programs that seek to make change happen. Because our RED Participatory Action Research (REDPAR) Project is a collaboration between researchers (Center for Evaluation & Research for STEM Equity at the University of Washington) and practitioners (Making Academic Change Happen Workshop at Rose-Hulman Institute of Technology), we have challenged ourselves to develop means of communication that allow for both aspects of the work—both research and practice—to be treated equitably. As a result, we have created a new dissemination channel—the RED Participatory Action Project Tipsheet. The tipsheet format accomplishes several important goals. First, the content is drawn from both the research conducted with the RED teams and the practitioners’ work with the teams. Each tipsheet takes up a single theme and grounds the theme in the research literature while offering practical tips for applying the information. Second, the format is accessible to a wide spectrum of potential users, remaining free of jargon and applicable to multiple program and departmental contexts. Third, by publishing the tipsheets ourselves, rather than submitting them to an engineering education research journal, we make the information timely and freely available. We can make a tipsheet as soon as a theme emerges from the intersection of research data and observations of practice. During the poster session at ASEE 2019, we will share the three REDPAR Tipsheets that have been produced thus far: Creating Strategic Partnerships, Communicating Change, and Shared Vision. We will also work with attendees to demonstrate how the tipsheet content is adaptable to the attendees’ specific academic context. Our goal for the poster session is to provide attendees with tipsheet resources that are useful to their specific change project. 
    more » « less
  5. Our NSF funded project—Creating National Leadership Cohorts to Make Academic Change Happen (NSF 1649318)—represents a strategic partnership between researchers and practitioners in the domain of academic change. The principle investigators from the Making Academic Change Happen team from Rose-Hulman Institute of Technology provide familiarity with the literature of practical organizational change and package this into action-oriented workshops and ongoing support for teams funded through the REvolutionizing engineering and computer science Departments (RED) program. The PIs from the Center for Evaluation & Research for STEM Equity at the University of Washington provide expertise in social science research in order to investigate how the the RED teams’ change projects unfold and how the teams develop as members of national leadership cohorts for change in engineering and computer science education. Our poster for ASEE 2018 will focus on what we have learned thus far regarding the dynamics of the researcher/practitioner partnership through the RED Participatory Action Research (REDPAR) Project. According to Worrall (2007), good partnerships are “founded on trust, respect, mutual benefit, good communities, and governance structures that allow democratic decision-making, process improvement, and resource sharing.” We have seen these elements emerge through the work of the partnership to create mutual benefits. For example, the researchers have been given an “insider’s” perspective on the practitioners’ approach—their goals, motivations for certain activities, and background information and research. The practitioners’ perspective is useful for the researchers to learn since the practitioners’ familiarity with the organizational change literature has influenced the researchers’ questions and theoretical models. The practitioners’ work with the RED teams has provided insights on the teams, how they are operating, the challenges they face, and aspects of the teams’ work that may not be readily available to the researchers. As a result, the researchers have had increased access to the teams to collect data. The researchers, in turn, have been able to consider how to make their analyses useful and actionable for change-makers, the population that the practitioners are more familiar with. Insights from the researchers provide both immediate and long-term benefits to programming and increased professional impact. The researchers are trained observers, each of whom brings a unique disciplinary perspective to their observations. The richness, depth, and clarity of their observations adds immeasurably to the quality of practitioners’ interactions with the RED teams. The practitioners, for example, have revised workshop content in response to the researchers’ observations, thus ensuring that the workshop content serves the needs of the RED teams. The practitioners also benefit from the joint effort on dissemination, since they can contribute to a variety of dissemination efforts (journal papers, conference presentations, workshops). We plan to share specific examples of the strategic partnership during the poster session. In doing so, we hope to encourage researchers to seek out partnerships with practitioners in order to bridge the gap between theory and practice in engineering and computer science education. 
    more » « less