More Like this

1. Is an Effective Team an Equitable Team? Protocol for a Scoping Review

   https://doi.org/10.1109/FIE56618.2022.9962452  

   Moffat, Andrew D. ; Fowler, Robin ; Matz, Rebecca L. ; Jeffrey, Madison ( October 2022 , Frontiers in Education)

   This full paper sets out a methodological protocol for conducting a scoping review of literature relating to teamwork effectiveness and equity. The goal of the study is to understand how academic discourse over the five-year period prior to the study being carried out has conceptualized teamwork success in educational and professional contexts, and to what extent equitable team practices are embedded within such conceptualizations. In line with ongoing initiatives to promote transparency in research, this protocol paper is intended for dissemination prior to the conduct of the study itself. The research context, questions, and rationale are set out, and a detailed methodology described, outlining procedures for data retrieval, screening, extraction, and analysis. The paper concludes with an outline of intended reporting methods for the study, including the reporting of deviations from the procedures set out herein. This paper contributes to the scoping review methodology, and especially its application in the field of engineering education research and education research more broadly. 

   more » « less
2. Methods for Conducting a Scoping Literature Review on Institutional Culture and Transformational Change in Engineering Education

   Fernandez, K. ; Buhler, A. G. ; Rivera-Jimenez, SM ( June 2023 , American Society for Engineering Education, 2023)

   There are a variety of urgent calls for institutional initiatives and actions to transform engineering education. For a transformational change to occur, the initiatives must alter the culture of the institutions (Eckel, Hill, and Green, 1998). In this work in progress, we detail the methods used to conduct a scoping literature review (ScR) concerning the current state of the literature surrounding institutional culture and transformational change in engineering education at institutions of higher learning in the United States. As institutional culture and transformational change are currently underexplored topics in the engineering education literature, we investigated the larger body of computer science and engineering literature in the United States. Once completed, this study aims to reveal the current trends, theories, and potential gaps in the literature regarding these topics. Arksey and O’Malley’s methodology for conducting scoping reviews informed the development of our scoping review protocol, which similarly includes five stages: (1) identify the research questions, (2) identify relevant studies, (3) select relevant studies, (4) chart the data, and (5) collate, summarize, and report results (Arksey and O’Malley, 2005). University librarians who specialize in conducting systematic reviews aided in the refinement of this protocol. From the research question and aim of the study, three main inclusion criteria were created: (1) the literature must discuss both organizational culture and transformational change, (2) discussion of transformational change must describe the institution where the change happened, and (3) the literature must emphasize the agents of transformational change. Additional inclusion and exclusion criteria were created in collaboration with both the librarians and reviewers. These criteria guided the search for existing literature in the following online databases: Elsevier (Engineering Village – Compendex and Engineering Village – INSPEC), ProQuest (ERIC and Education Database), Scopus, and Web of Science. These six databases were selected as they often include publications relevant to the field of engineering education. After the search was conducted, the inclusion and exclusion criteria were turned into questions to inform a three-step screening process (title, abstract, and full text) used by reviewers to determine whether a publication was eligible for the study. Reviewers were assigned to review papers through Covidence, a cloud-based systematic literature review management platform. There are currently two primary reviewers and a third additional reviewer to resolve any conflicts or disagreements if they should arise. Before each review cycle, the inclusion and exclusion criteria are revisited, revised, and agreed upon by the three reviewers. This screening process is performed iteratively, allowing for critical reflection at each stage to drive the resulting findings by the reviewers in consultation with content matter experts. We are currently conducting our first round of screening in the study selection (third stage) of the scoping review protocol. After the removal of duplicates, 999 publications were found by searching in the six selected databases. This number is expected to be further reduced with each step of the screening process. When this scoping review is complete, the resulting publication will contain an analysis of the literature and synthesis of our findings, and present the prominent themes, theories, and potential gaps in the literature. This publication is expected to unite disparate lines of research on institutional culture and transformational change, challenge the assumptions in the field, and change the way engineering education views transformational change. 

   more » « less
3. ABET & Engineering Accreditation - History, Theory, Practice: Initial Findings from a National Study on the Governance of Engineering Education

   Akera, A. ( January 2019 , ASEE Annual Conference proceedings)

   Who and by what means do we ensure that engineering education evolves to meet the ever changing needs of our society? This and other papers presented by our research team at this conference offer our initial set of findings from an NSF sponsored collaborative study on engineering education reform. Organized around the notion of higher education governance and the practice of educational reform, our open-ended study is based on conducting semi-structured interviews at over three dozen universities and engineering professional societies and organizations, along with a handful of scholars engaged in engineering education research. Organized as a multi-site, multi-scale study, our goal is to document differences in perspectives and interest the exist across organizational levels and institutions, and to describe the coordination that occurs (or fails to occur) in engineering education given the distributed structure of the engineering profession. This paper offers for all engineering educators and administrators a qualitative and retrospective analysis of ABET EC 2000 and its implementation. The paper opens with a historical background on the Engineers Council for Professional Development (ECPD) and engineering accreditation; the rise of quantitative standards during the 1950s as a result of the push to implement an engineering science curriculum appropriate to the Cold War era; EC 2000 and its call for greater emphasis on professional skill sets amidst concerns about US manufacturing productivity and national competitiveness; the development of outcomes assessment and its implementation; and the successive negotiations about assessment practice and the training of both of program evaluators and assessment coordinators for the degree programs undergoing evaluation. It was these negotiations and the evolving practice of assessment that resulted in the latest set of changes in ABET engineering accreditation criteria (“1-7” versus “a-k”). To provide an insight into the origins of EC 2000, the “Gang of Six,” consisting of a group of individuals loyal to ABET who used the pressure exerted by external organizations, along with a shared rhetoric of national competitiveness to forge a common vision organized around the expanded emphasis on professional skill sets. It was also significant that the Gang of Six was aware of the fact that the regional accreditation agencies were already contemplating a shift towards outcomes assessment; several also had a background in industrial engineering. However, this resulted in an assessment protocol for EC 2000 that remained ambiguous about whether the stated learning outcomes (Criterion 3) was something faculty had to demonstrate for all of their students, or whether EC 2000’s main emphasis was continuous improvement. When it proved difficult to demonstrate learning outcomes on the part of all students, ABET itself began to place greater emphasis on total quality management and continuous process improvement (TQM/CPI). This gave institutions an opening to begin using increasingly limited and proximate measures for the “a-k” student outcomes as evidence of effort and improvement. In what social scientific terms would be described as “tactical” resistance to perceived oppressive structures, this enabled ABET coordinators and the faculty in charge of degree programs, many of whom had their own internal improvement processes, to begin referring to the a-k criteria as “difficult to achieve” and “ambiguous,” which they sometimes were. Inconsistencies in evaluation outcomes enabled those most discontented with the a-k student outcomes to use ABET’s own organizational processes to drive the latest revisions to EAC accreditation criteria, although the organization’s own process for member and stakeholder input ultimately restored much of the professional skill sets found in the original EC 2000 criteria. Other refinements were also made to the standard, including a new emphasis on diversity. This said, many within our interview population believe that EC 2000 had already achieved much of the changes it set out to achieve, especially with regards to broader professional skills such as communication, teamwork, and design. Regular faculty review of curricula is now also a more routine part of the engineering education landscape. While programs vary in their engagement with ABET, there are many who are skeptical about whether the new criteria will produce further improvements to their programs, with many arguing that their own internal processes are now the primary drivers for change. 

   more » « less
4. Building Your Dream Team for Change

   https://doi.org/10.18260/1-2--32489  

   Margherio, Cara ; Doten-Snitker, Kerice ; Litzler, Elizabeth ; Williams, Julia ; Andrijcic, Eva ; Mohan, Sriram ( June 2020 , 2019 ASEE Virtual Annual Conference Content Access)

   null (Ed.)

   his panel paper presents research on connecting theory to practice and the lessons learned in a change project, with a focus on team formation during the early stages of change making. An important yet often overlooked step in any change project is pulling together individuals to form a competent and efficient team. A functional change-making team requires a variety of complementary skill sets, which may come from different disciplinary backgrounds and/or different prior experiences. Kotter (1996) uses the term “guiding coalition” to refer to an effective change-making team. He identifies four key characteristics of guiding coalitions: position power, expertise, credibility, leadership. Kotter also goes on to examine the importance of trust and a common goal. In a review of the literature on guiding coalitions, Have, Have, Huijsmans, and Otto (2017) found that though the concept of a guiding coalition is widely advocated in the literature, only one study showed a moderate correlation between the existence of a guiding coalition and the success of a change process (Abraham, Griffin, & Crawford, 1999). Have et al. (2017) conclude that while the literature provides little evidence to the value of a guiding coalition, it does provide evidence that Kotter’s characteristics of a guiding coalition (position power, expertise, credibility, leadership skills, trust in leadership, and setting common goals) individually have positive effects on the outcomes of a change project. However, we don’t know how these characteristics interact. This analysis of team building and complementary skill sets emerges from our participatory action research with the NSF REvolutionizing engineering and computer science Departments (RED) teams to investigate the change process within STEM higher education. The research-to-practice cycle is integral to our project; data gathered through working with the RED teams provides insights that are then translated into applied, hands-on practices. We utilize an abductive analysis approach, a qualitative methodology that moves recursively between the data and theory-building to remain open to new or contradictory findings, keeping existing theory in mind while not developing formal hypotheses (Timmermans & Tavory, 2012). We find that many of the teams have learned lessons in the early stages of the change process around the guiding coalition characteristics, and our analysis builds on the literature by examining how these characteristics interact. For example, the expertise of the social scientists and education researchers help discern which change strategies have supporting evidence and fit the context, in addition to what is reasonable for planning, implementation, and evaluation. The results presented in this paper connect theory to practice, clarifying practices for building effective change-making teams within higher education. 

   more » « less
5. An Inquiry into the Use of Intercoder Reliability Measures in Qualitative Research

   Wilson-Lopez, A. ; Minichiello, A. ; Green, T. ( July 2019 , ASEE Annual Conference proceedings)

   In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 

   more » « less