In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education.
It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public.
One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications.
To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes.
more »
« less
An Intersectional Perspective to Studying Microaggressions in Engineering Programs
Microaggressions are widespread in engineering but have received limited attention from engineering education scholars. This research presents the current state of literature on microaggressions and emphasizes the need to adopt an intersectionality perspective to studying mciroaggressions. The research presents a review of the literature including the (1) study context, (2) study methods, (3) study objectives, (4) microaggressions outcomes and (5) microaggressions types using data from 45 journal articles. Data analysis included coding of the journal articles to identify major themes representing different forms of microaggressions. The current results show that the research studying microaggressions using an intersectional lens is limited. This research contributes to improved understanding regarding microaggressions by identifying the gaps within existing literature on microaggressions. Practically, this research increases the visibility of subtle negative behaviors that engineering minority groups experience and their importance for students’ success and persistence.
more »
« less
- Award ID(s):
- 1828559
- NSF-PAR ID:
- 10129189
- Date Published:
- Journal Name:
- ASEE Annual Conference proceedings
- ISSN:
- 1524-4644
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Abstract Background Over the past decade, there has been a shift in science, technology, engineering and math education, especially in engineering, towards a competency‐based pedagogy. Competency‐based learning (CBL) is an outcome‐based, student‐centered form of instruction where students progress to more advanced work upon mastering the necessary prerequisite content and skills. Many articles have been published on the implementation of CBL in engineering higher education; however, the literature lacks a systematic review that summarizes prior work to inform both future research and practice.
Purpose The purpose of this review is to integrate previous literature as well as identify gaps in competency‐based engineering higher education research. It summarizes the different approaches for implementing CBL, the effects of the pedagogy on student outcomes, tools to enhance its effectiveness, and assessment strategies. In addition, suggestions and recommendations for future research are provided.
Method Engineering education articles were obtained from several EBSCO educational databases. The search was limited to articles published from 2005‐2015, and inclusion criteria consisted of peer‐reviewed journal articles that address the use of CBL in engineering higher education. Articles were then classified into several categories, summarized, and evaluated.
Conclusions Theoretical and applied perspectives are provided that address both the theoretical basis for the effectiveness of CBL and practical aspects of implementing successful CBL instruction in engineering education. There are gaps in the literature regarding how CBL programs should be structured and assessed. Future research directions include empirical quantitative evaluation of CBL's pedagogical effectiveness and the use of CBL for teaching professional skills.
-
Researchers, evaluators and designers from an array of academic disciplines and industry sectors are turning to participatory approaches as they seek to understand and address complex social problems. We refer to participatory approaches that collaboratively engage/ partner with stakeholders in knowledge creation/problem solving for action/social change outcomes as collaborative change research, evaluation and design (CCRED). We further frame CCRED practitioners by their desire to move beyond knowledge creation for its own sake to implementation of new knowledge as a tool for social change. In March and May of 2018, we conducted a literature search of multiple discipline-specific databases seeking collaborative, change-oriented scholarly publications. The search was limited to include peerreviewed journal articles, with English language abstracts available, published in the last five years. The search resulted in 526 citations, 236 of which met inclusion criteria. Though the search was limited to English abstracts, all major geographic regions (North America, Europe, Latin America/Caribbean, APAC, Africa and the Middle East) were represented within the results, although many articles did not state a specific region. Of those identified, most studies were located in North America, with the Middle East having only one identified study. We followed a qualitative thematic synthesis process to examine the abstracts of peer-reviewed articles to identify practices that transcend individual disciplines, sectors and contexts to achieve collaborative change. We surveyed the terminology used to describe CCRED, setting, content/topic of study, type of collaboration, and related benefits/outcomes in order to discern the words used to designate collaboration, the frameworks, tools and methods employed, and the presence of action, evaluation or outcomes. Forty-three percent of the reviewed articles fell broadly within the social sciences, followed by 26 percent in education and 25 percent in health/medicine. In terms of participants and/ or collaborators in the articles reviewed, the vast majority of the 236 articles (86%) described participants, that is, those who the research was about or from whom data was collected. In contrast to participants, partners/collaborators (n=32; 14%) were individuals or groups who participated in the design or implementation of the collaborative change effort described. In terms of the goal for collaboration and/or for doing the work, the most frequently used terminology related to some aspect of engagement and empowerment. Common descriptors for the work itself were ‘social change’ (n=74; 31%), ‘action’ (n=33; 14%), ‘collaborative or participatory research/practice’ (n=13; 6%), ‘transformation’ (n=13; 6%) and ‘community engagement’ (n=10; 4%). Of the 236 articles that mentioned a specific framework or approach, the three most common were some variation of Participatory Action Research (n=30; 50%), Action Research (n=40; 16.9%) or Community-Based Participatory Research (n=17; 7.2%). Approximately a third of the 236 articles did not mention a specific method or tool in the abstract. The most commonly cited method/tool (n=30; 12.7%) was some variation of an arts-based method followed by interviews (n=18; 7.6%), case study (n=16; 6.7%), or an ethnographic-related method (n=14; 5.9%). While some articles implied action or change, only 14 of the 236 articles (6%) stated a specific action or outcome. Most often, the changes described were: the creation or modification of a model, method, process, framework or protocol (n=9; 4%), quality improvement, policy change and social change (n=8; 3%), or modifications to education/training methods and materials (n=5; 2%). The infrequent use of collaboration as a descriptor of partner engagement, coupled with few reported findings of measurable change, raises questions about the nature of CCRED. It appears that conducting CCRED is as complex an undertaking as the problems that the work is attempting to address.more » « less
-
The current scoping review identified emerging evidence on social connectedness resource preferences of older adults in assisted living facilities (ALFs) and the community. A literature search was performed using several databases. We included review articles published between January 2000 and September 2022 in English and related to social connectedness resources in ALFs and the community. Of 134 titles and abstracts, eight studies were included. Study participants comprised 2,482 older adults from 233 ALFs in the United States. Themes were framed using the World Health Organization's International Classification of Functioning, Disability, and Health. For social activities, older adults preferred facility-based recreation and leisure resources. For community social connectedness, residents preferred participation in civic life activities. Participants of older age preferred facility resources, whereas those of younger age preferred more demanding physical activities. Those from larger enrollment facilities preferred facility-based resources compared to community resources. For moderately and less active residents, participation was limited to less demanding activities. Older adults' preferences varied based on age, physical limitations, and size and location of the facility. Findings suggest opportunities for further research on developing ALF- and community-based resources for older adults' social well-being and quality of life. [ Journal of Gerontological Nursing, 49 (9), 35–42.]more » « less
-
null (Ed.)This research paper presents a literature review of Computational Thinking (CT) frameworks and assessment practices. CT is a 21st century way of solving a problem. It refers specifically to the methods that are effective when trying to solve a problem with a machine or other computational tools. In the past few years, CT researchers and educationists' significant movement started to look for a formal definition and composition of CT in K-12 and higher education. From this effort, over 20 different definitions and frameworks for CT have emerged. Although the availability of literature on CT has been increasing over the last decade, there is limited research synthesis available on how to assess CT better. Besides, it is known that in higher education designing assessments for CT is challenging and one of the primary reasons is that the precise meaning of CT is still unknown. This research paper, therefore, presents a systematized literature review on CT frameworks and assessment practice. We search three different databases and review 19 journal articles that address the assessment of CT in higher education to answer the following two research questions: 1) What does the literature inform us about practices and types of assessments used to evaluate CT in higher education? 2) Which frameworks of CT are present in literature to support CT assessment in higher education? The critical components of this review focus on frameworks and assessment practices based on CT. We develop a synthesis of suggestions and explanations to answer the proposed questions based on literature from recent research in CT. Based on our initial synthesis, we found a disconnect between theory and practice. Specifically, neither the ideas within CT frameworks nor those from CT assessment research are being utilized by the other. Therefore, there is a dire need to connect the two for practical implementation and further research in CT in higher education.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
