skip to main content

Title: A Review of Digital Ethnographic Methods with Implications for Engineering Education Research
Full Paper: Digital transformations are reshaping engineering practices with implications for conducting engineering education research. Given the paucity of discussion of digital methods within engineering education research, we believe it is important to examine and present to the community an overview of how digital technology is changing research practices. In this paper we focus on digital ethnography as it has implications for studies of technical education and work, which necessarily involve using, and observing how others employ digital data sources, tools, systems, methods, etc. In this paper we report preliminary results from an in-depth literature search and review. To select the papers for the review, we first examined prior meta-review papers that identified new ethnographic methods appropriate for digital contexts (e.g., network ethnography, trace ethnography, rapid ethnography, connective ethnography, focused ethnography, etc.). We then used these as keywords to search for papers that were representative of these methods and selected the 100 most cited papers from this corpus, with further screening resulting in a final collection of 91 papers. We then conducted free/open coding of the articles followed by thematic coding to identify six categories and dived deeper into one of the categories, focused on different approaches to ethnography, to more » further explore the various types of ethnographic methods mentioned in the collected literature. We close by discussing how emerging techniques in ethnographic field research can be applied to engineering education research with engineering work practices as an exemplar. « less
; ; ;
Award ID(s):
1939105 1939272
Publication Date:
Journal Name:
2021 IEEE Frontiers in Education Conference (FIE)
Sponsoring Org:
National Science Foundation
More Like this
  1. In June 2020, at the annual conference of the American Society for Engineering Education (ASEE), which was held entirely online due to the impacts of COVID-19 (SARS-CoV-2), engineering education researchers and social justice scholars diagnosed the spread of two diseases in the United States: COVID-19 and racism. During a virtual workshop (T614A) titled, “Using Power, Privilege, and Intersectionality as Lenses to Understand our Experiences and Begin to Disrupt and Dismantle Oppressive Structures Within Academia,” Drs. Nadia Kellam, Vanessa Svihla, Donna Riley, Alice Pawley, Kelly Cross, Susannah Davis, and Jay Pembridge presented what we might call a pathological analysis of institutionalized racism and various other “isms.” In order to address the intersecting impacts of this double pandemic, they prescribed counter practices and protocols of anti-racism, and strategies against other oppressive “isms” in academia. At the beginning of the virtual workshop, the presenters were pleasantly surprised to see that they had around a hundred attendees. Did the online format of the ASEE conference afford broader exposure of the workshop? Did recent uprising of Black Lives Matter (BLM) protests across the country, and internationally, generate broader interest in their topic? Whatever the case, at a time when an in-person conference could not bemore »convened without compromising public health safety, ASEE’s virtual conference platform, furnished by Pathable and supplemented by Zoom, made possible the broader social impacts of Dr. Svihla’s land acknowledgement of the unceded Indigenous lands from which she was presenting. Svihla attempted to go beyond a hollow gesture by including a hyperlink in her slides to a COVID-19 relief fund for the Navajo Nation, and encouraged attendees to make a donation as they copied and pasted the link in the Zoom Chat. Dr. Cross’s statement that you are either a racist or an anti-racist at this point also promised broader social impacts in the context of the virtual workshop. You could feel the intensity of the BLM social movements and the broader political climate in the tone of the presenters’ voices. The mobilizing masses on the streets resonated with a cutting-edge of social justice research and education at the ASEE virtual conference. COVID-19 has both exacerbated and made more obvious the unevenness and inequities in our educational practices, processes, and infrastructures. This paper is an extension of a broader collaborative research project that accounts for how an exceptional group of engineering educators have taken this opportunity to socially broaden their curricula to include not just public health matters, but also contemporary political and social movements. Engineering educators for change and advocates for social justice quickly recognized the affordances of diverse forms of digital technologies, and the possibilities of broadening their impact through educational practices and infrastructures of inclusion, openness, and accessibility. They are makers of what Gary Downy calls “scalable scholarship”—projects in support of marginalized epistemologies that can be scaled up from ideation to practice in ways that unsettle and displace the dominant epistemological paradigm of engineering education.[1] This paper is a work in progress. It marks the beginning of a much lengthier project that documents the key positionality of engineering educators for change, and how they are socially situated in places where they can connect social movements with industrial transitions, and participate in the production of “undone sciences” that address “a structured absence that emerges from relations of inequality.”[2] In this paper, we offer a brief glimpse into ethnographic data we collected virtually through interviews, participant observation, and digital archiving from March 2019 to August 2019, during the initial impacts of COVID-19 in the United States. The collaborative research that undergirds this paper is ongoing, and what is presented here is a rough and early articulation of ideas and research findings that have begun to emerge through our engagement with engineering educators for change. This paper begins by introducing an image concept that will guide our analysis of how, in this historical moment, forms of social and racial justice are finding their way into the practices of engineering educators through slight changes in pedagogical techniques in response the debilitating impacts of the pandemic. Conceptually, we are interested in how small and subtle changes in learning conditions can socially broaden the impact of engineering educators for change. After introducing the image concept that guides this work, we will briefly discuss methodology and offer background information about the project. Next, we discuss literature that revolves around the question, what is engineering education for? Finally, we introduce the notion of situating engineering education and give readers a brief glimpse into our ethnographic data. The conclusion will indicate future directions for writing, research, and intervention.« less
  2. This paper is a research paper. Many engineering problems require efficient coordination across disciplinary boundaries. Few studies exist about how engineers negotiate and coordinate the knowledge required for working across these boundaries on large, intricate engineering problems. We approach knowledge as a complex and socially constructed system. Knowledge systems are inherently difficult to study because they are dynamic and ephemeral: they are only visible in interactions among the individuals of the community. The purpose of this research is to gain a better understanding of the knowledge system of practicing engineers through ethnographic observations of their practices. We used an ethnography-inspired situative approach based on observable knowledge practices to study the knowledge system of practicing engineers. Data was collected through observation of a Critical Design Review (CDR) of a satellite project at NASA. A CDR occurs after the technical design and specifications of a project nears completion and brings together the scientists and engineers on a project to present their plans to an external review board. A CDR therefore provides a unique opportunity to witness how knowledge is exchanged and negotiated within a complex, interdisciplinary setting. The resulting ethnographic observations were analyzed and categorized into peak events. Peak events were identifiedmore »when successive questions were asked pertaining to the engineering design. Focusing on these events is a useful lens to get insight about the overall knowledge system because they can represent moments where different understandings and disciplinary perspectives emerge. This paper reports on one such peak event concerning the thermal design of the satellite. We focus on one peak to provide sufficient detail so that the knowledge system and its context can be understood. Thermal design of a spacecraft is complex and dynamic with the engineer having to design for drastically different external thermal environments while balancing the changing thermal demands of internal systems. The thermal design discussion provides a particularly thorough example of a knowledge system since the engineer explained, justified, negotiated, and defended knowledge within a social setting. For example, a reviewer asked the engineer if they had taken into account what they considered to be the worst-case scenario. This required an extended discussion to negotiate the criteria by which the credibility and relevance of design components were assessed and to create a shared meaning of what “worst-case” meant. This discussion was centrally important to the technical success of the project and was unequivocally “engineering,” even though it was light on technical detail. This aspect of engineering work is focused more on the epistemic criteria by which knowledge is assessed (i.e. on the foundations of the knowledge system), rather than the technical knowledge of the design itself. Engineering students do not get much practice or instruction in explicitly negotiating knowledge systems and epistemic standards. Although this analysis is limited to a single discussion, we argue that such discussions are important in many engineering projects. Understanding how engineers communicate across different epistemic and disciplinary viewpoints is another step towards creating an engineering curriculum that more closely aligns with engineering practice. Furthermore, it shows that engineering knowledge is not only something to be possessed but instead something that must be negotiated within an interconnected and socially situated knowledge system.« less
  3. This work-in-progress paper reports on the early phases of an exploratory research project involving use of innovative approaches to collecting, analyzing, and archiving empirical data related to engineering practice. More specifically, our project takes an ethnographic approach to studying technical teams at multiple field sites representing multiple industry sectors using novel methods such as agile ethnography, trace ethnography, and network ethnography. In contrast to traditional ethnographic studies that may involve long periods of participant observation, these new approaches often involve less intensive fieldwork, and are instead designed around more targeted research questions and other sources of evidence (e.g., social network data, documentary traces in digital systems). These methods are new and evolving, and therefore have scarcely been used to study engineering practice. Thus, one major goal of the paper is to introduce the proposed methods to the engineering education research community. In doing so, we explore the potential for these methods to generate research findings more rapidly and with a greater focus on specific problems and questions of interest to both industry and researchers. Such methods have gained traction in workplace settings precisely due to such advantages, especially in software engineering and related fields where work is already very digitalmore »and distributed in character. A second major goal of this paper is to give a brief progress report on the early stages of our study, including the initial groundwork carried out to gain access to, and begin collecting data at, multiple field sites. We expect this paper will appeal to scholars who study engineering practice, and those who are interested in contemporary innovations in ethnographic and other qualitative research methods.« less
  4. In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itselfmore »is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes.« less
  5. This work in progress paper presents an example of conducting a systematic literature review (SLR) to understand students’ affective response to active learning practices, and it focuses on the development and testing of a coding form for analyzing the literature. Specifically, the full paper seeks to answer: (1) what affective responses do instructors measure, (2) what evidence is used to study those responses, and (3) how are course features connected with student response. We conducted database searches with carefully-defined search queries which resulted in 2,365 abstracts from 1990 to 2015. Each abstract was screened by two researchers based on meeting inclusion criteria, with an adjudication round in the case of disagreement. We used RefWorks, an online citation management program, to track abstracts during this process. We identified over 480 abstracts which satisfied our criteria. Following abstract screening, we developed and tested a manuscript coding guide to capture the salient characteristics of each paper. We created an initial coding form by determining what paper topics would address our research questions and reviewing the literature to determine the most frequent response categories. We then piloted and tested the reliability of the form over three rounds of independent pair-coding, with each round resultingmore »in clarifications to the form and mutual agreement on terms’ meanings. This process of developing a manuscript coding guide demonstrates how to use free online tools, such as Google Forms and Google Sheets, to inexpensively manage a large SLR team with significant turnover. Currently, we are in the process of applying the coding guide to the full texts. When complete, the resulting data will be synthesized by creating and testing relationships between variables, using each primary source as a case study to support or refute the hypothesized relationship.« less