More Like this

1. An Inquiry into the Use of Intercoder Reliability Measures in Qualitative Research

   Wilson-Lopez, A.; Minichiello, A.; Green, T. (July 2019, ASEE Annual Conference proceedings)

   In this theory paper, we set out to consider, as a matter of methodological interest, the use of quantitative measures of inter-coder reliability (e.g., percentage agreement, correlation, Cohen’s Kappa, etc.) as necessary and/or sufficient correlates for quality within qualitative research in engineering education. It is well known that the phrase qualitative research represents a diverse body of scholarship conducted across a range of epistemological viewpoints and methodologies. Given this diversity, we concur with those who state that it is ill advised to propose recipes or stipulate requirements for achieving qualitative research validity and reliability. Yet, as qualitative researchers ourselves, we repeatedly find the need to communicate the validity and reliability—or quality—of our work to different stakeholders, including funding agencies and the public. One method for demonstrating quality, which is increasingly used in qualitative research in engineering education, is the practice of reporting quantitative measures of agreement between two or more people who code the same qualitative dataset. In this theory paper, we address this common practice in two ways. First, we identify instances in which inter-coder reliability measures may not be appropriate or adequate for establishing quality in qualitative research. We query research that suggests that the numerical measure itself is the goal of qualitative analysis, rather than the depth and texture of the interpretations that are revealed. Second, we identify complexities or methodological questions that may arise during the process of establishing inter-coder reliability, which are not often addressed in empirical publications. To achieve this purposes, in this paper we will ground our work in a review of qualitative articles, published in the Journal of Engineering Education, that have employed inter-rater or inter-coder reliability as evidence of research validity. In our review, we will examine the disparate measures and scores (from 40% agreement to 97% agreement) used as evidence of quality, as well as the theoretical perspectives within which these measures have been employed. Then, using our own comparative case study research as an example, we will highlight the questions and the challenges that we faced as we worked to meet rigorous standards of evidence in our qualitative coding analysis, We will explain the processes we undertook and the challenges we faced as we assigned codes to a large qualitative data set approached from a post positivist perspective. We will situate these coding processes within the larger methodological literature and, in light of contrasting literature, we will describe the principled decisions we made while coding our own data. We will use this review of qualitative research and our own qualitative research experiences to elucidate inconsistencies and unarticulated issues related to evidence for qualitative validity as a means to generate further discussion regarding quality in qualitative coding processes. 

   more » « less
2. Surfacing Deeply Held Beliefs about Gender-and Race-Based Minoritization in Engineering: Emerging Insights After Two Years Focused on Data Collection

   https://doi.org/10.18260/1-2--41977  

   Delpech, Dira; Dringenberg, Emily (August 2022, ASEE Conferences)

   Engineering educators in many contexts are increasingly being called to contribute to equity. The focus of our CAREER project is to investigate the ways in which engineering faculty, staff and administrators think about the cause of gender- and race-based minoritization in engineering. Specifically, we investigate the beliefs they express about why women and people of color remain minoritized in engineering and how they arrive at those beliefs. Long term goals of the work include designing evidence-based professional development that can support faculty at any stage in their development as change agents for equity in engineering. The overarching project design includes a series of four one-on-one interviews with participants. The first two interviews are focused on their beliefs about gender- and race-based minoritization, respectively. The third interview will explore their narrative, and the fourth interview (or some type of interaction) will be designed based on how the project evolves and what we learn. We are currently in the second year of the project. To start, our research team used crowdsourcing as a method of recruiting our participants. We asked students to identify engineering educators that they considered to be inclusive based on their lived experience. We oversampled for students from minoritized groups (non-male, non-white). We also allowed those nominated by students to refer to any peers that they felt were inclusive. This resulted in the following participant pool with at least one majority identity (race or gender): 11 white men, 11 white women, and 5 men of color. After piloting our initial interview protocol, we completed gender beliefs interviews with all 27 participants during the 2020-2021 academic year. We had the interviews transcribed, and members of our research checked them for accuracy and de-identified them. The clean transcripts were then sent back to the participants for review. We began data condensation by generating a summary sheet for each participant, which includes the main concepts captured in each section of their gender beliefs interview. We are currently (2021-2022) conducting race beliefs interviews with those same participants. We published the results of piloting the use of our methodological framework, Thinking as Argument (TaA) in the 2021 ASEE proceedings. In short, we believe the framework shows promise for studying beliefs at a deeper level by inviting participants to work through the types of evidence they draw on to commit to their beliefs about the cause of minoritization in engineering. In this paper, we offer some insights that are emerging at this early stage of the project: Different participants draw on diverse ways of knowing to commit to their beliefs, including lived experience and scholarship. These ways of knowing seem potentially related to their own identities. For example, several participants who identify as men of color leverage their own experiences with racism to explain the cause of sexism. This insight has given us pause on the ways in which our framework, TaA, privileges academic or argumentative ways of knowing. We are gaining awareness of the incredible complexity that exists within trying to characterize or evaluate someone's contributions to equity as they relate to their ways of thinking. This finding has given us a pause about the ways in which we, as researchers, assign value to ways of being or acting. At this current stage, we are exploring further by engaging ourselves in reflection of other ways in which beliefs in this context are formed and we are inviting others to do the same. Future work will include ongoing analysis and sensemaking. With the race beliefs completed, we will be able to use data display techniques to explore any patterns between the participants’ beliefs and positionalities. We look forward to honing our protocol for the narrative interviews and are soliciting feedback in terms of how to use the fourth and final interaction of the project in a more participatory way to encourage and give back to our participants. 

   more » « less
3. Privacy Changes Everything

   https://doi.org/10.1007/978-3-030-33752-0_7  

   Rogers J., Bater J. (October 2019, Lecture notes in computer science)

   We are storing and querying datasets with the private information of individuals at an unprecedented scale in settings ranging from IoT devices in smart homes to mining enormous collections of click trails for targeted advertising. Here, the privacy of the people described in these datasets is usually addressed as an afterthought, engineered on top of a DBMS optimized for performance. At best, these systems support security or managing access to sensitive data. This status quo has brought us a plethora of data breaches in the news. In response, governments are stepping in to enact privacy regulations such as the EU’s GDPR. We posit that there is an urgent need for trustworthy database system that offer end-to-end privacy guarantees for their records with user interfaces that closely resemble that of a relational database. As we shall see, these guarantees inform everything in the database’s design from how we store data to what query results we make available to untrusted clients. In this position paper we first define trustworthy database systems and put their research challenges in the context of relevant tools and techniques from the security community. We then use this backdrop to walk through the “life of a query” in a trustworthy database system. We start with the query parsing and follow the query’s path as the system plans, optimizes, and executes it. We highlight how we will need to rethink each step to make it efficient, robust, and usable for database clients. 

   more » « less
4. Characterizing prospective secondary teachers’ foundation and contingency knowledge for definitions of transformations

   Lai, Y.; Lischka, A. E.; Strayer, J. F. (January 2021, Proceedings of the forty-third annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education)

   D. Olanoff; K. Johnson; S. Spitzer (Ed.)

   To examine differences in dimensions of :MKT observable in different creation mediums for teacher-created representations of practice, we analyzed 54 representations of practice (pairs of 27 video and 27 written) created by 27 PSMTs. We found 7 pairs where we observed higher MKT in the video than in the written representation of practice created by the same PSMT. We found 0 cases where we obse1ved higher MKT in the written as compared to the video representation of practice. We examined these 7 pairs for qualities characterizing the pairs as a set. We use findings to problematize issues central to mathematics teacher education: how mathematics is conceptualized, and how representations of practice are used as assessment of dimensions of teachers' MKT. 

   more » « less
5. Revealing prospective secondary teachers’ mathematical knowledge for teaching in teacher-created representations of practice

   Lischka, A. E.; Lai, Y.; Czap, L.; Strayer, J. (January 2021, The mathematician educator)

   To examine differences in dimensions of MKT observable in different creation mediums for teacher-created representations of practice, we analyzed 54 representations of practice (pairs of 27 video and 27 written) created by 27 PSMTs. We found 7 pairs where we observed higher MKT in the video than in the written representation of practice created by the same PSMT. We found 0 cases where we observed higher MKT in the written as compared to the video representation of practice. We examined these 7 pairs for qualities characterizing the pairs as a set. We use findings to problematize issues central to mathematics teacher education: how mathematics is conceptualized, and how representations of practice are used as assessment of dimensions of teachers’ MKT. 

   more » « less