Title: Why the Data Revolution Needs Qualitative Methods
This essay draws on qualitative social science to propose a critical intellectual
infrastructure for data science of social phenomena. Qualitative sensibilities—
interpretivism, abductive reasoning, and reflexivity in particular—could address
methodological problems that have emerged in data science and help extend the
frontiers of social knowledge. First, an interpretivist lens—which is concerned with the
construction of meaning in a given context—can enable the deeper insights that are
requisite to understanding high-level behavioral patterns from digital trace data.
Without such contextual insights, researchers often misinterpret what they find in
large-scale analysis. Second, abductive reasoning—which is the process of using
observations to generate a new explanation, grounded in prior assumptions about the
world—is common in data science, but its application often is not systematized.
Incorporating norms and practices from qualitative traditions for executing,
describing, and evaluating the application of abduction would allow for greater
transparency and accountability. Finally, data scientists would benefit from increased
reflexivity—which is the process of evaluating how researchers’ own assumptions,
experiences, and relationships influence their research. Studies demonstrate such
aspects of a researcher’s experience that typically are unmentioned in quantitative
traditions can influence research findings. Qualitative researchers have long faced
these same concerns, and their training in how to deconstruct and document personal
and intellectual starting points could prove instructive for data scientists. We believe
these and other qualitative sensibilities have tremendous potential to facilitate the
production of data science research that is more meaningful, reliable, and ethical. more »« less
his panel paper presents research on connecting theory to practice and the lessons learned in a change project, with a focus on team formation during the early stages of change making. An important yet often overlooked step in any change project is pulling together individuals to form a competent and efficient team. A functional change-making team requires a variety of complementary skill sets, which may come from different disciplinary backgrounds and/or different prior experiences.
Kotter (1996) uses the term “guiding coalition” to refer to an effective change-making team. He identifies four key characteristics of guiding coalitions: position power, expertise, credibility, leadership. Kotter also goes on to examine the importance of trust and a common goal. In a review of the literature on guiding coalitions, Have, Have, Huijsmans, and Otto (2017) found that though the concept of a guiding coalition is widely advocated in the literature, only one study showed a moderate correlation between the existence of a guiding coalition and the success of a change process (Abraham, Griffin, & Crawford, 1999). Have et al. (2017) conclude that while the literature provides little evidence to the value of a guiding coalition, it does provide evidence that Kotter’s characteristics of a guiding coalition (position power, expertise, credibility, leadership skills, trust in leadership, and setting common goals) individually have positive effects on the outcomes of a change project. However, we don’t know how these characteristics interact.
This analysis of team building and complementary skill sets emerges from our participatory action research with the NSF REvolutionizing engineering and computer science Departments (RED) teams to investigate the change process within STEM higher education. The research-to-practice cycle is integral to our project; data gathered through working with the RED teams provides insights that are then translated into applied, hands-on practices. We utilize an abductive analysis approach, a qualitative methodology that moves recursively between the data and theory-building to remain open to new or contradictory findings, keeping existing theory in mind while not developing formal hypotheses (Timmermans & Tavory, 2012). We find that many of the teams have learned lessons in the early stages of the change process around the guiding coalition characteristics, and our analysis builds on the literature by examining how these characteristics interact. For example, the expertise of the social scientists and education researchers help discern which change strategies have supporting evidence and fit the context, in addition to what is reasonable for planning, implementation, and evaluation. The results presented in this paper connect theory to practice, clarifying practices for building effective change-making teams within higher education.
ackground: Historically Black College and Universities (HBCUs) have for decades played a pivotal role in producing Black scientists. Research found that HBCUs, despite being under funded and resourced, were responsible for over 10% of Black scientists with doctorates. Even though most earn their doctorates at Historically White Institutions (HWIS), understanding the experience of Black STEM doctoral students at HBCUs is of paramount importance to impacting opportunity for success for underrepresented population groups.
HBCUs are recognized for approaches to learning and learning environments that are more relational, encouraging peer to peer and student to faculty relationships, particularly in the form of same-race and same sex mentorships, resulting in less negative racialized gendered experiences and less competitive atmospheres. In spite of what appears to be accepted truths, such as HBCUs offering more culturally affirming experiences, some researchers suggests that little empirical research exists on the quality of support structures available for graduate students at HBCUS in STEM academic fields, particularly mentoring. Increased understanding would provide essential framing necessary for developing more effective mentors at HBCUs, especially given that there are limited numbers of Black faculty in STEM, even at HBCUs.
Theoretical Framework: Anti-racism and critical capital theory are employed as theoretical frameworks. Both are well suited for questioning taken-for-granted assumptions about the lived experiences of racialized others and for deconstructing systemic issues influencing common faculty practices. These frameworks highlight the contextual experiences of STEM doctoral learning.
Research Design: The researchers were interested in understanding how STEM doctoral faculty at HBCUs perceive their role as mentors.
An NSF AGEP sponsored social science research project explored the dispositions, skills, and knowledge of eight STEM faculty at a HBCU. Attitudes towards culturally liberative mentoring were explored through a qualitative case study. The participating faculty were involved in an institutional change program and were interviewed for an average of 60 minutes. Constant comparative data analysis method was used. Additionally, STEM faculty from participating departments completed two mentoring competency and attitude inventories. This case was drawn from a larger multiple embedded case study.
Research Findings: The research findings indicate that STEM doctoral faculty mentors at HBCUs express attitudes about mentoring that are not all that different from their PWIS counterparts. They have a tendency to hold deficit views of domestic Black students and have minimal awareness of how culture inhibits or facilitates a positive learning experience for Black students. Further the culture of science tended to blind them from the culture of people.
Research Implications: In order to enhance the learning experiences of Black STEM doctoral students at HBCUs, the Black student experience at HBCUs must be deromanticized. Understanding the impact of anti-Black racism even within an environment historically and predominantly Black is imperative. Recognizing the ways in which anti-Black attitudes are insidiously present in faculty attitudes and practices and in environments perceived as friendly and supportive for Black students highlights opportunities for STEM faculty development that can move toward a more culturally liberative framework.
Madamanchi, Aasakiran; Cardella, Monica E.; Glazier, James A.; Umulis, David M.(
, 2018 IEEE Frontiers in Education Conference (FIE))
This Work-in-Progress paper in the Research Category uses a retrospective mixed-methods study to better understand the factors that mediate learning of computational modeling by life scientists. Key stakeholders, including leading scientists, universities and funding agencies, have promoted computational modeling to enable life sciences research and improve the translation of genetic and molecular biology high- throughput data into clinical results. Software platforms to facilitate computational modeling by biologists who lack advanced mathematical or programming skills have had some success, but none has achieved widespread use among life scientists. Because computational modeling is a core engineering skill of value to other STEM fields, it is critical for engineering and computer science educators to consider how we help students from across STEM disciplines learn computational modeling. Currently we lack sufficient research on how best to help life scientists learn computational modeling. To address this gap, in 2017, we observed a short-format summer course designed for life scientists to learn computational modeling. The course used a simulation environment designed to lower programming barriers. We used semi-structured interviews to understand students' experiences while taking the course and in applying computational modeling after the course. We conducted interviews with graduate students and post- doctoral researchers who had completed the course. We also interviewed students who took the course between 2010 and 2013. Among these past attendees, we selected equal numbers of interview subjects who had and had not successfully published journal articles that incorporated computational modeling. This Work-in-Progress paper applies social cognitive theory to analyze the motivations of life scientists who seek training in computational modeling and their attitudes towards computational modeling. Additionally, we identify important social and environmental variables that influence successful application of computational modeling after course completion. The findings from this study may therefore help us educate biomedical and biological engineering students more effectively. Although this study focuses on life scientists, its findings can inform engineering and computer science education more broadly. Insights from this study may be especially useful in aiding incoming engineering and computer science students who do not have advanced mathematical or programming skills and in preparing undergraduate engineering students for collaborative work with life scientists.
McCowan, K.(
, Annual Conference of the American Psychology-Law Society.)
Abstract
We investigate the link between individual differences in science reasoning skills and mock jurors’ deliberation behavior; specifically, how much they talk about the scientific evidence presented in a complicated, ecologically valid case during deliberation. Consistent with our preregistered hypothesis, mock jurors strong in scientific reasoning discussed the scientific evidence more during deliberation than those with weaker science reasoning skills.
Summary
With increasing frequency, legal disputes involve complex scientific information (Faigman et al., 2014; Federal Judicial Center, 2011; National Research Council, 2009). Yet people often have trouble consuming scientific information effectively (McAuliff et al., 2009; National Science Board, 2014; Resnick et al., 2016). Individual differences in reasoning styles and skills can affect how people comprehend complex evidence (e.g., Hans, Kaye, Dann, Farley, Alberston, 2011; McAuliff & Kovera, 2008). Recently, scholars have highlighted the importance of studying group deliberation contexts as well as individual decision contexts (Salerno & Diamond, 2010; Kovera, 2017). If individual differences influence how jurors understand scientific evidence, it invites questions about how these individual differences may affect the way jurors discuss science during group deliberations. The purpose of the current study was to examine how individual differences in the way people process scientific information affects the extent to which jurors discuss scientific evidence during deliberations.
Methods
We preregistered the data collection plan, sample size, and hypotheses on the Open Science Framework. Jury-eligible community participants (303 jurors across 50 juries) from Phoenix, AZ (Mage=37.4, SD=16.9; 58.8% female; 51.5% White, 23.7% Latinx, 9.9% African-American, 4.3% Asian) were paid $55 for a 3-hour mock jury study.
Participants completed a set of individual questionnaires related to science reasoning skills and attitudes toward science prior to watching a 45-minute mock armed-robbery trial. The trial included various pieces of evidence and testimony, including forensic experts testifying about mitochondrial DNA evidence (mtDNA; based on Hans et al. 2011 materials). Participants were then given 45 minutes to deliberate.
The deliberations were video recorded and transcribed to text for analysis. We analyzed the deliberation content for discussions related to the scientific evidence presented during trial. We hypothesized that those with stronger scientific and numeric reasoning skills, higher need for cognition, and more positive views towards science would discuss scientific evidence more than their counterparts during deliberation.
Measures
We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by the National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition.
Coding
We identified verbal utterances related to the scientific evidence presented in court. For instance, references to DNA evidence in general (e.g. nuclear DNA being more conclusive than mtDNA), the database that was used to compare the DNA sample (e.g. the database size, how representative it was), exclusion rates (e.g. how many other people could not be excluded as a possible match), and the forensic DNA experts (e.g. how credible they were perceived).
We used word count to operationalize the extent to which each juror discussed scientific information. First we calculated the total word count for each complete jury deliberation transcript. Based on the above coding scheme we determined the number of words each juror spent discussing scientific information. To compare across juries, we wanted to account for the differing length of deliberation; thus, we calculated each juror’s scientific deliberation word count as a proportion of their jury’s total word count.
Results
On average, jurors discussed the science for about 4% of their total deliberation (SD=4%, range 0-22%). We regressed proportion of the deliberation jurors spend discussing scientific information on the four individual difference measures (i.e., SRS, NFC, WNS, ATS). Using the adjusted R-squared, the measures significantly accounted for 5.5% of the variability in scientific information deliberation discussion, SE=0.04, F(4, 199)=3.93, p=0.004. When controlling for all other variables in the model, the Scientific Reasoning Scale was the only measure that remained significant, b=0.003, SE=0.001, t(203)=2.02, p=0.045.
To analyze how much variability each measure accounted for, we performed a stepwise regression, with NFC entered at step 1, ATS entered at step 2, WNS entered at step 3, and SRS entered at step 4. At step 1, NFC accounted for 2.4% of the variability, F(1, 202)=5.95, p=0.02. At step 2, ATS did not significantly account for any additional variability. At step 3, WNS accounted for an additional 2.4% of variability, ΔF(1, 200)=5.02, p=0.03. Finally, at step 4, SRS significantly accounted for an additional 1.9% of variability in scientific information discussion, ΔF(1, 199)=4.06, p=0.045, total adjusted R-squared of 0.055.
Discussion
This study provides additional support for previous findings that scientific reasoning skills affect the way jurors comprehend and use scientific evidence. It expands on previous findings by suggesting that these individual differences also impact the way scientific evidence is discussed during juror deliberations. In addition, this study advances the literature by identifying Scientific Reasoning Skills as a potentially more robust explanatory individual differences variable than more well-studied constructs like Need for Cognition in jury research.
Our next steps for this research, which we plan to present at AP-LS as part of this presentation, incudes further analysis of the deliberation content (e.g., not just the mention of, but the accuracy of the references to scientific evidence in discussion). We are currently coding this data with a software program called Noldus Observer XT, which will allow us to present more sophisticated results from this data during the presentation.
Learning Objective:
Participants will be able to describe how individual differences in scientific reasoning skills affect how much jurors discuss scientific evidence during deliberation.
McCowan, K.(
, Annual Conference of the American Psychology-Law Society.)
Abstract
Expert testimony varies in scientific quality and jurors have a difficult time evaluating evidence quality (McAuliff et al., 2009). In the current study, we apply Fuzzy Trace Theory principles, examining whether visual and gist aids help jurors calibrate to the strength of scientific evidence. Additionally we were interested in the role of jurors’ individual differences in scientific reasoning skills in their understanding of case evidence. Contrary to our preregistered hypotheses, there was no effect of evidence condition or gist aid on evidence understanding. However, individual differences between jurors’ numeracy skills predicted evidence understanding.
Summary
Poor-quality expert evidence is sometimes admitted into court (Smithburn, 2004). Jurors’ calibration to evidence strength varies widely and is not robustly understood. For instance, previous research has established jurors lack understanding of the role of control groups, confounds, and sample sizes in scientific research (McAuliff, Kovera, & Nunez, 2009; Mill, Gray, & Mandel, 1994). Still others have found that jurors can distinguish weak from strong evidence when the evidence is presented alone, yet not when simultaneously presented with case details (Smith, Bull, & Holliday, 2011). This research highlights the need to present evidence to jurors in a way they can understand. Fuzzy Trace Theory purports that people encode information in exact, verbatim representations and through “gist” representations, which represent summary of meaning (Reyna & Brainerd, 1995). It is possible that the presenting complex scientific evidence to people with verbatim content or appealing to the gist, or bottom-line meaning of the information may influence juror understanding of that evidence. Application of Fuzzy Trace Theory in the medical field has shown that gist representations are beneficial for helping laypeople better understand risk and benefits of medical treatment (Brust-Renck, Reyna, Wilhelms, & Lazar, 2016). Yet, little research has applied Fuzzy Trace Theory to information comprehension and application within the context of a jury (c.f. Reyna et. al., 2015). Additionally, it is likely that jurors’ individual characteristics, such as scientific reasoning abilities and cognitive tendencies, influence their ability to understand and apply complex scientific information (Coutinho, 2006).
Methods
The purpose of this study was to examine how jurors calibrate to the strength of scientific information, and whether individual difference variables and gist aids inspired by Fuzzy Trace Theory help jurors better understand complicated science of differing quality. We used a 2 (quality of scientific evidence: high vs. low) x 2 (decision aid to improve calibration - gist information vs. no gist information), between-subjects design. All hypotheses were preregistered on the Open Science Framework.
Jury-eligible community participants (430 jurors across 90 juries; Mage = 37.58, SD = 16.17, 58% female, 56.93% White). Each jury was randomly assigned to one of the four possible conditions. Participants were asked to individually fill out measures related to their scientific reasoning skills prior to watching a mock jury trial. The trial was about an armed bank robbery and consisted of various pieces of testimony and evidence (e.g. an eyewitness testimony, police lineup identification, and a sweatshirt found with the stolen bank money). The key piece of evidence was mitochondrial DNA (mtDNA) evidence collected from hair on a sweatshirt (materials from Hans et al., 2011). Two experts presented opposing opinions about the scientific evidence related to the mtDNA match estimate for the defendant’s identification. The quality and content of this mtDNA evidence differed based on the two conditions. The high quality evidence condition used a larger database than the low quality evidence to compare to the mtDNA sample and could exclude a larger percentage of people. In the decision aid condition, experts in the gist information group presented gist aid inspired visuals and examples to help explain the proportion of people that could not be excluded as a match. Those in the no gist information group were not given any aid to help them understand the mtDNA evidence presented. After viewing the trial, participants filled out a questionnaire on how well they understood the mtDNA evidence and their overall judgments of the case (e.g. verdict, witness credibility, scientific evidence strength). They filled this questionnaire out again after a 45-minute deliberation.
Measures
We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition. We developed a 20-item multiple-choice comprehension test for the mtDNA scientific information in the cases (modeled on Hans et al., 2011, and McAuliff et al., 2009). Participants were shown 20 statements related to DNA evidence and asked whether these statements were True or False. The test was then scored out of 20 points.
Results
For this project, we measured calibration to the scientific evidence in a few different ways. We are building a full model with these various operationalizations to be presented at APLS, but focus only on one of the calibration DVs (i.e., objective understanding of the mtDNA evidence) in the current proposal. We conducted a general linear model with total score on the mtDNA understanding measure as the DV and quality of scientific evidence condition, decision aid condition, and the four individual difference measures (i.e., NFC, ATS, WNS, and SRS) as predictors. Contrary to our main hypotheses, neither evidence quality nor decision aid condition affected juror understanding. However, the individual difference variables did: we found significant main effects for Scientific Reasoning Skills, F(1, 427) = 16.03, p <.001, np2 = .04, Weller Numeracy Scale, F(1, 427) = 15.19, p <.001, np2 = .03, and Need for Cognition, F(1, 427) = 16.80, p <.001, np2 = .04, such that those who scored higher on these measures displayed better understanding of the scientific evidence. In addition there was a significant interaction of evidence quality condition and scores on the Weller’s Numeracy Scale, F(1, 427) = 4.10, p = .04, np2 = .01. Further results will be discussed.
Discussion
These data suggest jurors are not sensitive to differences in the quality of scientific mtDNA evidence, and also that our attempt at helping sensitize them with Fuzzy Trace Theory-inspired aids did not improve calibration. Individual scientific reasoning abilities and general cognition styles were better predictors of understanding this scientific information. These results suggest a need for further exploration of approaches to help jurors differentiate between high and low quality evidence.
Note: The 3rd author was supported by an AP-LS AP Award for her role in this research.
Learning Objective:
Participants will be able to describe how individual differences in scientific reasoning skills help jurors understand complex scientific evidence.
Tanweer, Anissa, Gade, Emily Kalah, Krafft, P.M., and Dreier, Sarah K. Why the Data Revolution Needs Qualitative Methods. Retrieved from https://par.nsf.gov/biblio/10302851. Harvard Data Science Review . Web. doi:10.1162/99608f92.eee0b0da.
Tanweer, Anissa, Gade, Emily Kalah, Krafft, P.M., & Dreier, Sarah K. Why the Data Revolution Needs Qualitative Methods. Harvard Data Science Review, (). Retrieved from https://par.nsf.gov/biblio/10302851. https://doi.org/10.1162/99608f92.eee0b0da
@article{osti_10302851,
place = {Country unknown/Code not available},
title = {Why the Data Revolution Needs Qualitative Methods},
url = {https://par.nsf.gov/biblio/10302851},
DOI = {10.1162/99608f92.eee0b0da},
abstractNote = {This essay draws on qualitative social science to propose a critical intellectual infrastructure for data science of social phenomena. Qualitative sensibilities— interpretivism, abductive reasoning, and reflexivity in particular—could address methodological problems that have emerged in data science and help extend the frontiers of social knowledge. First, an interpretivist lens—which is concerned with the construction of meaning in a given context—can enable the deeper insights that are requisite to understanding high-level behavioral patterns from digital trace data. Without such contextual insights, researchers often misinterpret what they find in large-scale analysis. Second, abductive reasoning—which is the process of using observations to generate a new explanation, grounded in prior assumptions about the world—is common in data science, but its application often is not systematized. Incorporating norms and practices from qualitative traditions for executing, describing, and evaluating the application of abduction would allow for greater transparency and accountability. Finally, data scientists would benefit from increased reflexivity—which is the process of evaluating how researchers’ own assumptions, experiences, and relationships influence their research. Studies demonstrate such aspects of a researcher’s experience that typically are unmentioned in quantitative traditions can influence research findings. Qualitative researchers have long faced these same concerns, and their training in how to deconstruct and document personal and intellectual starting points could prove instructive for data scientists. We believe these and other qualitative sensibilities have tremendous potential to facilitate the production of data science research that is more meaningful, reliable, and ethical.},
journal = {Harvard Data Science Review},
author = {Tanweer, Anissa and Gade, Emily Kalah and Krafft, P.M. and Dreier, Sarah K.},
}
Warning: Leaving National Science Foundation Website
You are now leaving the National Science Foundation website to go to a non-government website.
Website:
NSF takes no responsibility for and exercises no control over the views expressed or the accuracy of
the information contained on this site. Also be aware that NSF's privacy policy does not apply to this site.
