More Like this

1. The Effectiveness of Synchronous vs Asynchronous Modes of Instruction in an Online Flipped Design Thinking Course

   Mohandas, L. ( July 2021 , 2021 ASEE Annual Conference & Exposition)

   Motivation: This is a complete paper. There was a sudden shift from traditional learning to online learning in Spring 2020 with the outbreak of COVID-19. Although online learning is not a new topic of discussion, universities, faculty, and students were not prepared for this sudden change in learning. According to a recent article in ‘The Chronicle of Higher Education, “even under the best of circumstances, virtual learning requires a different, carefully crafted approach to engagement”. The Design Thinking course under study is a required freshmen level course offered in a Mid-western University. The Design Thinking course is offered in a flipped format where all the content to be learned is given to students beforehand and the in-class session is used for active discussions and hands-on learning related to the content provided at the small group level. The final learning objective of the course is a group project where student groups are expected to come up with functional prototypes to solve a real-world problem following the Design Thinking process. There were eighteen sections of the Design Thinking course offered in Spring 2020, and with the outbreak of COVID-19, a few instructors decided to offer synchronous online classes (where instructors were presentmore »online during class time and provided orientation and guidance just like a normal class) and a few others decided to offer asynchronous online classes (where orientation from the instructor was delivered asynchronous and the instructor was online during officially scheduled class time but interactions were more like office hours). Students were required to be present synchronously at the team level during the class time in a synchronous online class. In an asynchronous online class, students could be synchronous at the team level to complete their assignment any time prior to the deadline such that they could work during class time but they were not required to work at that time. Through this complete paper, we are trying to understand student learning, social presence and learner satisfaction with respect to different modes of instruction in a freshmen level Design Thinking course. Background: According to literature, synchronous online learning has advantages such as interaction, a classroom environment, and better course quality whereas asynchronous online learning has advantages such as self-controlled and self-directed learning. The disadvantages of synchronous online learning include the learning process, technology issues, and distraction. Social isolation, lack of interaction, and technology issue are a few disadvantages related to asynchronous online learning. Problem Being Addressed: There is a limited literature base investigating different modes of online instruction in a Design Thinking course. Through this paper, we are trying to understand and share the effectiveness of synchronous and asynchronous modes of instruction in an online Flipped Design Thinking Course. The results of the paper could also help in this time of pandemic by shedding light on the more effective way to teach highly active group-based classrooms for better student learning, social presence, and learner satisfaction. Method/Assessment: An end of semester survey was monitored in Spring 2020 to understand student experiences in synchronous and asynchronous Design Thinking course sections. The survey was sent to 720 students enrolled in the course in Spring 2020 and 324 students responded to the survey. Learning was measured using the survey instrument developed by Walker (2003) and the social presence and learner satisfaction was measured by the survey modified by Richardson and Swan (2003). Likert scale was used to measure survey responses. Anticipated Results: Data would be analyzed and the paper would be completed by draft paper submission. As the course under study is a flipped and active course with a significant component of group work, the anticipated results after analysis could be that one mode of instruction has higher student learning, social presence, and learner satisfaction compared to the other.« less
2. Student inquiry in interesting lessons

   Nieves, Hector I. ; Singh, Rashmi ; Dietiker, Leslie ( January 2020 , Mathematics Education Across Cultures: Proceedings of the 42nd Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education)

   Sacristán, A. I. ; Cortés-Zavala, J. C. ; Ruiz-Arias, P. M. (Ed.)

   What impact, if any, do interesting lessons have on the types of questions students ask? To explore this question, we used lesson observations of six teachers from three high schools in the Northeast who were part of a larger study. Lessons come from a range of courses, spanning Algebra through Calculus. After each lesson, students reported interest via lesson experience surveys (Author, 2019). These interest measures were then used to identify each teachers’ highest and lowest interest lessons. The two lessons per teacher allows us to compare across the same set of students per teacher. We compiled 145 student questions and identified whether questions were asked within a group work setting or part of a whole class discussion. Two coders coded 10% of data to improve the rubric for type of students’ questions (what, why, how, and if) and perceived intent (factual, procedural, reasoning, and exploratory). Factual questions asked for definitions or explicit answers. Procedural questions were raised when students looked for algorithms or a solving process. Reasoning questions asked about why procedures worked, or facts were true. Exploratory questions expanded beyond the topic of focus, such as asking about changing the parameters to make sense of a problem. Themore »remaining 90% of data were coded independently to determine interrater reliability (see Landis & Koch, 1977). A Cohen’s Kappa statistic (K=0.87, p<0.001) indicates excellent reliability. Furthermore, both coders reconciled codes before continuing with data analysis. Initial results showed differences between high- and low-interest lessons. Although students raised fewer mathematical questions in high-interest lessons (59) when compared with low-interest lessons (86), high-interest lessons contained more “exploratory” questions (10 versus 6). A chi-square test of independence shows a significant difference, χ2 (3, N = 145) = 12.99, p = .005 for types of students’ questions asked in high- and low-interest lessons. The high-interest lessons had more student questions arise during whole class discussions, whereas low-interest lessons had more student questions during group work. By partitioning each lesson into acts at points where the mathematical content shifted, we were able to examine through how many acts questions remained open. The average number of acts the students’ questions remained unanswered for high-interest lessons (2.66) was higher than that of low-interest lessons (1.68). Paired samples t-tests suggest that this difference is significant t(5)=2.58, p = 0.049. Therefore, student interest in the lesson did appear to impact the type of questions students ask. One possible reason for the differences in student questions is the nature of the lessons students found interesting, which may allow for student freedom to wonder and chase their mathematical ideas. There may be more overall student questions in low-interest lessons because of confusion, but more research is needed to unpack the reasoning behind student questions.« less
3. Predictors of Jurors’ Understanding of Evidence Strength

   McCowan, K. ( March 2020 , Annual Conference of the American Psychology-Law Society.)

   Abstract Expert testimony varies in scientific quality and jurors have a difficult time evaluating evidence quality (McAuliff et al., 2009). In the current study, we apply Fuzzy Trace Theory principles, examining whether visual and gist aids help jurors calibrate to the strength of scientific evidence. Additionally we were interested in the role of jurors’ individual differences in scientific reasoning skills in their understanding of case evidence. Contrary to our preregistered hypotheses, there was no effect of evidence condition or gist aid on evidence understanding. However, individual differences between jurors’ numeracy skills predicted evidence understanding. Summary Poor-quality expert evidence is sometimes admitted into court (Smithburn, 2004). Jurors’ calibration to evidence strength varies widely and is not robustly understood. For instance, previous research has established jurors lack understanding of the role of control groups, confounds, and sample sizes in scientific research (McAuliff, Kovera, & Nunez, 2009; Mill, Gray, & Mandel, 1994). Still others have found that jurors can distinguish weak from strong evidence when the evidence is presented alone, yet not when simultaneously presented with case details (Smith, Bull, & Holliday, 2011). This research highlights the need to present evidence to jurors in a way they can understand. Fuzzy Trace Theory purportsmore »that people encode information in exact, verbatim representations and through “gist” representations, which represent summary of meaning (Reyna & Brainerd, 1995). It is possible that the presenting complex scientific evidence to people with verbatim content or appealing to the gist, or bottom-line meaning of the information may influence juror understanding of that evidence. Application of Fuzzy Trace Theory in the medical field has shown that gist representations are beneficial for helping laypeople better understand risk and benefits of medical treatment (Brust-Renck, Reyna, Wilhelms, & Lazar, 2016). Yet, little research has applied Fuzzy Trace Theory to information comprehension and application within the context of a jury (c.f. Reyna et. al., 2015). Additionally, it is likely that jurors’ individual characteristics, such as scientific reasoning abilities and cognitive tendencies, influence their ability to understand and apply complex scientific information (Coutinho, 2006). Methods The purpose of this study was to examine how jurors calibrate to the strength of scientific information, and whether individual difference variables and gist aids inspired by Fuzzy Trace Theory help jurors better understand complicated science of differing quality. We used a 2 (quality of scientific evidence: high vs. low) x 2 (decision aid to improve calibration - gist information vs. no gist information), between-subjects design. All hypotheses were preregistered on the Open Science Framework. Jury-eligible community participants (430 jurors across 90 juries; Mage = 37.58, SD = 16.17, 58% female, 56.93% White). Each jury was randomly assigned to one of the four possible conditions. Participants were asked to individually fill out measures related to their scientific reasoning skills prior to watching a mock jury trial. The trial was about an armed bank robbery and consisted of various pieces of testimony and evidence (e.g. an eyewitness testimony, police lineup identification, and a sweatshirt found with the stolen bank money). The key piece of evidence was mitochondrial DNA (mtDNA) evidence collected from hair on a sweatshirt (materials from Hans et al., 2011). Two experts presented opposing opinions about the scientific evidence related to the mtDNA match estimate for the defendant’s identification. The quality and content of this mtDNA evidence differed based on the two conditions. The high quality evidence condition used a larger database than the low quality evidence to compare to the mtDNA sample and could exclude a larger percentage of people. In the decision aid condition, experts in the gist information group presented gist aid inspired visuals and examples to help explain the proportion of people that could not be excluded as a match. Those in the no gist information group were not given any aid to help them understand the mtDNA evidence presented. After viewing the trial, participants filled out a questionnaire on how well they understood the mtDNA evidence and their overall judgments of the case (e.g. verdict, witness credibility, scientific evidence strength). They filled this questionnaire out again after a 45-minute deliberation. Measures We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition. We developed a 20-item multiple-choice comprehension test for the mtDNA scientific information in the cases (modeled on Hans et al., 2011, and McAuliff et al., 2009). Participants were shown 20 statements related to DNA evidence and asked whether these statements were True or False. The test was then scored out of 20 points. Results For this project, we measured calibration to the scientific evidence in a few different ways. We are building a full model with these various operationalizations to be presented at APLS, but focus only on one of the calibration DVs (i.e., objective understanding of the mtDNA evidence) in the current proposal. We conducted a general linear model with total score on the mtDNA understanding measure as the DV and quality of scientific evidence condition, decision aid condition, and the four individual difference measures (i.e., NFC, ATS, WNS, and SRS) as predictors. Contrary to our main hypotheses, neither evidence quality nor decision aid condition affected juror understanding. However, the individual difference variables did: we found significant main effects for Scientific Reasoning Skills, F(1, 427) = 16.03, p <.001, np2 = .04, Weller Numeracy Scale, F(1, 427) = 15.19, p <.001, np2 = .03, and Need for Cognition, F(1, 427) = 16.80, p <.001, np2 = .04, such that those who scored higher on these measures displayed better understanding of the scientific evidence. In addition there was a significant interaction of evidence quality condition and scores on the Weller’s Numeracy Scale, F(1, 427) = 4.10, p = .04, np2 = .01. Further results will be discussed. Discussion These data suggest jurors are not sensitive to differences in the quality of scientific mtDNA evidence, and also that our attempt at helping sensitize them with Fuzzy Trace Theory-inspired aids did not improve calibration. Individual scientific reasoning abilities and general cognition styles were better predictors of understanding this scientific information. These results suggest a need for further exploration of approaches to help jurors differentiate between high and low quality evidence. Note: The 3rd author was supported by an AP-LS AP Award for her role in this research. Learning Objective: Participants will be able to describe how individual differences in scientific reasoning skills help jurors understand complex scientific evidence.« less
4. Juror Scientific Reasoning Skills and Discussion of Scientific Evidence During Deliberation

   McCowan, K. ( March 2020 , Annual Conference of the American Psychology-Law Society.)

   Abstract We investigate the link between individual differences in science reasoning skills and mock jurors’ deliberation behavior; specifically, how much they talk about the scientific evidence presented in a complicated, ecologically valid case during deliberation. Consistent with our preregistered hypothesis, mock jurors strong in scientific reasoning discussed the scientific evidence more during deliberation than those with weaker science reasoning skills. Summary With increasing frequency, legal disputes involve complex scientific information (Faigman et al., 2014; Federal Judicial Center, 2011; National Research Council, 2009). Yet people often have trouble consuming scientific information effectively (McAuliff et al., 2009; National Science Board, 2014; Resnick et al., 2016). Individual differences in reasoning styles and skills can affect how people comprehend complex evidence (e.g., Hans, Kaye, Dann, Farley, Alberston, 2011; McAuliff & Kovera, 2008). Recently, scholars have highlighted the importance of studying group deliberation contexts as well as individual decision contexts (Salerno & Diamond, 2010; Kovera, 2017). If individual differences influence how jurors understand scientific evidence, it invites questions about how these individual differences may affect the way jurors discuss science during group deliberations. The purpose of the current study was to examine how individual differences in the way people process scientific information affects the extentmore »to which jurors discuss scientific evidence during deliberations. Methods We preregistered the data collection plan, sample size, and hypotheses on the Open Science Framework. Jury-eligible community participants (303 jurors across 50 juries) from Phoenix, AZ (Mage=37.4, SD=16.9; 58.8% female; 51.5% White, 23.7% Latinx, 9.9% African-American, 4.3% Asian) were paid $55 for a 3-hour mock jury study. Participants completed a set of individual questionnaires related to science reasoning skills and attitudes toward science prior to watching a 45-minute mock armed-robbery trial. The trial included various pieces of evidence and testimony, including forensic experts testifying about mitochondrial DNA evidence (mtDNA; based on Hans et al. 2011 materials). Participants were then given 45 minutes to deliberate. The deliberations were video recorded and transcribed to text for analysis. We analyzed the deliberation content for discussions related to the scientific evidence presented during trial. We hypothesized that those with stronger scientific and numeric reasoning skills, higher need for cognition, and more positive views towards science would discuss scientific evidence more than their counterparts during deliberation. Measures We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by the National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition. Coding We identified verbal utterances related to the scientific evidence presented in court. For instance, references to DNA evidence in general (e.g. nuclear DNA being more conclusive than mtDNA), the database that was used to compare the DNA sample (e.g. the database size, how representative it was), exclusion rates (e.g. how many other people could not be excluded as a possible match), and the forensic DNA experts (e.g. how credible they were perceived). We used word count to operationalize the extent to which each juror discussed scientific information. First we calculated the total word count for each complete jury deliberation transcript. Based on the above coding scheme we determined the number of words each juror spent discussing scientific information. To compare across juries, we wanted to account for the differing length of deliberation; thus, we calculated each juror’s scientific deliberation word count as a proportion of their jury’s total word count. Results On average, jurors discussed the science for about 4% of their total deliberation (SD=4%, range 0-22%). We regressed proportion of the deliberation jurors spend discussing scientific information on the four individual difference measures (i.e., SRS, NFC, WNS, ATS). Using the adjusted R-squared, the measures significantly accounted for 5.5% of the variability in scientific information deliberation discussion, SE=0.04, F(4, 199)=3.93, p=0.004. When controlling for all other variables in the model, the Scientific Reasoning Scale was the only measure that remained significant, b=0.003, SE=0.001, t(203)=2.02, p=0.045. To analyze how much variability each measure accounted for, we performed a stepwise regression, with NFC entered at step 1, ATS entered at step 2, WNS entered at step 3, and SRS entered at step 4. At step 1, NFC accounted for 2.4% of the variability, F(1, 202)=5.95, p=0.02. At step 2, ATS did not significantly account for any additional variability. At step 3, WNS accounted for an additional 2.4% of variability, ΔF(1, 200)=5.02, p=0.03. Finally, at step 4, SRS significantly accounted for an additional 1.9% of variability in scientific information discussion, ΔF(1, 199)=4.06, p=0.045, total adjusted R-squared of 0.055. Discussion This study provides additional support for previous findings that scientific reasoning skills affect the way jurors comprehend and use scientific evidence. It expands on previous findings by suggesting that these individual differences also impact the way scientific evidence is discussed during juror deliberations. In addition, this study advances the literature by identifying Scientific Reasoning Skills as a potentially more robust explanatory individual differences variable than more well-studied constructs like Need for Cognition in jury research. Our next steps for this research, which we plan to present at AP-LS as part of this presentation, incudes further analysis of the deliberation content (e.g., not just the mention of, but the accuracy of the references to scientific evidence in discussion). We are currently coding this data with a software program called Noldus Observer XT, which will allow us to present more sophisticated results from this data during the presentation. Learning Objective: Participants will be able to describe how individual differences in scientific reasoning skills affect how much jurors discuss scientific evidence during deliberation.« less
5. Constructing a comprehensive and adaptive survey for cultural analysis of engineering departments

   Berger, E. ; Briody, E. ; DeBoer, J. ; Rhoads, J. ; Francis, J. ; Witek, L. ; Rothstein, R. ; Lee, Y. ( December 2021 , Proceedings of REES AAEE 2021)

   ABSTRACT CONTEXT Culture influences the dynamics and outcomes of organizations in profound ways, including individual-level outcomes (like the quality of work products) and collective impacts (such as reputation or influence). As such, understanding organizational culture is a crucial element of understanding performance; from an anthropological perspective, ‘performance’ is not an outcome of culture, it is a part of culture. A key challenge in understanding organizational culture, especially in complex academic organizations, is the lack of a flexible, scalable approach for data collection and analysis. PURPOSE OR GOAL In this study, we report on our development of a survey-based cultural characterization tool that leverages both lightweight data collection from stakeholders in the organization and public information about that organization. We also integrate perspectives from prior literature about faculty, students, and staff in academic departments. Taken together, the resulting survey covers key elements of culture and allows for scalable data collection across settings via customizations and embedded logic in the survey itself. The outcome of this work is a design process for a new and promising tool for scalable cultural characterization, and we have deployed this tool across two institutions. APPROACH OR METHODOLOGY/METHODS We leverage prior research, our own preliminary data collection,more »and our experience with this approach in a different setting to develop a cultural characterization survey suitable for delivery to multiple engineering department stakeholders (faculty, staff, and students). We start with a modest number of interviews, stratified by these three groups and achieving saturation of responses, to understand their views on their organization, its strengths and weaknesses, and their perceptions of how it ‘works’. We merge this information with public data (for instance, departmental vision or mission statements, which convey a sense of priorities or values) as well as prior literature about higher education culture. We also draw upon our experience in another setting as well as pilot testing data, and the result is a carefully-constructed set of dichotomous items that are offered to department stakeholders in survey form using an electronic survey platform. We also collect background and demographic information in the survey. The resulting data are analyzed using Cultural Consensus Theory (CCT) to extract meaningful information about the departmental culture from the perspectives of the stakeholder groups. ACTUAL OR ANTICIPATED OUTCOMES The resulting survey consists of two parts, each with sub-components. The two top level survey parts contain: (i) items common to all respondents in all settings (i.e. all institutions in this study), and (ii) a set of institution-specific items. Within those sections, the framing of the items is calibrated for the stakeholder groups so that items make sense to them within the context of their experience. The survey has been administered, and the data are being analyzed and interpreted presently. We expect the results to capture the specific elements of local culture within these institutions, as well as differences in perspectives and experience among the three primary stakeholder groups. CONCLUSIONS/RECOMMENDATIONS/SUMMARY This study demonstrates a scalable approach to survey development for the purposes of cultural characterization, and its use across settings and with multiple stakeholder groups. This work enables a very nuanced view of culture within a department, and these results can be used within academic departments to enable discussion about change, priorities, performance, and the work environment.« less