More Like this

1. Micro-expressions in Animated Agents

   https://doi.org/10.54941/ahfe1001081  

   Hou, Tianyu ; Adamo, Nicoletta ; Villani, Nicholas ( January 2022 , AHFE International)

   The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/

    

   more » « less
2. Impact of aversive affect on neural mechanisms of categorization decisions

   https://doi.org/10.1002/brb3.3312  

   Levitas, Daniel J. ; Folco, Kess L. ; James, Thomas W. ( November 2023 , Brain and Behavior)

   Many theories contend that evidence accumulation is a critical component of decision‐making. Cognitive accumulation models typically interpret two main parameters: a drift rate and decision threshold. The former is the rate of accumulation, based on the quality of evidence, and the latter is the amount of evidence required for a decision. Some studies have found neural signals that mimic evidence accumulators and can be described by the two parameters. However, few studies have related these neural parameters to experimental manipulations of sensory data or memory representations. Here, we investigated the influence of affective salience on neural accumulation parameters. High affective salience has been repeatedly shown to influence decision‐making, yet its effect on neural evidence accumulation has been unexamined.

   The current study used a two‐choice object categorization task of body images (feet or hands). Half the images in each category were high in affective salience because they contained highly aversive features (gore and mutilation). To study such quick categorization decisions with a relatively slow technique like functional magnetic resonance imaging, we used a gradual reveal paradigm to lengthen cognitive processing time through the gradual “unmasking” of stimuli.

   Because the aversive features were task‐irrelevant, high affective salience produced a distractor effect, slowing decision time. In visual accumulation regions of interest, high affective salience produced a longer time to peak activation. Unexpectedly, the later peak appeared to be the product of changes to both drift rate and decision threshold. The drift rate for high affective salience was shallower, and the decision threshold was greater. To our knowledge, this is the first demonstration of an experimental manipulation of sensory data or memory representations that changed the neural decision threshold.

   These findings advance our knowledge of the neural mechanisms underlying affective responses in general and the influence of high affective salience on object representations and categorization decisions.

    

   more » « less
3. Predicting Affective Cognitions in the Resting Adult Brain

   Bush, Keith A. ; Privratsky, Anthony A. ; Kilts, Clinton D. ( January 2018 , Proceedings of the Conference on Cognitive Computational Neuroscience)

   Patterns of estimated neural activity derived from resting state functional magnetic resonance imaging (rs-fMRI) have been shown to predict a wide range of cognitive and behavioral outcomes in both normative and clinical populations. Yet, without links to established cognitive processes, the functional brain states associated with the resting brain will remain unexplained, and potentially confounded, markers of individual differences. In this work we demonstrate the application of multivoxel pattern classifiers (MVPCs) to predict the valence and arousal properties of spontaneous affect processing in the task-non-engaged resting state. rs-fMRI data were acquired from subjects that were held out from a subject set that underwent image-based affect induction concurrent with fMRI to train the MVPCs. We also validated these affective predictions against a well-established, independent measure of autonomic arousal, skin conductance response. These findings suggest a new neuroimaging methodology for resting state analysis in which models, trained on cognition-specific task-based fMRI acquired from well-matched cohorts, capably predict hidden cognitive processes operating within the resting brain. 

   more » « less
4. Walleye spawning, ice phenology, and covariate data for Upper Midwestern Lakes: 1939-2019

   https://doi.org/10.6073/pasta/f7a55f08dfe2a9514067e5c633313ef4  

   Feiner, Zachary S ; Barta, Martha ; Sass, Greg ; Reed, Jeffrey ; Cichosz, Thomas ; Shultz, Aaron ; Luehring, Mark ( January 2023 , Environmental Data Initiative)

   The phenology of critical biological events in aquatic ecosystems are rapidly shifting due to climate change. Growing variability in phenological cues can increase the likelihood of trophic mismatches, causing recruitment failures in commercially, culturally, and recreationally important fisheries. We tested for changes in spawning phenology of regionally important walleye (Sander vitreus) populations in 194 Midwest US lakes in Minnesota, Michigan, and Wisconsin spanning 1939-2019 to investigate factors influencing walleye phenological responses to climate change and associated climate variability, including ice-off timing, lake physical characteristics, and population stocking history. Data from Wisconsin and Michigan lakes (185 and 5 out of 194 total lakes, respectively) were collected by the Wisconsin Department of Natural Resources (WDNR) and the Great Lakes Indian Fish and Wildlife Commission (GLIFWC) through standardized spring walleye mark-recapture surveys and spring tribal harvest season records. Standardized spring mark-recapture population estimates are performed shortly after ice-off, where following a marking event, a subsequent recapture sampling event is conducted using nighttime electrofishing (typically AC – WDNR, pulsed-DC – GLIFWC) of the entire shoreline including islands for small lakes and index stations for large lakes (Hansen et al. 2015) that is timed to coincide with peak walleye spawning activity (G. Hatzenbeler, WDNR, personal communication; M. Luehring, GLIFWC, personal communication; Beard et al. 1997). Data for four additional Minnesota lakes were collected by the Minnesota Department of Natural Resources (MNDNR) beginning in 1939 during annual collections of walleye eggs and broodstock (Schneider et al. 2010), where date of peak egg take was used to index peak spawning activity. For lakes where spawning location did not match the lake for which the ice-off data was collected, the spawning location either flowed into (Pike River) or was within 50 km of a lake where ice-off data were available (Pine River) and these ice-off data were used. Following the affirmation of off-reservation Ojibwe tribal fishing rights in the Ceded Territories of Wisconsin and the Upper Peninsula of Michigan in 1987, tribal spearfishers have targeted walleye during spring spawning (Mrnak et al. 2018). Nightly harvests are recorded as part of a compulsory creel survey (US Department of the Interior 1991). Using these records, we calculated the date of peak spawning activity in a given lake-year as the day of maximum tribal harvest. Although we were unable to account for varying effort in these data, a preliminary analysis comparing spawning dates estimated using tribal harvest to those determined from standardized agency surveys in the same lake and year showed that they were highly correlated (Pearson’s correlation: r = 0.91, P < 0.001). For lakes that had walleye spawning data from both agency surveys and tribal harvest, we used the data source with the greatest number of observation years. Ice-off phenology data was collected from two sources – either observed from the Global Lake and River Ice Phenology database (Benson et al. 2000)t, or modeled from a USGS region-wide machine-learning model which used North American Land Data Assimilation System (NLDAS) meteorological inputs combined with lake characteristics (lake position, clarity, size, depth, hypsography, etc.) to predict daily water column temperatures from 1979 - 2022, from which ice-off dates could be derived (https://www.sciencebase.gov/catalog/item/6206d3c2d34ec05caca53071; see Corson-Dosch et al. 2023 for details). Modeled data for our study lakes (see (Read et al. 2021) for modeling details), which performed well in reflecting ice phenology when compared to observed data (i.e., highly significant correlation between observed and modeled ice-off dates when both were available; r = 0.71, p < 0.001). Lake surface area (ha), latitude, and maximum depth (m) were acquired from agency databases and lake reports. Lake class was based on a WDNR lakes classification system (Rypel et al. 2019) that categorized lakes based on temperature, water clarity, depth, and fish community. Walleye stocking history was defined using the walleye stocking classification system developed by the Wisconsin Technical Working Group (see also Sass et al. 2021), which categorized lakes based on relative contributions of naturally-produced and stocked fish to adult recruitment by relying heavily on historic records of age-0 and age-1 catch rates and stocking histories. Wisconsin lakes were divided into three groups: natural recruitment (NR), a combination of stocking and natural recruitment (C-ST), and stocked only (ST). Walleye natural recruitment was indexed as age-0 walleye CPE (number of age-0 walleye captured per km of shoreline electrofished) from WDNR and GLIFWC fall electrofishing surveys (see Hansen et al. 2015 for details). We excluded lake-years where stocking of age-0 fish occurred before age-0 surveys to only include measurements of naturally-reproduced fish. 

   more » « less
5. Predictors of Jurors’ Understanding of Evidence Strength

   McCowan, K. ( March 2020 , Annual Conference of the American Psychology-Law Society.)

   Abstract Expert testimony varies in scientific quality and jurors have a difficult time evaluating evidence quality (McAuliff et al., 2009). In the current study, we apply Fuzzy Trace Theory principles, examining whether visual and gist aids help jurors calibrate to the strength of scientific evidence. Additionally we were interested in the role of jurors’ individual differences in scientific reasoning skills in their understanding of case evidence. Contrary to our preregistered hypotheses, there was no effect of evidence condition or gist aid on evidence understanding. However, individual differences between jurors’ numeracy skills predicted evidence understanding. Summary Poor-quality expert evidence is sometimes admitted into court (Smithburn, 2004). Jurors’ calibration to evidence strength varies widely and is not robustly understood. For instance, previous research has established jurors lack understanding of the role of control groups, confounds, and sample sizes in scientific research (McAuliff, Kovera, & Nunez, 2009; Mill, Gray, & Mandel, 1994). Still others have found that jurors can distinguish weak from strong evidence when the evidence is presented alone, yet not when simultaneously presented with case details (Smith, Bull, & Holliday, 2011). This research highlights the need to present evidence to jurors in a way they can understand. Fuzzy Trace Theory purports that people encode information in exact, verbatim representations and through “gist” representations, which represent summary of meaning (Reyna & Brainerd, 1995). It is possible that the presenting complex scientific evidence to people with verbatim content or appealing to the gist, or bottom-line meaning of the information may influence juror understanding of that evidence. Application of Fuzzy Trace Theory in the medical field has shown that gist representations are beneficial for helping laypeople better understand risk and benefits of medical treatment (Brust-Renck, Reyna, Wilhelms, & Lazar, 2016). Yet, little research has applied Fuzzy Trace Theory to information comprehension and application within the context of a jury (c.f. Reyna et. al., 2015). Additionally, it is likely that jurors’ individual characteristics, such as scientific reasoning abilities and cognitive tendencies, influence their ability to understand and apply complex scientific information (Coutinho, 2006). Methods The purpose of this study was to examine how jurors calibrate to the strength of scientific information, and whether individual difference variables and gist aids inspired by Fuzzy Trace Theory help jurors better understand complicated science of differing quality. We used a 2 (quality of scientific evidence: high vs. low) x 2 (decision aid to improve calibration - gist information vs. no gist information), between-subjects design. All hypotheses were preregistered on the Open Science Framework. Jury-eligible community participants (430 jurors across 90 juries; Mage = 37.58, SD = 16.17, 58% female, 56.93% White). Each jury was randomly assigned to one of the four possible conditions. Participants were asked to individually fill out measures related to their scientific reasoning skills prior to watching a mock jury trial. The trial was about an armed bank robbery and consisted of various pieces of testimony and evidence (e.g. an eyewitness testimony, police lineup identification, and a sweatshirt found with the stolen bank money). The key piece of evidence was mitochondrial DNA (mtDNA) evidence collected from hair on a sweatshirt (materials from Hans et al., 2011). Two experts presented opposing opinions about the scientific evidence related to the mtDNA match estimate for the defendant’s identification. The quality and content of this mtDNA evidence differed based on the two conditions. The high quality evidence condition used a larger database than the low quality evidence to compare to the mtDNA sample and could exclude a larger percentage of people. In the decision aid condition, experts in the gist information group presented gist aid inspired visuals and examples to help explain the proportion of people that could not be excluded as a match. Those in the no gist information group were not given any aid to help them understand the mtDNA evidence presented. After viewing the trial, participants filled out a questionnaire on how well they understood the mtDNA evidence and their overall judgments of the case (e.g. verdict, witness credibility, scientific evidence strength). They filled this questionnaire out again after a 45-minute deliberation. Measures We measured Attitudes Toward Science (ATS) with indices of scientific promise and scientific reservations (Hans et al., 2011; originally developed by National Science Board, 2004; 2006). We used Drummond and Fischhoff’s (2015) Scientific Reasoning Scale (SRS) to measure scientific reasoning skills. Weller et al.’s (2012) Numeracy Scale (WNS) measured proficiency in reasoning with quantitative information. The NFC-Short Form (Cacioppo et al., 1984) measured need for cognition. We developed a 20-item multiple-choice comprehension test for the mtDNA scientific information in the cases (modeled on Hans et al., 2011, and McAuliff et al., 2009). Participants were shown 20 statements related to DNA evidence and asked whether these statements were True or False. The test was then scored out of 20 points. Results For this project, we measured calibration to the scientific evidence in a few different ways. We are building a full model with these various operationalizations to be presented at APLS, but focus only on one of the calibration DVs (i.e., objective understanding of the mtDNA evidence) in the current proposal. We conducted a general linear model with total score on the mtDNA understanding measure as the DV and quality of scientific evidence condition, decision aid condition, and the four individual difference measures (i.e., NFC, ATS, WNS, and SRS) as predictors. Contrary to our main hypotheses, neither evidence quality nor decision aid condition affected juror understanding. However, the individual difference variables did: we found significant main effects for Scientific Reasoning Skills, F(1, 427) = 16.03, p <.001, np2 = .04, Weller Numeracy Scale, F(1, 427) = 15.19, p <.001, np2 = .03, and Need for Cognition, F(1, 427) = 16.80, p <.001, np2 = .04, such that those who scored higher on these measures displayed better understanding of the scientific evidence. In addition there was a significant interaction of evidence quality condition and scores on the Weller’s Numeracy Scale, F(1, 427) = 4.10, p = .04, np2 = .01. Further results will be discussed. Discussion These data suggest jurors are not sensitive to differences in the quality of scientific mtDNA evidence, and also that our attempt at helping sensitize them with Fuzzy Trace Theory-inspired aids did not improve calibration. Individual scientific reasoning abilities and general cognition styles were better predictors of understanding this scientific information. These results suggest a need for further exploration of approaches to help jurors differentiate between high and low quality evidence. Note: The 3rd author was supported by an AP-LS AP Award for her role in this research. Learning Objective: Participants will be able to describe how individual differences in scientific reasoning skills help jurors understand complex scientific evidence. 

   more » « less