More Like this

1. The Temple University Digital Pathology Corpus: The Breast Tissue Subset

   https://doi.org/10.1109/SPMB52430.2021.9672275  

   Wevodau, Z. ; Doshna, B. ; Jhala, N. ; Akhtar, I. ; Obeid, I. ; Picone, J. ( December 2021 , Proceedings of the IEEE Signal Processing in Medicine and Biology Symposium (SPMB))

   Obeid, I. (Ed.)

   The Neural Engineering Data Consortium (NEDC) is developing the Temple University Digital Pathology Corpus (TUDP), an open source database of high-resolution images from scanned pathology samples [1], as part of its National Science Foundation-funded Major Research Instrumentation grant titled “MRI: High Performance Digital Pathology Using Big Data and Machine Learning” [2]. The long-term goal of this project is to release one million images. We have currently scanned over 100,000 images and are in the process of annotating breast tissue data for our first official corpus release, v1.0.0. This release contains 3,505 annotated images of breast tissue including 74 patients with cancerous diagnoses (out of a total of 296 patients). In this poster, we will present an analysis of this corpus and discuss the challenges we have faced in efficiently producing high quality annotations of breast tissue. It is well known that state of the art algorithms in machine learning require vast amounts of data. Fields such as speech recognition [3], image recognition [4] and text processing [5] are able to deliver impressive performance with complex deep learning models because they have developed large corpora to support training of extremely high-dimensional models (e.g., billions of parameters). Other fields that do not have access to such data resources must rely on techniques in which existing models can be adapted to new datasets [6]. A preliminary version of this breast corpus release was tested in a pilot study using a baseline machine learning system, ResNet18 [7], that leverages several open-source Python tools. The pilot corpus was divided into three sets: train, development, and evaluation. Portions of these slides were manually annotated [1] using the nine labels in Table 1 [8] to identify five to ten examples of pathological features on each slide. Not every pathological feature is annotated, meaning excluded areas can include focuses particular to these labels that are not used for training. A summary of the number of patches within each label is given in Table 2. To maintain a balanced training set, 1,000 patches of each label were used to train the machine learning model. Throughout all sets, only annotated patches were involved in model development. The performance of this model in identifying all the patches in the evaluation set can be seen in the confusion matrix of classification accuracy in Table 3. The highest performing labels were background, 97% correct identification, and artifact, 76% correct identification. A correlation exists between labels with more than 6,000 development patches and accurate performance on the evaluation set. Additionally, these results indicated a need to further refine the annotation of invasive ductal carcinoma (“indc”), inflammation (“infl”), nonneoplastic features (“nneo”), normal (“norm”) and suspicious (“susp”). This pilot experiment motivated changes to the corpus that will be discussed in detail in this poster presentation. To increase the accuracy of the machine learning model, we modified how we addressed underperforming labels. One common source of error arose with how non-background labels were converted into patches. Large areas of background within other labels were isolated within a patch resulting in connective tissue misrepresenting a non-background label. In response, the annotation overlay margins were revised to exclude benign connective tissue in non-background labels. Corresponding patient reports and supporting immunohistochemical stains further guided annotation reviews. The microscopic diagnoses given by the primary pathologist in these reports detail the pathological findings within each tissue site, but not within each specific slide. The microscopic diagnoses informed revisions specifically targeting annotated regions classified as cancerous, ensuring that the labels “indc” and “dcis” were used only in situations where a micropathologist diagnosed it as such. Further differentiation of cancerous and precancerous labels, as well as the location of their focus on a slide, could be accomplished with supplemental immunohistochemically (IHC) stained slides. When distinguishing whether a focus is a nonneoplastic feature versus a cancerous growth, pathologists employ antigen targeting stains to the tissue in question to confirm the diagnosis. For example, a nonneoplastic feature of usual ductal hyperplasia will display diffuse staining for cytokeratin 5 (CK5) and no diffuse staining for estrogen receptor (ER), while a cancerous growth of ductal carcinoma in situ will have negative or focally positive staining for CK5 and diffuse staining for ER [9]. Many tissue samples contain cancerous and non-cancerous features with morphological overlaps that cause variability between annotators. The informative fields IHC slides provide could play an integral role in machine model pathology diagnostics. Following the revisions made on all the annotations, a second experiment was run using ResNet18. Compared to the pilot study, an increase of model prediction accuracy was seen for the labels indc, infl, nneo, norm, and null. This increase is correlated with an increase in annotated area and annotation accuracy. Model performance in identifying the suspicious label decreased by 25% due to the decrease of 57% in the total annotated area described by this label. A summary of the model performance is given in Table 4, which shows the new prediction accuracy and the absolute change in error rate compared to Table 3. The breast tissue subset we are developing includes 3,505 annotated breast pathology slides from 296 patients. The average size of a scanned SVS file is 363 MB. The annotations are stored in an XML format. A CSV version of the annotation file is also available which provides a flat, or simple, annotation that is easy for machine learning researchers to access and interface to their systems. Each patient is identified by an anonymized medical reference number. Within each patient’s directory, one or more sessions are identified, also anonymized to the first of the month in which the sample was taken. These sessions are broken into groupings of tissue taken on that date (in this case, breast tissue). A deidentified patient report stored as a flat text file is also available. Within these slides there are a total of 16,971 total annotated regions with an average of 4.84 annotations per slide. Among those annotations, 8,035 are non-cancerous (normal, background, null, and artifact,) 6,222 are carcinogenic signs (inflammation, nonneoplastic and suspicious,) and 2,714 are cancerous labels (ductal carcinoma in situ and invasive ductal carcinoma in situ.) The individual patients are split up into three sets: train, development, and evaluation. Of the 74 cancerous patients, 20 were allotted for both the development and evaluation sets, while the remain 34 were allotted for train. The remaining 222 patients were split up to preserve the overall distribution of labels within the corpus. This was done in hope of creating control sets for comparable studies. Overall, the development and evaluation sets each have 80 patients, while the training set has 136 patients. In a related component of this project, slides from the Fox Chase Cancer Center (FCCC) Biosample Repository (https://www.foxchase.org/research/facilities/genetic-research-facilities/biosample-repository -facility) are being digitized in addition to slides provided by Temple University Hospital. This data includes 18 different types of tissue including approximately 38.5% urinary tissue and 16.5% gynecological tissue. These slides and the metadata provided with them are already anonymized and include diagnoses in a spreadsheet with sample and patient ID. We plan to release over 13,000 unannotated slides from the FCCC Corpus simultaneously with v1.0.0 of TUDP. Details of this release will also be discussed in this poster. Few digitally annotated databases of pathology samples like TUDP exist due to the extensive data collection and processing required. The breast corpus subset should be released by November 2021. By December 2021 we should also release the unannotated FCCC data. We are currently annotating urinary tract data as well. We expect to release about 5,600 processed TUH slides in this subset. We have an additional 53,000 unprocessed TUH slides digitized. Corpora of this size will stimulate the development of a new generation of deep learning technology. In clinical settings where resources are limited, an assistive diagnoses model could support pathologists’ workload and even help prioritize suspected cancerous cases. ACKNOWLEDGMENTS This material is supported by the National Science Foundation under grants nos. CNS-1726188 and 1925494. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. REFERENCES [1] N. Shawki et al., “The Temple University Digital Pathology Corpus,” in Signal Processing in Medicine and Biology: Emerging Trends in Research and Applications, 1st ed., I. Obeid, I. Selesnick, and J. Picone, Eds. New York City, New York, USA: Springer, 2020, pp. 67 104. https://www.springer.com/gp/book/9783030368432. [2] J. Picone, T. Farkas, I. Obeid, and Y. Persidsky, “MRI: High Performance Digital Pathology Using Big Data and Machine Learning.” Major Research Instrumentation (MRI), Division of Computer and Network Systems, Award No. 1726188, January 1, 2018 – December 31, 2021. https://www. isip.piconepress.com/projects/nsf_dpath/. [3] A. Gulati et al., “Conformer: Convolution-augmented Transformer for Speech Recognition,” in Proceedings of the Annual Conference of the International Speech Communication Association (INTERSPEECH), 2020, pp. 5036-5040. https://doi.org/10.21437/interspeech.2020-3015. [4] C.-J. Wu et al., “Machine Learning at Facebook: Understanding Inference at the Edge,” in Proceedings of the IEEE International Symposium on High Performance Computer Architecture (HPCA), 2019, pp. 331–344. https://ieeexplore.ieee.org/document/8675201. [5] I. Caswell and B. Liang, “Recent Advances in Google Translate,” Google AI Blog: The latest from Google Research, 2020. [Online]. Available: https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html. [Accessed: 01-Aug-2021]. [6] V. Khalkhali, N. Shawki, V. Shah, M. Golmohammadi, I. Obeid, and J. Picone, “Low Latency Real-Time Seizure Detection Using Transfer Deep Learning,” in Proceedings of the IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2021, pp. 1 7. https://www.isip. piconepress.com/publications/conference_proceedings/2021/ieee_spmb/eeg_transfer_learning/. [7] J. Picone, T. Farkas, I. Obeid, and Y. Persidsky, “MRI: High Performance Digital Pathology Using Big Data and Machine Learning,” Philadelphia, Pennsylvania, USA, 2020. https://www.isip.piconepress.com/publications/reports/2020/nsf/mri_dpath/. [8] I. Hunt, S. Husain, J. Simons, I. Obeid, and J. Picone, “Recent Advances in the Temple University Digital Pathology Corpus,” in Proceedings of the IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 2019, pp. 1–4. https://ieeexplore.ieee.org/document/9037859. [9] A. P. Martinez, C. Cohen, K. Z. Hanley, and X. (Bill) Li, “Estrogen Receptor and Cytokeratin 5 Are Reliable Markers to Separate Usual Ductal Hyperplasia From Atypical Ductal Hyperplasia and Low-Grade Ductal Carcinoma In Situ,” Arch. Pathol. Lab. Med., vol. 140, no. 7, pp. 686–689, Apr. 2016. https://doi.org/10.5858/arpa.2015-0238-OA. 

   more » « less
2. The emergence of the formal category “symmetry” in a new sign language

   https://doi.org/10.1073/pnas.1819872116  

   Gleitman, Lila ; Senghas, Ann ; Flaherty, Molly ; Coppola, Marie ; Goldin-Meadow, Susan ( January 2019 , Proceedings of the National Academy of Sciences)

   Logical properties such as negation, implication, and symmetry, despite the fact that they are foundational and threaded through the vocabulary and syntax of known natural languages, pose a special problem for language learning. Their meanings are much harder to identify and isolate in the child’s everyday interaction with referents in the world than concrete things (like spoons and horses) and happenings and acts (like running and jumping) that are much more easily identified, and thus more easily linked to their linguistic labels (spoon, horse, run, jump). Here we concentrate attention on the category of symmetry [a relation R is symmetrical if and only if (iff) for all x, y: if R ( x, y), then R (y, x)], expressed in English by such terms as similar, marry, cousin, and near. After a brief introduction to how symmetry is expressed in English and other well-studied languages, we discuss the appearance and maturation of this category in Nicaraguan Sign Language (NSL). NSL is an emerging language used as the primary, daily means of communication among a population of deaf individuals who could not acquire the surrounding spoken language because they could not hear it, and who were not exposed to a preexisting sign language because there was none available in their community. Remarkably, these individuals treat symmetry, in both semantic and syntactic regards, much as do learners exposed to a previously established language. These findings point to deep human biases in the structures underpinning and constituting human language. 

   more » « less
3. RTD Light Emission around 1550 nm with IQE up to 6% at 300 K

   https://doi.org/10.1109/DRC50226.2020.9135175  

   Brown, E. R. ; Zhang, W-D. ; Fakhimi, P. ; Growden, T. A. ; Berger, P.R. ( June 2020 , IEEE Device Research Conference (DRC), Columbus, OH (June 22-24, 2020))

   Resonant tunneling diodes (RTDs) have come full-circle in the past 10 years after their demonstration in the early 1990s as the fastest room-temperature semiconductor oscillator, displaying experimental results up to 712 GHz and fmax values exceeding 1.0 THz [1]. Now the RTD is once again the preeminent electronic oscillator above 1.0 THz and is being implemented as a coherent source [2] and a self-oscillating mixer [3], amongst other applications. This paper concerns RTD electroluminescence – an effect that has been studied very little in the past 30+ years of RTD development, and not at room temperature. We present experiments and modeling of an n-type In0.53Ga0.47As/AlAs double-barrier RTD operating as a cross-gap light emitter at ~300K. The MBE-growth stack is shown in Fig. 1(a). A 15-μm-diam-mesa device was defined by standard planar processing including a top annular ohmic contact with a 5-μm-diam pinhole in the center to couple out enough of the internal emission for accurate free-space power measurements [4]. The emission spectra have the behavior displayed in Fig. 1(b), parameterized by bias voltage (VB). The long wavelength emission edge is at  = 1684 nm - close to the In0.53Ga0.47As bandgap energy of Ug ≈ 0.75 eV at 300 K. The spectral peaks for VB = 2.8 and 3.0 V both occur around  = 1550 nm (h = 0.75 eV), so blue-shifted relative to the peak of the “ideal”, bulk InGaAs emission spectrum shown in Fig. 1(b) [5]. These results are consistent with the model displayed in Fig. 1(c), whereby the broad emission peak is attributed to the radiative recombination between electrons accumulated on the emitter side, and holes generated on the emitter side by interband tunneling with current density Jinter. The blue-shifted main peak is attributed to the quantum-size effect on the emitter side, which creates a radiative recombination rate RN,2 comparable to the band-edge cross-gap rate RN,1. Further support for this model is provided by the shorter wavelength and weaker emission peak shown in Fig. 1(b) around = 1148 nm. Our quantum mechanical calculations attribute this to radiative recombination RR,3 in the RTD quantum well between the electron ground-state level E1,e, and the hole level E1,h. To further test the model and estimate quantum efficiencies, we conducted optical power measurements using a large-area Ge photodiode located ≈3 mm away from the RTD pinhole, and having spectral response between 800 and 1800 nm with a peak responsivity of ≈0.85 A/W at  =1550 nm. Simultaneous I-V and L-V plots were obtained and are plotted in Fig. 2(a) with positive bias on the top contact (emitter on the bottom). The I-V curve displays a pronounced NDR region having a current peak-to-valley current ratio of 10.7 (typical for In0.53Ga0.47As RTDs). The external quantum efficiency (EQE) was calculated from EQE = e∙IP/(∙IE∙h) where IP is the photodiode dc current and IE the RTD current. The plot of EQE is shown in Fig. 2(b) where we see a very rapid rise with VB, but a maximum value (at VB= 3.0 V) of only ≈2×10-5. To extract the internal quantum efficiency (IQE), we use the expression EQE= c ∙i ∙r ≡ c∙IQE where ci, and r are the optical-coupling, electrical-injection, and radiative recombination efficiencies, respectively [6]. Our separate optical calculations yield c≈3.4×10-4 (limited primarily by the small pinhole) from which we obtain the curve of IQE plotted in Fig. 2(b) (right-hand scale). The maximum value of IQE (again at VB = 3.0 V) is 6.0%. From the implicit definition of IQE in terms of i and r given above, and the fact that the recombination efficiency in In0.53Ga0.47As is likely limited by Auger scattering, this result for IQE suggests that i might be significantly high. To estimate i, we have used the experimental total current of Fig. 2(a), the Kane two-band model of interband tunneling [7] computed in conjunction with a solution to Poisson’s equation across the entire structure, and a rate-equation model of Auger recombination on the emitter side [6] assuming a free-electron density of 2×1018 cm3. We focus on the high-bias regime above VB = 2.5 V of Fig. 2(a) where most of the interband tunneling should occur in the depletion region on the collector side [Jinter,2 in Fig. 1(c)]. And because of the high-quality of the InGaAs/AlAs heterostructure (very few traps or deep levels), most of the holes should reach the emitter side by some combination of drift, diffusion, and tunneling through the valence-band double barriers (Type-I offset) between InGaAs and AlAs. The computed interband current density Jinter is shown in Fig. 3(a) along with the total current density Jtot. At the maximum Jinter (at VB=3.0 V) of 7.4×102 A/cm2, we get i = Jinter/Jtot = 0.18, which is surprisingly high considering there is no p-type doping in the device. When combined with the Auger-limited r of 0.41 and c ≈ 3.4×10-4, we find a model value of IQE = 7.4% in good agreement with experiment. This leads to the model values for EQE plotted in Fig. 2(b) - also in good agreement with experiment. Finally, we address the high Jinter and consider a possible universal nature of the light-emission mechanism. Fig. 3(b) shows the tunneling probability T according to the Kane two-band model in the three materials, In0.53Ga0.47As, GaAs, and GaN, following our observation of a similar electroluminescence mechanism in GaN/AlN RTDs (due to strong polarization field of wurtzite structures) [8]. The expression is Tinter = (2/9)∙exp[(-2 ∙Ug 2 ∙me)/(2h∙P∙E)], where Ug is the bandgap energy, P is the valence-to-conduction-band momentum matrix element, and E is the electric field. Values for the highest calculated internal E fields for the InGaAs and GaN are also shown, indicating that Tinter in those structures approaches values of ~10-5. As shown, a GaAs RTD would require an internal field of ~6×105 V/cm, which is rarely realized in standard GaAs RTDs, perhaps explaining why there have been few if any reports of room-temperature electroluminescence in the GaAs devices. [1] E.R. Brown,et al., Appl. Phys. Lett., vol. 58, 2291, 1991. [5] S. Sze, Physics of Semiconductor Devices, 2nd Ed. 12.2.1 (Wiley, 1981). [2] M. Feiginov et al., Appl. Phys. Lett., 99, 233506, 2011. [6] L. Coldren, Diode Lasers and Photonic Integrated Circuits, (Wiley, 1995). [3] Y. Nishida et al., Nature Sci. Reports, 9, 18125, 2019. [7] E.O. Kane, J. of Appl. Phy 32, 83 (1961). [4] P. Fakhimi, et al., 2019 DRC Conference Digest. [8] T. Growden, et al., Nature Light: Science & Applications 7, 17150 (2018). [5] S. Sze, Physics of Semiconductor Devices, 2nd Ed. 12.2.1 (Wiley, 1981). [6] L. Coldren, Diode Lasers and Photonic Integrated Circuits, (Wiley, 1995). [7] E.O. Kane, J. of Appl. Phy 32, 83 (1961). [8] T. Growden, et al., Nature Light: Science & Applications 7, 17150 (2018). 

   more » « less
4. The Effects of Gist Information and Scientific Quality on Damages in a Civil Trial

   Dellapaolera, K.S. ( March 2020 , Annual Conference of the American Psychology-Law Society.)

   Abstract: 100 words Jurors are increasingly exposed to scientific information in the courtroom. To determine whether providing jurors with gist information would assist in their ability to make well-informed decisions, the present experiment utilized a Fuzzy Trace Theory-inspired intervention and tested it against traditional legal safeguards (i.e., judge instructions) by varying the scientific quality of the evidence. The results indicate that jurors who viewed high quality evidence rated the scientific evidence significantly higher than those who viewed low quality evidence, but were unable to moderate the credibility of the expert witness and apply damages appropriately resulting in poor calibration. Summary: <1000 words Jurors and juries are increasingly exposed to scientific information in the courtroom and it remains unclear when they will base their decisions on a reasonable understanding of the relevant scientific information. Without such knowledge, the ability of jurors and juries to make well-informed decisions may be at risk, increasing chances of unjust outcomes (e.g., false convictions in criminal cases). Therefore, there is a critical need to understand conditions that affect jurors’ and juries’ sensitivity to the qualities of scientific information and to identify safeguards that can assist with scientific calibration in the courtroom. The current project addresses these issues with an ecologically valid experimental paradigm, making it possible to assess causal effects of evidence quality and safeguards as well as the role of a host of individual difference variables that may affect perceptions of testimony by scientific experts as well as liability in a civil case. Our main goal was to develop a simple, theoretically grounded tool to enable triers of fact (individual jurors) with a range of scientific reasoning abilities to appropriately weigh scientific evidence in court. We did so by testing a Fuzzy Trace Theory-inspired intervention in court, and testing it against traditional legal safeguards. Appropriate use of scientific evidence reflects good calibration – which we define as being influenced more by strong scientific information than by weak scientific information. Inappropriate use reflects poor calibration – defined as relative insensitivity to the strength of scientific information. Fuzzy Trace Theory (Reyna & Brainerd, 1995) predicts that techniques for improving calibration can come from presentation of easy-to-interpret, bottom-line “gist” of the information. Our central hypothesis was that laypeople’s appropriate use of scientific information would be moderated both by external situational conditions (e.g., quality of the scientific information itself, a decision aid designed to convey clearly the “gist” of the information) and individual differences among people (e.g., scientific reasoning skills, cognitive reflection tendencies, numeracy, need for cognition, attitudes toward and trust in science). Identifying factors that promote jurors’ appropriate understanding of and reliance on scientific information will contribute to general theories of reasoning based on scientific evidence, while also providing an evidence-based framework for improving the courts’ use of scientific information. All hypotheses were preregistered on the Open Science Framework. Method Participants completed six questionnaires (counterbalanced): Need for Cognition Scale (NCS; 18 items), Cognitive Reflection Test (CRT; 7 items), Abbreviated Numeracy Scale (ABS; 6 items), Scientific Reasoning Scale (SRS; 11 items), Trust in Science (TIS; 29 items), and Attitudes towards Science (ATS; 7 items). Participants then viewed a video depicting a civil trial in which the defendant sought damages from the plaintiff for injuries caused by a fall. The defendant (bar patron) alleged that the plaintiff (bartender) pushed him, causing him to fall and hit his head on the hard floor. Participants were informed at the outset that the defendant was liable; therefore, their task was to determine if the plaintiff should be compensated. Participants were randomly assigned to 1 of 6 experimental conditions: 2 (quality of scientific evidence: high vs. low) x 3 (safeguard to improve calibration: gist information, no-gist information [control], jury instructions). An expert witness (neuroscientist) hired by the court testified regarding the scientific strength of fMRI data (high [90 to 10 signal-to-noise ratio] vs. low [50 to 50 signal-to-noise ratio]) and gist or no-gist information both verbally (i.e., fairly high/about average) and visually (i.e., a graph). After viewing the video, participants were asked if they would like to award damages. If they indicated yes, they were asked to enter a dollar amount. Participants then completed the Positive and Negative Affect Schedule-Modified Short Form (PANAS-MSF; 16 items), expert Witness Credibility Scale (WCS; 20 items), Witness Credibility and Influence on damages for each witness, manipulation check questions, Understanding Scientific Testimony (UST; 10 items), and 3 additional measures were collected, but are beyond the scope of the current investigation. Finally, participants completed demographic questions, including questions about their scientific background and experience. The study was completed via Qualtrics, with participation from students (online vs. in-lab), MTurkers, and non-student community members. After removing those who failed attention check questions, 469 participants remained (243 men, 224 women, 2 did not specify gender) from a variety of racial and ethnic backgrounds (70.2% White, non-Hispanic). Results and Discussion There were three primary outcomes: quality of the scientific evidence, expert credibility (WCS), and damages. During initial analyses, each dependent variable was submitted to a separate 3 Gist Safeguard (safeguard, no safeguard, judge instructions) x 2 Scientific Quality (high, low) Analysis of Variance (ANOVA). Consistent with hypotheses, there was a significant main effect of scientific quality on strength of evidence, F(1, 463)=5.099, p=.024; participants who viewed the high quality evidence rated the scientific evidence significantly higher (M= 7.44) than those who viewed the low quality evidence (M=7.06). There were no significant main effects or interactions for witness credibility, indicating that the expert that provided scientific testimony was seen as equally credible regardless of scientific quality or gist safeguard. Finally, for damages, consistent with hypotheses, there was a marginally significant interaction between Gist Safeguard and Scientific Quality, F(2, 273)=2.916, p=.056. However, post hoc t-tests revealed significantly higher damages were awarded for low (M=11.50) versus high (M=10.51) scientific quality evidence F(1, 273)=3.955, p=.048 in the no gist with judge instructions safeguard condition, which was contrary to hypotheses. The data suggest that the judge instructions alone are reversing the pattern, though nonsignificant, those who received the no gist without judge instructions safeguard awarded higher damages in the high (M=11.34) versus low (M=10.84) scientific quality evidence conditions F(1, 273)=1.059, p=.30. Together, these provide promising initial results indicating that participants were able to effectively differentiate between high and low scientific quality of evidence, though inappropriately utilized the scientific evidence through their inability to discern expert credibility and apply damages, resulting in poor calibration. These results will provide the basis for more sophisticated analyses including higher order interactions with individual differences (e.g., need for cognition) as well as tests of mediation using path analyses. [References omitted but available by request] Learning Objective: Participants will be able to determine whether providing jurors with gist information would assist in their ability to award damages in a civil trial. 

   more » « less
5. Micro-expressions in Animated Agents

   https://doi.org/10.54941/ahfe1001081  

   Hou, Tianyu ; Adamo, Nicoletta ; Villani, Nicholas ( January 2022 , AHFE International)

   The overall goal of our research is to develop a system of intelligent multimodal affective pedagogical agents that are effective for different types of learners (Adamo et al., 2021). While most of the research on pedagogical agents tends to focus on the cognitive aspects of online learning and instruction, this project explores the less-studied role of affective (or emotional) factors. We aim to design believable animated agents that can convey realistic, natural emotions through speech, facial expressions, and body gestures and that can react to the students’ detected emotional states with emotional intelligence. Within the context of this goal, the specific objective of the work reported in the paper was to examine the extent to which the agents’ facial micro-expressions affect students’ perception of the agents’ emotions and their naturalness. Micro-expressions are very brief facial expressions that occur when a person either deliberately or unconsciously conceals an emotion being felt (Ekman &Friesen, 1969). Our assumption is that if the animated agents display facial micro expressions in addition to macro expressions, they will convey higher expressive richness and naturalness to the viewer, as “the agents can possess two emotional streams, one based on interaction with the viewer and the other based on their own internal state, or situation” (Queiroz et al. 2014, p.2).The work reported in the paper involved two studies with human subjects. The objectives of the first study were to examine whether people can recognize micro-expressions (in isolation) in animated agents, and whether there are differences in recognition based on the agent’s visual style (e.g., stylized versus realistic). The objectives of the second study were to investigate whether people can recognize the animated agents’ micro-expressions when integrated with macro-expressions, the extent to which the presence of micro + macro-expressions affect the perceived expressivity and naturalness of the animated agents, the extent to which exaggerating the micro expressions, e.g. increasing the amplitude of the animated facial displacements affects emotion recognition and perceived agent naturalness and emotional expressivity, and whether there are differences based on the agent’s design characteristics. In the first study, 15 participants watched eight micro-expression animations representing four different emotions (happy, sad, fear, surprised). Four animations featured a stylized agent and four a realistic agent. For each animation, subjects were asked to identify the agent’s emotion conveyed by the micro-expression. In the second study, 234 participants watched three sets of eight animation clips (24 clips in total, 12 clips per agent). Four animations for each agent featured the character performing macro-expressions only, four animations for each agent featured the character performing macro- + micro-expressions without exaggeration, and four animations for each agent featured the agent performing macro + micro-expressions with exaggeration. Participants were asked to recognize the true emotion of the agent and rate the emotional expressivity ad naturalness of the agent in each clip using a 5-point Likert scale. We have collected all the data and completed the statistical analysis. Findings and discussion, implications for research and practice, and suggestions for future work will be reported in the full paper. ReferencesAdamo N., Benes, B., Mayer, R., Lei, X., Meyer, Z., &Lawson, A. (2021). Multimodal Affective Pedagogical Agents for Different Types of Learners. In: Russo D., Ahram T., Karwowski W., Di Bucchianico G., Taiar R. (eds) Intelligent Human Systems Integration 2021. IHSI 2021. Advances in Intelligent Systems and Computing, 1322. Springer, Cham. https://doi.org/10.1007/978-3-030-68017-6_33Ekman, P., &Friesen, W. V. (1969, February). Nonverbal leakage and clues to deception. Psychiatry, 32(1), 88–106. https://doi.org/10.1080/00332747.1969.11023575 Queiroz, R. B., Musse, S. R., &Badler, N. I. (2014). Investigating Macroexpressions and Microexpressions in Computer Graphics Animated Faces. Presence, 23(2), 191-208. http://dx.doi.org/10.1162/

    

   more » « less