skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: A Hybrid Learning-Architecture for Mental Disorder Detection Using Emotion Recognition
Mental illness has grown to become a prevalent and global health concern that affects individuals across various demographics. Timely detection and accurate diagnosis of mental disorders are crucial for effective treatment and support as late diagnosis could result in suicidal, harmful behaviors and ultimately death. To this end, the present study introduces a novel pipeline for the analysis of facial expressions, leveraging both the AffectNet and 2013 Facial Emotion Recognition (FER) datasets. Consequently, this research goes beyond traditional diagnostic methods by contributing a system capable of generating a comprehensive mental disorder dataset and concurrently predicting mental disorders based on facial emotional cues. Particularly, we introduce a hybrid architecture for mental disorder detection leveraging the state-of-the-art object detection algorithm, YOLOv8 to detect and classify visual cues associated with specific mental disorders. To achieve accurate predictions, an integrated learning architecture based on the fusion of Convolution Neural Networks (CNNs) and Visual Transformer (ViT) models is developed to form an ensemble classifier that predicts the presence of mental illness (e.g., depression, anxiety, and other mental disorder). The overall accuracy is improved to about 81% using the proposed ensemble technique. To ensure transparency and interpretability, we integrate techniques such as Gradient-weighted Class Activation Mapping (Grad-CAM) and saliency maps to highlight the regions in the input image that significantly contribute to the model’s predictions thus providing healthcare professionals with a clear understanding of the features influencing the system’s decisions thereby enhancing trust and more informed diagnostic process.  more » « less
Award ID(s):
2131207
PAR ID:
10561019
Author(s) / Creator(s):
; ; ; ;
Publisher / Repository:
IEEE
Date Published:
Journal Name:
IEEE Access
Volume:
12
ISSN:
2169-3536
Page Range / eLocation ID:
91410 to 91425
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Stephanidis, Constantine; Chen, Jessie Y.; Fragomeni, Gino (Ed.)
    Post-traumatic stress disorder (PTSD) is a mental health condition affecting people who experienced a traumatic event. In addition to the clinical diagnostic criteria for PTSD, behavioral changes in voice, language, facial expression and head movement may occur. In this paper, we demonstrate how a machine learning model trained on a general population with self-reported PTSD scores can be used to provide behavioral metrics that could enhance the accuracy of the clinical diagnosis with patients. Both datasets were collected from a clinical interview conducted by a virtual agent (SimSensei) [10]. The clinical data was recorded from PTSD patients, who were victims of sexual assault, undergoing a VR exposure therapy. A recurrent neural network was trained on verbal, visual and vocal features to recognize PTSD, according to self-reported PCL-C scores [4]. We then performed decision fusion to fuse three modalities to recognize PTSD in patients with a clinical diagnosis, achieving an F1-score of 0.85. Our analysis demonstrates that machine-based PTSD assessment with self-reported PTSD scores can generalize across different groups and be deployed to assist diagnosis of PTSD. 
    more » « less
  2. null (Ed.)
    Prior papers have explored the functional connectivity changes for patients suffering from major depressive disorder (MDD). This paper introduces an approach for classifying adolescents suffering from MDD using resting-state fMRI. Accurate diagnosis of MDD involves interviews with adolescent patients and their parents, symptom rating scales based on Diagnostic and Statistical Manual of Mental Disorders (DSM), behavioral observation as well as the experience of a clinician. Discovering predictive biomarkers for diagnosing MDD patients using functional magnetic resonance imaging (fMRI) scans can assist the clinicians in their diagnostic assessments. This paper investigates various static and dynamic connectivity measures extracted from resting-state fMRI for assisting with MDD diagnosis. First, absolute Pearson correlation matrices from 85 brain regions are computed and they are used to calculate static features for predicting MDD. A predictive sub-network extracted using sub-graph entropy classifies adolescent MDD vs. typical healthy controls with high accuracy, sensitivity and specificity. Next, approaches utilizing dynamic connectivity are employed to extract tensor based, independent component based and principal component based subject specific attributes. Finally, features from static and dynamic approaches are combined to create a feature vector for classification. A leave-one-out cross-validation method is used for the final predictor performance. Out of 49 adolescents with MDD and 33 matched healthy controls, a support vector machine (SVM) classifier using a radial basis function (RBF) kernel using differential sub-graph entropy combined with dynamic connectivity features classifies MDD vs. healthy controls with an accuracy of 0.82 for leave-one-out cross-validation. This classifier has specificity and sensitivity of 0.79 and 0.84, respectively. This performance demonstrates the utility of MRI based diagnosis of psychiatric disorders like MDD using a combination of static and dynamic functional connectivity features of the brain. 
    more » « less
  3. Loss of a loved one is a painful event that substantially elevates the risk for physical and mental illness and impaired daily function. Socially monogamous prairie voles are laboratory‐amenable rodents that form life‐long pair bonds and exhibit distress upon partner separation, mirroring phenotypes seen in humans. These attributes make voles an excellent model for studying the biology of loss. In this review, we highlight parallels between humans and prairie voles, focusing on reward system engagement during pair bonding and loss. As yearning is a unique feature that differentiates loss from other negative mental states, we posit a model in which the homeostatic reward mechanisms that help to maintain bonds are disrupted upon loss, resulting in yearning and other negative impacts. Finally, we synthesize studies in humans and voles that delineate the remodeling of reward systems during loss adaptation. The stalling of these processes likely contributes to prolonged grief disorder, a diagnosis recently added to the Diagnostic and Statistical Manual for Psychiatry. 
    more » « less
  4. Currently doctors rely on tools such as the Unified Parkinson’s Disease Rating Scale (MDS-UDPRS) and the Scale for the Assessment and Rating of Ataxia (SARA) to make diagnoses for movement disorders based on clinical observations of a patient’s motor movement. Observation-based assessments however are inherently subjective and can differ by person. Moreover, different movement disorders show overlapping symptoms, challenging neurologists to make a correct diagnosis based on eyesight alone. In this work, we create an intelligent interface to highlight movements and gestures that are indicative of a movement disorder to observing doctors. First, we analyzed the walking patterns of 43 participants with Parkinson’s Disease (PD), 60 participants with ataxia, and 52 participants with no movement disorder to find ten metrics that can be used to distinguish PD from ataxia. Next, we designed an interface that provides context to the gestures that are relevant to a movement disorder diagnosis. Finally, we surveyed two neurologists (one who specializes in PD and the other who specializes in ataxia) on how useful this interface is for making a diagnosis. Our results not only showcase additional metrics that can be used to evaluate movement disorders quantitatively but also outline steps to be taken when designing an interface for these kinds of diagnostic tasks. 
    more » « less
  5. Previous studies, both in psychology and linguistics, have shown that individuals with mental illnesses show deviations from normal language use, that these differences can be used to make predictions, and used as a diagnostic tool. Recent studies have shown that machine learning can be used to predict people with mental illnesses based on their writing. However, little attention is paid to the interpretability of the machine learning models. In this talk we will describe our analysis of the machine learning models, the different language patterns that distinguish individuals having mental illnesses from a control group, and the associated privacy concerns. We use a dataset of Tweets that are collected from users who reported a diagnosis of a mental illnesses on Twitter. Given the self-reported nature of the dataset, it is possible that some of these individuals are actively talking about their mental illness on social media. We investigated if the machine learning models are detecting the active mentions of the mental illness or if they are detecting more complex language patterns. We then conducted a feature analysis by creating feature vectors using word unigrams, part of speech tags and word clusters and used feature importance measures and statistical methods to identify important features. This analysis serves two purposes: to understand the machine learning model, and to discover language patterns that would help in identifying people with mental illnesses. Finally, we conducted a qualitative analysis of the misclassifications to understand the potential causes for the misclassifications. 
    more » « less