skip to main content


Title: Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction
New modes of technology are offering unprecedented opportunities to unobtrusively collect data about people's behavior. While there are many use cases for such information, we explore its utility for predicting multiple clinical assessment scores. Because clinical assessments are typically used as screening tools for impairment and disease, such as mild cognitive impairment (MCI), automatically mapping behavioral data to assessment scores can help detect changes in health and behavior across time. In this article, we aim to extract behavior markers from two modalities, a smart home environment and a custom digital memory notebook app, for mapping to 10 clinical assessments that are relevant for monitoring MCI onset and changes in cognitive health. Smart-home-based behavior markers reflect hourly, daily, and weekly activity patterns, while app-based behavior markers reflect app usage and writing content/style derived from free-form journal entries. We describe machine learning techniques for fusing these multimodal behavior markers and utilizing joint prediction. We evaluate our approach using three regression algorithms and data from 14 participants with MCI living in a smart-home environment. We observed moderate to large correlations between predicted and ground-truth assessment scores, ranging from r = 0.601 to r = 0.871 for each clinical assessment.  more » « less
Award ID(s):
1954372
NSF-PAR ID:
10418201
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
ACM Transactions on Computing for Healthcare
Volume:
3
Issue:
4
ISSN:
2691-1957
Page Range / eLocation ID:
1 to 25
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. In recent years, behavioral markers such as spoken language and lexical preferences have been studied in the early detection of mild cognitive impairment (MCI) using conversations. While the combination of linguistic and acoustic signals have been shown to be effective in detecting MCI, they have generally been restricted to structured conversations in which the interviewee responds to fixed prompts. In this study, we show that linguistic and acoustic features can be combined synergistically to identify MCI in semi-structured conversations. Using conversational data from an on-going clinical trial ( Clinicaltrials.gov : NCT02871921), we find that the combination of linguistic and acoustic features on semi-structured conversations achieves a mean AUC of 82.7, significantly ( p < 0.01) out-performing linguistic-only (74.9 mean AUC) or acoustic-only (65.0 mean AUC) detections on hold-out data. Additionally, features (linguistic, acoustic and combination) obtained from semi-structured conversations outperform their counterparts obtained from structured weekly conversations in identifying MCI. Some linguistic categories are significantly better at predicting MCI status (e.g., death, home) than others. 
    more » « less
  2. Abstract Study Objectives

    Dementia is a growing cause of disability and loss of independence in the elderly, yet remains largely underdiagnosed. Early detection and classification of dementia can help close this diagnostic gap and improve management of disease progression. Altered oscillations in brain activity during sleep are an early feature of neurodegenerative diseases and be used to identify those on the verge of cognitive decline.

    Methods

    Our observational cross-sectional study used a clinical dataset of 10 784 polysomnography from 8044 participants. Sleep macro- and micro-structural features were extracted from the electroencephalogram (EEG). Microstructural features were engineered from spectral band powers, EEG coherence, spindle, and slow oscillations. Participants were classified as dementia (DEM), mild cognitive impairment (MCI), or cognitively normal (CN) based on clinical diagnosis, Montreal Cognitive Assessment, Mini-Mental State Exam scores, clinical dementia rating, and prescribed medications. We trained logistic regression, support vector machine, and random forest models to classify patients into DEM, MCI, and CN groups.

    Results

    For discriminating DEM versus CN, the best model achieved an area under receiver operating characteristic curve (AUROC) of 0.78 and area under precision-recall curve (AUPRC) of 0.22. For discriminating MCI versus CN, the best model achieved an AUROC of 0.73 and AUPRC of 0.18. For discriminating DEM or MCI versus CN, the best model achieved an AUROC of 0.76 and AUPRC of 0.32.

    Conclusions

    Our dementia classification algorithms show promise for incorporating dementia screening techniques using routine sleep EEG. The findings strengthen the concept of sleep as a window into neurodegenerative diseases.

     
    more » « less
  3. Abstract

    In the Alzheimer’s disease (AD) continuum, the prodromal state of mild cognitive impairment (MCI) precedes AD dementia and identifying MCI individuals at risk of progression is important for clinical management. Our goal was to develop generalizable multivariate models that integrate high-dimensional data (multimodal neuroimaging and cerebrospinal fluid biomarkers, genetic factors, and measures of cognitive resilience) for identification of MCI individuals who progress to AD within 3 years. Our main findings were i) we were able to build generalizable models with clinically relevant accuracy (~93%) for identifying MCI individuals who progress to AD within 3 years; ii) markers of AD pathophysiology (amyloid, tau, neuronal injury) accounted for large shares of the variance in predicting progression; iii) our methodology allowed us to discover that expression ofCR1(complement receptor 1), an AD susceptibility gene involved in immune pathways, uniquely added independent predictive value. This work highlights the value of optimized machine learning approaches for analyzing multimodal patient information for making predictive assessments.

     
    more » « less
  4. Abstract Background

    In Alzheimer’s Diseases (AD) research, multimodal imaging analysis can unveil complementary information from multiple imaging modalities and further our understanding of the disease. One application is to discover disease subtypes using unsupervised clustering. However, existing clustering methods are often applied to input features directly, and could suffer from the curse of dimensionality with high-dimensional multimodal data. The purpose of our study is to identify multimodal imaging-driven subtypes in Mild Cognitive Impairment (MCI) participants using a multiview learning framework based on Deep Generalized Canonical Correlation Analysis (DGCCA), to learn shared latent representation with low dimensions from 3 neuroimaging modalities.

    Results

    DGCCA applies non-linear transformation to input views using neural networks and is able to learn correlated embeddings with low dimensions that capture more variance than its linear counterpart, generalized CCA (GCCA). We designed experiments to compare DGCCA embeddings with single modality features and GCCA embeddings by generating 2 subtypes from each feature set using unsupervised clustering. In our validation studies, we found that amyloid PET imaging has the most discriminative features compared with structural MRI and FDG PET which DGCCA learns from but not GCCA. DGCCA subtypes show differential measures in 5 cognitive assessments, 6 brain volume measures, and conversion to AD patterns. In addition, DGCCA MCI subtypes confirmed AD genetic markers with strong signals that existing late MCI group did not identify.

    Conclusion

    Overall, DGCCA is able to learn effective low dimensional embeddings from multimodal data by learning non-linear projections. MCI subtypes generated from DGCCA embeddings are different from existing early and late MCI groups and show most similarity with those identified by amyloid PET features. In our validation studies, DGCCA subtypes show distinct patterns in cognitive measures, brain volumes, and are able to identify AD genetic markers. These findings indicate the promise of the imaging-driven subtypes and their power in revealing disease structures beyond early and late stage MCI.

     
    more » « less
  5. Abstract Background

    Alzheimer's disease (AD), the most prevalent form of dementia, affects 6.5 million Americans and over 50 million people globally. Clinical, genetic, and phenotypic studies of dementia provide some insights of the observed progressive neurodegenerative processes, however, the mechanisms underlying AD onset remain enigmatic.

    Aims

    This paper examines late‐onset dementia‐related cognitive impairment utilizing neuroimaging‐genetics biomarker associations.

    Materials and Methods

    The participants, ages 65–85, included 266 healthy controls (HC), 572 volunteers with mild cognitive impairment (MCI), and 188 Alzheimer's disease (AD) patients. Genotype dosage data for AD‐associated single nucleotide polymorphisms (SNPs) were extracted from the imputed ADNI genetics archive using sample‐major additive coding. Such 29 SNPs were selected, representing a subset of independent SNPs reported to be highly associated with AD in a recent AD meta‐GWAS study by Jansen and colleagues.

    Results

    We identified the significant correlations between the 29 genomic markers (GMs) and the 200 neuroimaging markers (NIMs). The odds ratios and relative risks for AD and MCI (relative to HC) were predicted using multinomial linear models.

    Discussion

    In the HC and MCI cohorts, mainly cortical thickness measures were associated with GMs, whereas the AD cohort exhibited different GM‐NIM relations. Network patterns within the HC and AD groups were distinct in cortical thickness, volume, and proportion of White to Gray Matter (pct), but not in the MCI cohort. Multinomial linear models of clinical diagnosis showed precisely the specific NIMs and GMs that were most impactful in discriminating between AD and HC, and between MCI and HC.

    Conclusion

    This study suggests that advanced analytics provide mechanisms for exploring the interrelations between morphometric indicators and GMs. The findings may facilitate further clinical investigations of phenotypic associations that support deep systematic understanding of AD pathogenesis.

     
    more » « less