More Like this

1. Deep Learning on Eye Gaze Data to Classify Student Distraction Level in an Educational VR Environment

   https://doi.org/10.2312/egve.20211326  

   Asish, S. M.; Hossain, E.; Kulshreshth. A. K.; Borst, C. W. (January 2021, International Conference on Artificial Reality and Telexistence Eurographics Symposium on Virtual Environments (ICAT-EGVE))

   Educational VR may increase engagement and retention compared to traditional learning, for some topics or students. However, a student could still get distracted and disengaged due to stress, mind-wandering, unwanted noise, external alerts, etc. Student eye gaze can be useful for detecting distraction. For example, we previously considered gaze visualizations to help teachers understand student attention to better identify or guide distracted students. However, it is not practical for a teacher to monitor a large numbers of student indicators while teaching. To help filter students based on distraction level, we consider a deep learning approach to detect distraction from gaze data. The key aspects are: (1) we created a labeled eye gaze dataset (3.4M data points) from an educational VR environment, (2) we propose an automatic system to gauge a student's distraction level from gaze data, and (3) we apply and compare three deep neural classifiers for this purpose. A proposed CNN-LSTM classifier achieved an accuracy of 89.8\% for classifying distraction, per educational activity section, into one of three levels. 

   more » « less
2. Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework

   https://doi.org/10.1364/BOE.413666  

   Deligani, Roohollah_Jafari; Borgheai, Seyyed_Bahram; McLinden, John; Shahriari, Yalda (February 2021, Biomedical Optics Express)

   Multimodal data fusion is one of the current primary neuroimaging research directions to overcome the fundamental limitations of individual modalities by exploiting complementary information from different modalities. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are especially compelling modalities due to their potentially complementary features reflecting the electro-hemodynamic characteristics of neural responses. However, the current multimodal studies lack a comprehensive systematic approach to properly merge the complementary features from their multimodal data. Identifying a systematic approach to properly fuse EEG-fNIRS data and exploit their complementary potential is crucial in improving performance. This paper proposes a framework for classifying fused EEG-fNIRS data at the feature level, relying on a mutual information-based feature selection approach with respect to the complementarity between features. The goal is to optimize the complementarity, redundancy and relevance between multimodal features with respect to the class labels as belonging to a pathological condition or healthy control. Nine amyotrophic lateral sclerosis (ALS) patients and nine controls underwent multimodal data recording during a visuo-mental task. Multiple spectral and temporal features were extracted and fed to a feature selection algorithm followed by a classifier, which selected the optimized subset of features through a cross-validation process. The results demonstrated considerably improved hybrid classification performance compared to the individual modalities and compared to conventional classification without feature selection, suggesting a potential efficacy of our proposed framework for wider neuro-clinical applications. 

   more » « less
3. Ensemble Machine Learning for Alzheimer’s disease Classification from Retinal Vasculature

   Lin, Hely; Fang, Ruogu (October 2021, Biomedical Engineering Society Annual Meeting)

   null (Ed.)

   Introduction: Alzheimer’s disease (AD) causes progressive irreversible cognitive decline and is the leading cause of dementia. Therefore, a timely diagnosis is imperative to maximize neurological preservation. However, current treatments are either too costly or limited in availability. In this project, we explored using retinal vasculature as a potential biomarker for early AD diagnosis. This project focuses on stage 3 of a three-stage modular machine learning pipeline which consisted of image quality selection, vessel map generation, and classification [1]. The previous model only used support vector machine (SVM) to classify AD labels which limited its accuracy to 82%. In this project, random forest and gradient boosting were added and, along with SVM, combined into an ensemble classifier, raising the classification accuracy to 89%. Materials and Methods: Subjects classified as AD were those who were diagnosed with dementia in “Dementia Outcome: Alzheimer’s disease” from the UK Biobank Electronic Health Records. Five control groups were chosen with a 5:1 ratio of control to AD patients where the control patients had the same age, gender, and eye side image as the AD patient. In total, 122 vessel images from each group (AD and control) were used. The vessel maps were then segmented from fundus images through U-net. A t-test feature selection was first done on the training folds and the selected features was fed into the classifiers with a p-value threshold of 0.01. Next, 20 repetitions of 5-fold cross validation were performed where the hyperparameters were solely tuned on the training data. An ensemble classifier consisting of SVM, gradient boosting tree, and random forests was built and the final prediction was made through majority voting and evaluated on the test set. Results and Discussion: Through ensemble classification, accuracy increased by 4-12% relative to the individual classifiers, precision by 9-15%, sensitivity by 2-9%, specificity by at least 9-16%, and F1 score by 712%. Conclusions: Overall, a relatively high classification accuracy was achieved using machine learning ensemble classification with SVM, random forest, and gradient boosting. Although the results are very promising, a limitation of this study is that the requirement of needing images of sufficient quality decreased the amount of control parameters that can be implemented. However, through retinal vasculature analysis, this project shows machine learning’s high potential to be an efficient, more cost-effective alternative to diagnosing Alzheimer’s disease. Clinical Application: Using machine learning for AD diagnosis through retinal images will make screening available for a broader population by being more accessible and cost-efficient. Mobile device based screening can also be enabled at primary screening in resource-deprived regions. It can provide a pathway for future understanding of the association between biomarkers in the eye and brain. 

   more » « less
4. Crossed Eyes: Domain Adaptation for Gaze-Based Mind Wandering Models

   https://doi.org/10.1145/3448017.3457386  

   Bixler, Robert E. (May 2021, ETRA '21 Full Papers: ACM Symposium on Eye Tracking Research and Applications)

   null (Ed.)

   The effectiveness of user interfaces are limited by the tendency for the human mind to wander. Intelligent user interfaces can combat this by detecting when mind wandering occurs and attempting to regain user attention through a variety of intervention strategies. However, collecting data to build mind wandering detection models can be expensive, especially considering the variety of media available and potential differences in mind wandering across them. We explored the possibility of using eye gaze to build cross-domain models of mind wandering where models trained on data from users in one domain are used for different users in another domain. We built supervised classification models using a dataset of 132 users whose mind wandering reports were collected in response to thought-probes while they completed tasks from seven different domains for six minutes each (five domains are investigated here: Illustrated Text, Narrative Film, Video Lecture, Naturalistic Scene, and Reading Text). We used global eye gaze features to build within- and cross- domain models using 5-fold user-independent cross validation. The best performing within-domain models yielded AUROCs ranging from .57 to .72, which were comparable for the cross-domain models (AUROCs of .56 to .68). Models built from coarse-grained locality features capturing the spatial distribution of gaze resulted in slightly better transfer on average (transfer ratios of .61 vs .54 for global models) due to improved performance in certain domains. Instance-based and feature-level domain adaptation did not result in any improvements in transfer. We found that seven gaze features likely contributed to transfer as they were among the top ten features for at least four domains. Our results indicate that gaze features are suitable for domain adaptation from similar domains, but more research is needed to improve domain adaptation between more dissimilar domains. 

   more » « less
5. Merging fNIRS-EEG Brain Monitoring and Body Motion Capture to Distinguish Parkinson’s Disease

   https://doi.org/10.1109/TNSRE.2020.2987888  

   Abtahi, Mohammadreza; Borgheai, Seyed Bahram; Jafari, Roohollah; Constant, Nicholas; Diouf, Rassoul; Shahriari, Yalda; Mankodiya, Kunal (January 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering)

   Functional connectivity between the brain and body kinematics has largely not been investigated due to the requirement of motionlessness in neuroimaging techniques such as functional magnetic resonance imaging (fMRI). However, this connectivity is disrupted in many neurodegenerative disorders, including Parkinson’s Disease (PD), a neurological progressive disorder characterized by movement symptoms including slowness of movement, stiffness, tremors at rest, and walking and standing instability. In this study, brain activity is recorded through functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), and body kinematics were captured by a motion capture system (Mocap) based on an inertial measurement unit (IMU) for gross movements (large movements such as limb kinematics), and the WearUp glove for fine movements (small range movements such as finger kinematics). PD and neurotypical (NT) participants were recruited to perform 8 different movement tasks. The recorded data from each modality have been analyzed individually, and the processed data has been used for classification between the PD and NT groups. The average changes in oxygenated hemoglobin (HbO2) from fNIRS, EEG power spectral density in the Theta, Alpha, and Beta bands, acceleration vector from Mocap, and normalized WearUp flex sensor data were used for classification. 12 different support vector machine (SVM) classifiers have been used on different datasets such as only fNIRS data, only EEG data, hybrid fNIRS/EEG data, and all the fused data for two classification scenarios: classifying PD and NT based on individual activities, and all activity data fused together. The PD and NT group could be distinguished with more than 83% accuracy for each individual activity. For all the fused data, the PD and NT groups are classified with 81.23%, 92.79%, 92.27%, and 93.40% accuracy for the fNIRS only, EEG only, hybrid fNIRS/EEG, and all fused data, respectively. The results indicate that the overall performance of classification in distinguishing PD and NT groups improves when using both brain and body data. 

   more » « less