skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: The Perils and Pitfalls of Block Design for EEG Classification Experiments
A recent paper [1] claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to employ a representation derived from this processing to construct a novel object classifier. That paper, together with a series of subsequent papers [2] , [3] , [4] , [5] , [6] , [7] , [8] , claims to achieve successful results on a wide variety of computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they employ, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that are known to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to classifying arbitrary temporal artifacts of the data instead of stimulus-related activity. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the reported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit from the brain-derived representation. Together, our results illustrate the far-reaching implications of the temporal autocorrelations that exist in all neuroimaging data for classification experiments. Further, our results calibrate the underlying difficulty of the tasks involved and caution against overly optimistic, but incorrect, claims to the contrary.  more » « less
Award ID(s):
1734938 1522954
PAR ID:
10284053
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Volume:
43
Issue:
1
ISSN:
0162-8828
Page Range / eLocation ID:
316-333
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR))
    null (Ed.)
    New results suggest strong limits to the feasibility of object classification from human brain activity evoked by image stimuli, as measured through EEG. Considerable prior work suffers from a confound between the stimulus class and the time since the start of the experiment. A prior attempt to avoid this confound using randomized trials was unable to achieve results above chance in a statistically significant fashion when the data sets were of the same size as the original experiments. Here, we attempt object classification from EEG using an array of methods that are representative of the state-of-the-art, with a far larger (20x) dataset of randomized EEG trials, 1,000 stimulus presentations of each of forty classes, all from a single subject. To our knowledge, this is the largest such EEG data-collection effort from a single subject and is at the bounds of feasibility. We obtain classification accuracy that is marginally above chance and above chance in a statistically significant fashion, and further assess how accuracy depends on the classifier used, the amount of training data used, and the number of classes. Reaching the limits of data collection with only marginally above-chance performance suggests that the prevailing literature substantially exaggerates the feasibility of object classification from EEG. 
    more » « less
  2. (, Journal of Computational Neuroscience)
    Objectively differentiating patient mental states based on electrical activity, as opposed to overt behavior, is a fundamental neuroscience problem with medical applications, such as identifying patients in locked-in state vs. coma. Electroencephalography (EEG), which detects millisecond-level changes in brain activity across a range of frequencies, allows for assessment of external stimulus processing by the brain in a non-invasive manner. We applied machine learning methods to 26-channel EEG data of 24 fluent Deaf signers watching videos of sign language sentences (comprehension condition), and the same videos reversed in time (non-comprehension condition), to objectively separate vision-based high-level cognition states. While spectrotemporal parameters of the stimuli were identical in comprehension vs. non-comprehension conditions, the neural responses of participants varied based on their ability to linguistically decode visual data. We aimed to determine which subset of parameters (specific scalp regions or frequency ranges) would be necessary and sufficient for high classification accuracy of comprehension state. Optical flow, characterizing distribution of velocities of objects in an image, was calculated for each pixel of stimulus videos using MATLAB Vision toolbox. Coherence between optical flow in the stimulus and EEG neural response (per video, per participant) was then computed using canonical component analysis with NoiseTools toolbox. Peak correlations were extracted for each frequency for each electrode, participant, and video. A set of standard ML algorithms were applied to the entire dataset (26 channels, frequencies from .2 Hz to 12.4 Hz, binned in 1 Hz increments), with consistent out-of-sample 100% accuracy for frequencies in .2-1 Hz range for all regions, and above 80% accuracy for frequencies < 4 Hz. Sparse Optimal Scoring (SOS) was then applied to the EEG data to reduce the dimensionality of the features and improve model interpretability. SOS with elastic-net penalty resulted in out-of-sample classification accuracy of 98.89%. The sparsity pattern in the model indicated that frequencies between 0.2–4 Hz were primarily used in the classification, suggesting that underlying data may be group sparse. Further, SOS with group lasso penalty was applied to regional subsets of electrodes (anterior, posterior, left, right). All trials achieved greater than 97% out-of-sample classification accuracy. The sparsity patterns from the trials using 1 Hz bins over individual regions consistently indicated frequencies between 0.2–1 Hz were primarily used in the classification, with anterior and left regions performing the best with 98.89% and 99.17% classification accuracy, respectively. While the sparsity pattern may not be the unique optimal model for a given trial, the high classification accuracy indicates that these models have accurately identified common neural responses to visual linguistic stimuli. Cortical tracking of spectro-temporal change in the visual signal of sign language appears to rely on lower frequencies proportional to the N400/P600 time-domain evoked response potentials, indicating that visual language comprehension is grounded in predictive processing mechanisms. 
    more » « less
  3. ; (, Journal of Signal Processing Systems)
    This paper investigates scalp electroencephalogram (EEG) data from 14 subjects with unilateral prefrontal cortex (pFC) lesions and 20 healthy controls during lateral visuospatial working memory (WM) tasks. The goal is to differentiate the brain networks involved in WM processing between these groups. The EEG recordings are transformed into graph signals, with proximity-weighted brain connectivity measures as edges and centrality measures as nodal features. Graph convolutional network (GCN) layers are used for feature representation, followed by a fully connected layer for classification. The GCN-based model effectively handles nine classification tasks, proving that graph-based network representation is versatile for describing brain interactions. The sparse MI-GCI-based graph model’s accuracy effectively captures the functional segregation of distinct WM tasks. The classifier using mutual information-guided Granger causality index (MI-GCI) with 20% of top edges matched prior classification performance with 67% fewer parameters and 80% less graph density, identifying the correct class of all 34 subjects in group identification using leave-one-out cross-validation and two-thirds majority voting. 
    more » « less
  4. ; ; ; ; ; ; ; ; ; ; et al (, Epilepsia)
    Abstract ObjectiveAnterior temporal lobectomy (ATL) is a widely performed and successful intervention for drug‐resistant temporal lobe epilepsy (TLE). However, up to one third of patients experience seizure recurrence within 1 year after ATL. Despite the extensive literature on presurgical electroencephalography (EEG) and magnetic resonance imaging (MRI) abnormalities to prognosticate seizure freedom following ATL, the value of quantitative analysis of visually reviewed normal interictal EEG in such prognostication remains unclear. In this retrospective multicenter study, we investigate whether machine learning analysis of normal interictal scalp EEG studies can inform the prediction of postoperative seizure freedom outcomes in patients who have undergone ATL. MethodsWe analyzed normal presurgical scalp EEG recordings from 41 Mayo Clinic (MC) and 23 Cleveland Clinic (CC) patients. We used an unbiased automated algorithm to extract eyes closed awake epochs from scalp EEG studies that were free of any epileptiform activity and then extracted spectral EEG features representing (a) spectral power and (b) interhemispheric spectral coherence in frequencies between 1 and 25 Hz across several brain regions. We analyzed the differences between the seizure‐free and non–seizure‐free patients and employed a Naïve Bayes classifier using multiple spectral features to predict surgery outcomes. We trained the classifier using a leave‐one‐patient‐out cross‐validation scheme within the MC data set and then tested using the out‐of‐sample CC data set. Finally, we compared the predictive performance of normal scalp EEG‐derived features against MRI abnormalities. ResultsWe found that several spectral power and coherence features showed significant differences correlated with surgical outcomes and that they were most pronounced in the 10–25 Hz range. The Naïve Bayes classification based on those features predicted 1‐year seizure freedom following ATL with area under the curve (AUC) values of 0.78 and 0.76 for the MC and CC data sets, respectively. Subsequent analyses revealed that (a) interhemispheric spectral coherence features in the 10–25 Hz range provided better predictability than other combinations and (b) normal scalp EEG‐derived features provided superior and potentially distinct predictive value when compared with MRI abnormalities (>10% higher F1 score). SignificanceThese results support that quantitative analysis of even a normal presurgical scalp EEG may help prognosticate seizure freedom following ATL in patients with drug‐resistant TLE. Although the mechanism for this result is not known, the scalp EEG spectral and coherence properties predicting seizure freedom may represent activity arising from the neocortex or the networks responsible for temporal lobe seizure generation within vs outside the margins of an ATL. 
    more » « less
  5. ; (, Journal of Cognitive Neuroscience)
    Serial dependence—an attractive perceptual bias whereby a current stimulus is perceived to be similar to previously seen ones—is thought to represent the process that facilitates the stability and continuity of visual perception. Recent results demonstrate a neural signature of serial dependence in numerosity perception, emerging very early in the time course during perceptual processing. However, whether such a perceptual signature is retained after the initial processing remains unknown. Here, we address this question by investigating the neural dynamics of serial dependence using a recently developed technique that allowed a reactivation of hidden memory states. Participants performed a numerosity discrimination task during EEG recording, with task-relevant dot array stimuli preceded by a task-irrelevant stimulus inducing serial dependence. Importantly, the neural network storing the representation of the numerosity stimulus was perturbed (or pinged) so that the hidden states of that representation can be explicitly quantified. The results first show that a neural signature of serial dependence emerges early in the brain signals, starting soon after stimulus onset. Critical to the central question, the pings at a later latency could successfully reactivate the biased representation of the initial stimulus carrying the signature of serial dependence. These results provide one of the first pieces of empirical evidence that the biased neural representation of a stimulus initially induced by serial dependence is preserved throughout a relatively long period. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email