Decoding auditory stimulus from neural activity can enable neuroprosthetics and direct communication with the brain. Some recent studies have shown successful speech decoding from intracranial recording using deep learning models. However, scarcity of training data leads to low quality speech reconstruction which prevents a complete brain-computer-interface (BCI) application. In this work, we propose a transfer learning approach with a pre-trained GAN to disentangle representation and generation layers for decoding. We first pre-train a generator to produce spectrograms from a representation space using a large corpus of natural speech data. With a small amount of paired data containing the stimulus speech and corresponding ECoG signals, we then transfer it to a bigger network with an encoder attached before, which maps the neural signal to the representation space. To further improve the network generalization ability, we introduce a Gaussian prior distribution regularizer on the latent representation during the transfer phase. With at most 150 training samples for each tested subject, we achieve a state-of-the-art decoding performance. By visualizing the attention mask embedded in the encoder, we observe brain dynamics that are consistent with findings from previous studies investigating dynamics in the superior temporal gyrus (STG), pre-central gyrus (motor) and inferior frontal gyrus (IFG). Our findings demonstrate a high reconstruction accuracy using deep learning networks together with the potential to elucidate interactions across different brain regions during a cognitive task.
more »
« less
Differential activation of a frontoparietal network explains population-level differences in statistical learning from speech
People of all ages display the ability to detect and learn from patterns in seemingly random stimuli. Referred to as statistical learning (SL), this process is particularly critical when learning a spoken language, helping in the identification of discrete words within a spoken phrase. Here, by considering individual differences in speech auditory–motor synchronization, we demonstrate that recruitment of a specific neural network supports behavioral differences in SL from speech. While independent component analysis (ICA) of fMRI data revealed that a network of auditory and superior pre/motor regions is universally activated in the process of learning, a frontoparietal network is additionally and selectively engaged by only some individuals (high auditory–motor synchronizers). Importantly, activation of this frontoparietal network is related to a boost in learning performance, and interference with this network via articulatory suppression (AS; i.e., producing irrelevant speech during learning) normalizes performance across the entire sample. Our work provides novel insights on SL from speech and reconciles previous contrasting findings. These findings also highlight a more general need to factor in fundamental individual differences for a precise characterization of cognitive phenomena.
more »
« less
- Award ID(s):
- 2043717
- NSF-PAR ID:
- 10410147
- Editor(s):
- Rushworth, Matthew F.
- Date Published:
- Journal Name:
- PLOS Biology
- Volume:
- 20
- Issue:
- 7
- ISSN:
- 1545-7885
- Page Range / eLocation ID:
- e3001712
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Statistical learning (SL), the ability to pick up patterns in sensory input, serves as one of the building blocks of language acquisition. Although SL has been studied extensively in developmental dyslexia (DD), much less is known about the way SL evolves over time. The handful of studies examining this question were all limited to the acquisition of motor sequential knowledge or highly learned segmented linguistic units. Here we examined memory consolidation of statistical regularities in adults with DD and typically developed (TD) readers by using auditory SL requiring the segmentation of units from continuous input, which represents one of the earliest learning challenges in language acquisition. DD and TD groups were exposed to tones in a probabilistically determined sequential structure varying in difficulty and subsequently tested for recognition of novel short sequences that adhered to this statistical pattern in immediate and delayed-recall sessions separated by a night of sleep. SL performance of the DD group at the easy and hard difficulty levels was poorer than that of the TD group in the immediate-recall session. Importantly, DD participants showed a significant overnight deterioration in SL performance at the medium difficulty level compared to TD, who instead showed overnight stabilization of the learned information. These findings imply that SL difficulties in DD may arise not only from impaired initial learning but also due to a failure to consolidate statistically structured information into long-term memory. We hypothesize that these deficits disrupt the typical course of language acquisition in those with DD.more » « less
-
Stuttering is a neurodevelopmental speech disorder associated with motor timing that differs from non-stutterers. While neurodevelopmental disorders impacted by timing are associated with compromised auditory-motor integration and interoception, the interplay between those abilities and stuttering remains unexplored. Here, we studied the relationships between speech auditory-motor synchronization (a proxy for auditory-motor integration), interoceptive awareness, and self-reported stuttering severity using remotely delivered assessments. Results indicate that in general, stutterers and non-stutterers exhibit similar auditory-motor integration and interoceptive abilities. However, while speech auditory-motor synchrony (i.e., integration) and interoceptive awareness were not related, speech synchrony was inversely related to the speaker’s perception of stuttering severity as perceived by others, and interoceptive awareness was inversely related to self-reported stuttering impact. These findings support claims that stuttering is a heterogeneous, multi-faceted disorder such that uncorrelated auditory-motor integration and interoception measurements predicted different aspects of stuttering, suggesting two unrelated sources of timing differences associated with the disorder.more » « less
-
more » « less
Abstract Human populations show large individual differences in math performance and math learning abilities. Early math skill acquisition is critical for providing the foundation for higher quantitative skill acquisition and succeeding in modern society. However, the neural bases underlying individual differences in math competence remain unclear. Modern neuroimaging techniques allow us to not only identify distinct local cortical regions but also investigate large-scale neural networks underlying math competence both structurally and functionally. To gain insights into the neural bases of math competence, this review provides an overview of the structural and functional neural markers for math competence in both typical and atypical populations of children and adults. Although including discussion of arithmetic skills in children, this review primarily focuses on the neural markers associated with complex math skills. Basic number comprehension and number comparison skills are outside the scope of this review. By synthesizing current research findings, we conclude that neural markers related to math competence are not confined to one particular region; rather, they are characterized by a distributed and interconnected network of regions across the brain, primarily focused on frontal and parietal cortices. Given that human brain is a complex network organized to minimize the cost of information processing, an efficient brain is capable of integrating information from different regions and coordinating the activity of various brain regions in a manner that maximizes the overall efficiency of the network to achieve the goal. We end by proposing that frontoparietal network efficiency is critical for math competence, which enables the recruitment of task-relevant neural resources and the engagement of distributed neural circuits in a goal-oriented manner. Thus, it will be important for future studies to not only examine brain activation patterns of discrete regions but also examine distributed network patterns across the brain, both structurally and functionally.
-
more » « less
Abstract The existence of a neural representation for whole words (i.e., a lexicon) is a common feature of many models of speech processing. Prior studies have provided evidence for a visual lexicon containing representations of whole written words in an area of the ventral visual stream known as the visual word form area. Similar experimental support for an auditory lexicon containing representations of spoken words has yet to be shown. Using functional magnetic resonance imaging rapid adaptation techniques, we provide evidence for an auditory lexicon in the auditory word form area in the human left anterior superior temporal gyrus that contains representations highly selective for individual spoken words. Furthermore, we show that familiarization with novel auditory words sharpens the selectivity of their representations in the auditory word form area. These findings reveal strong parallels in how the brain represents written and spoken words, showing convergent processing strategies across modalities in the visual and auditory ventral streams.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1371/journal.pbio.3001712
