An official website of the United States government
Humans rarely speak without producing co-speech gestures of the hands, head, and other parts of the body. Co-speech gestures are also highly restricted in how they are timed with speech, typically synchronizing with prosodically-prominent syllables. What functional principles underlie this relationship? Here, we examine how the production of co-speech manual gestures influences spatiotemporal patterns of the oral articulators during speech production. We provide novel evidence that words uttered with accompanying co-speech gestures are produced with more extreme tongue and jaw displacement, and that presence of a co-speech gesture contributes to greater temporal stability of oral articulatory movements. This effect–which we term coupling enhancement–differs from stress-based hyperarticulation in that differences in articulatory magnitude are not vowel-specific in their patterning. Speech and gesture synergies therefore constitute an independent variable to consider when modeling the effects of prosodic prominence on articulatory patterns. Our results are consistent with work in language acquisition and speech-motor control suggesting that synchronizing speech to gesture can entrain acoustic prominence.
more »
« less
This content will become publicly available on December 1, 2025
Deliberate synchronization of speech and gesture: effects of neurodiversity and development
Abstract The production of speech and gesture is exquisitely temporally coordinated. In autistic individuals, speech-gesture synchrony during spontaneous discourse is disrupted. To evaluate whether this asynchrony reflects motor coordination versus language production processes, the current study examineddeliberatelyperformed hand movements during speech in youth with autism spectrum disorder (ASD) compared to neurotypical youth. Neurotypical adult performance provided a mature baseline. Participants read aloud rhythmic nursery rhymes, while producing a beat-like hand movement. An automated pixel-change video measure identified kinematic peaks; using smoothed acoustic envelope analyses, we identified peaks in speech. Results indicated few diagnostic group differences in explicit speech-movement coordination, although adolescent performance differed from adults. Adults demonstrated higher tempo and greater rhythmicity in their coordination; this group difference suggests that the method is sufficiently subtle to reveal individual differences and that this form of complex coordination undergoes ongoing maturation beyond adolescence. The sample is small, and thus results are necessarily preliminary. In the context of prior speech-gesture coordination studies, these findings of intact synchrony are consistent with the hypothesis that it is the demands of discourse planning, rather than motor coordination, that have led to prior findings of asynchrony during spontaneous speech; this possibility awaits future research.
more »
« less
- Award ID(s):
- 2152202
- PAR ID:
- 10613713
- Publisher / Repository:
- Cambridge
- Date Published:
- Journal Name:
- Language and Cognition
- Volume:
- 16
- Issue:
- 4
- ISSN:
- 1866-9808
- Page Range / eLocation ID:
- 1812 to 1833
- Subject(s) / Keyword(s):
- autism spectrum disorder beat gesture perception-action coordination praxis motor control
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Stuttering is a neurodevelopmental speech disorder associated with motor timing that differs from non-stutterers. While neurodevelopmental disorders impacted by timing are associated with compromised auditory-motor integration and interoception, the interplay between those abilities and stuttering remains unexplored. Here, we studied the relationships between speech auditory-motor synchronization (a proxy for auditory-motor integration), interoceptive awareness, and self-reported stuttering severity using remotely delivered assessments. Results indicate that in general, stutterers and non-stutterers exhibit similar auditory-motor integration and interoceptive abilities. However, while speech auditory-motor synchrony (i.e., integration) and interoceptive awareness were not related, speech synchrony was inversely related to the speaker’s perception of stuttering severity as perceived by others, and interoceptive awareness was inversely related to self-reported stuttering impact. These findings support claims that stuttering is a heterogeneous, multi-faceted disorder such that uncorrelated auditory-motor integration and interoception measurements predicted different aspects of stuttering, suggesting two unrelated sources of timing differences associated with the disorder.more » « less
-
Abstract Teaching a new concept through gestures—hand movements that accompany speech—facilitates learning above‐and‐beyond instruction through speech alone (e.g., Singer & Goldin‐Meadow,). However, the mechanisms underlying this phenomenon are still under investigation. Here, we use eye tracking to explore one often proposed mechanism—gesture's ability to direct visual attention. Behaviorally, we replicate previous findings: Children perform significantly better on a posttest after learning through Speech+Gesture instruction than through Speech Alone instruction. Using eye tracking measures, we show that children who watch a math lesson with gesturedoallocate their visual attention differently from children who watch a math lesson without gesture—they look more to the problem being explained, less to the instructor, and are more likely to synchronize their visual attention with information presented in the instructor's speech (i.e.,follow along with speech) than children who watch the no‐gesture lesson. The striking finding is that, even though these looking patterns positively predict learning outcomes, the patterns do notmediatethe effects of training condition (Speech Alone vs. Speech+Gesture) on posttest success. We find instead a complex relation between gesture and visual attention in which gesturemoderatesthe impact of visual looking patterns on learning—following along with speechpredicts learning for children in the Speech+Gesture condition, but not for children in the Speech Alone condition. Gesture's beneficial effects on learning thus come not merely from its ability to guide visual attention, but also from its ability to synchronize with speech and affect what learners glean from that speech.more » « less
-
Examining age differences in motor learning using real-world tasks is often problematic due to task novelty and biomechanical confounds. Here, we investigated how children and adults acquire a novel motor skill in a virtual environment. Participants of three different age groups (9-year-olds, 12-year-olds, and adults) learned to use their upper body movements to control a cursor on a computer screen. Results showed that 9-year-old and 12-year-old children showed poorer ability to control the cursor at the end of practice. Critically, when we investigated the movement coordination, we found that the lower task performance of children was associated with limited exploration of their movement repertoire. These results reveal the critical role of motor exploration in understanding developmental differences in motor learning.more » « less
-
With few exceptions, most research in automated assessment of depression has considered only the patient’s behavior to the exclusion of the therapist’s behavior. We investigated the interpersonal coordination (synchrony) of head movement during patient-therapist clinical interviews. Participants with major depressive disorder were recorded in clinical interviews (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. For each session, patient and therapist 3D head movement was tracked from 2D videos. Head angles in the horizontal (pitch) and vertical (yaw) axes were used to measure head movement. Interpersonal coordination of head movement between patients and therapists was measured using windowed cross-correlation. Patterns of coordination in head movement were investigated using the peak picking algorithm. Changes in head movement coordination over the course of treatment were measured using a hierarchical linear model (HLM). The results indicated a strong effect for patient-therapist head movement synchrony. Within-dyad variability in head movement coordination was higher than between-dyad variability, meaning that differences over time in a dyad were higher as compared to the differences between dyads. Head movement synchrony did not change over the course of treatment. To the best of our knowledge, this study is the first attempt to analyze the mutual influence of patient-therapist head movement in relation to depression severity.more » « less
