skip to main content

Attention:

The NSF Public Access Repository (PAR) system and access will be unavailable from 11:00 PM ET on Friday, December 13 until 2:00 AM ET on Saturday, December 14 due to maintenance. We apologize for the inconvenience.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Dynamic Mode Decomposition with Control as a Model of Multimodal Behavioral Coordination
Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination.  more » « less
Award ID(s):
1706964
PAR ID:
10354821
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ( , Infancy)
    Abstract

    Parental responsiveness to infant behaviors is a strong predictor of infants' language and cognitive outcomes. The mechanisms underlying this effect, however, are relatively unknown. We examined the effects of parent speech on infants' visual attention, manual actions, hand‐eye coordination, and dyadic joint attention during parent‐infant free play. We report on two studies that used head‐mounted eye trackers in increasingly naturalistic laboratory environments. In Study 1, 12‐to‐24‐month‐old infants and their parents played on the floor of a seminaturalistic environment with 24 toys. In Study 2, a different sample of dyads played in a home‐like laboratory with 10 toys and no restrictions on their movement. In both studies, we present evidence that responsive parent speech extends the duration of infants' multimodal attention. This social “boost” of parent speech impacts multiple behaviors that have been linked to later outcomes—visual attention, manual actions, hand‐eye coordination, and joint attention. Further, the amount that parents talked during the interaction was negatively related to the effects of parent speech on infant attention. Together, these results provide evidence of a trade‐off between quantity of speech and its effects, suggesting multiple pathways through which parents impact infants' multimodal attention to shape the moment‐by‐moment dynamics of an interaction.

     
    more » « less
  2. ; ( , PloS one)
    This study investigates the presence of dynamical patterns of interpersonal coordination in extended deceptive conversations across multimodal channels of behavior. Using a novel "devil’s advocate" paradigm, we experimentally elicited deception and truth across topics in which conversational partners either agreed or disagreed, and where one partner was surreptitiously asked to argue an opinion opposite of what he or she really believed. We focus on interpersonal coordination as an emergent behavioral signal that captures interdependencies between conversational partners, both as the coupling of head movements over the span of milliseconds, measured via a windowed lagged cross correlation (WLCC) technique, and more global temporal dependencies across speech rate, using cross recurrence quantification analysis (CRQA). Moreover, we considered how interpersonal coordination might be shaped by strategic, adaptive conversational goals associated with deception. We found that deceptive conversations displayed more structured speech rate and higher head movement coordination, the latter with a peak in deceptive disagreement conversations. Together the results allow us to posit an adaptive account, whereby interpersonal coordination is not beholden to any single functional explanation, but can strategically adapt to diverse conversational demands. 
    more » « less
  3. ; ; ; ( , Infancy)

    The identification of infants who are most susceptible to both negative and positive social environments is critical for understanding early behavioral development. This study longitudinally assessed the interactive effects of infant vagal tone (respiratory sinus arrhythmia;RSA) and maternal social support on behavioral problems and competence among 322 low‐income Mexican American mother–infant dyads (infants: 54.1% female) and explored sex differences. InfantRSAwas calculated from restingHRdata at 6 weeks of age. Mothers reported on general social support, partner support, and family support at 6 months, and infant behavioral problems and competence at 1 year. Two‐way interactions (RSA × support source) were evaluated to predict behavioral problems and competence, adjusting for covariates. Results indicated higher competence among infants with lowerRSAwhose mothers reported higher general support or higher partner support. Interactive effects on behavior problems ofRSAwith maternal partner or family support were only found for female infants: Girls with higherRSAshowed more behavior problems when mothers reported low support, but fewer problem levels in the context of high support. Our results suggest that infantRSAis an important moderator of the effects of the early social environment on early development.

     
    more » « less
  4. ; ; ; ; ( , Developmental Psychobiology)
    Abstract

    While experimental stress paradigms of infants (arm restraint; the Still‐Face) are powerful tools for infant research, no study has experimentally stressed mothers to observe its independent effects on infant stress regulation. Extant caretaker/maternal stress studies essentially are correlational and confounded by other conditions (e.g., depression). Here, we present a standard procedure, the Caregiver Acute Stress Paradigm (CASP), for stressing mothers duringen faceinteractions with their infants. We hypothesized that infants of the stressed mothers would be more distressed than infants of non‐stressed mothers. A total of 106 four‐month‐old infants and their mothers were randomly assigned to the experimental stress or non‐stress manipulation. Confirming our hypothesis, infants of the stressed mothers were significantly more likely to become distressed and require terminating the procedure. While objective ratings of maternal behavior showed no difference between groups, mother in the stress condition self‐rated the episode following the caretaker stress significantly lower than mothers in the non‐stress group. The self‐ratings in the maternal stress‐group were reflected in infant cortisol. The findings indicate that CASP is an effective experimental paradigm for exploring the independent effects of an acute stress on caretakers, including effects of conditions, such as poverty or mental illness.

     
    more » « less
  5. ; ; ( , Developmental Cognitive Neuroscience)
    Van_Den_Heuvel, M ; Wass, S V (Ed.)
    During everyday interactions, mothers and infants achieve behavioral synchrony at multiple levels. The ebb-and-flow of mother-infant physical proximity may be a central type of synchrony that establishes a common ground for infant-mother interaction. However, the role of proximity in language exchanges is relatively unstudied, perhaps because structured tasks—the common setup for observing infant-caregiver interactions—establish proximity by design. We videorecorded 100 mothers (U.S. Hispanic N =50, U.S. Non-Hispanic N =50) and their 13- to 23-month-old infants during natural activity at home (1-to-2 h per dyad), transcribed mother and infant speech, and coded proximity continuously (i.e., infants and mother within arms reach). In both samples, dyads entered proximity in a bursty temporal pattern, with bouts of proximity interspersed with bouts of physical distance. As hypothesized, Non-Hispanic and Hispanic mothers produced more words and a greater variety of words when within arms reach than out of arms reach. Similarly, infants produced more utterances that contained words when close to mother than when not. However, infants babbled equally often regardless of proximity, generating abundant opportunities to play with sounds. Physical proximity expands opportunities for language exchanges and infants’ communicative word use, although babies accumulate massive practice babbling even when caregivers are not proximal. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email