skip to main content

Explore Research Products in the NSF-PAR

Title: Dynamic Mode Decomposition with Control as a Model of Multimodal Behavioral Coordination
Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination.  more » « less
Award ID(s):
1706964
NSF-PAR ID:
10354821
Author(s) / Creator(s):
; ; ; ; ;
Date Published:
Journal Name:
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ( , Infancy)
    Abstract

    Parental responsiveness to infant behaviors is a strong predictor of infants' language and cognitive outcomes. The mechanisms underlying this effect, however, are relatively unknown. We examined the effects of parent speech on infants' visual attention, manual actions, hand‐eye coordination, and dyadic joint attention during parent‐infant free play. We report on two studies that used head‐mounted eye trackers in increasingly naturalistic laboratory environments. In Study 1, 12‐to‐24‐month‐old infants and their parents played on the floor of a seminaturalistic environment with 24 toys. In Study 2, a different sample of dyads played in a home‐like laboratory with 10 toys and no restrictions on their movement. In both studies, we present evidence that responsive parent speech extends the duration of infants' multimodal attention. This social “boost” of parent speech impacts multiple behaviors that have been linked to later outcomes—visual attention, manual actions, hand‐eye coordination, and joint attention. Further, the amount that parents talked during the interaction was negatively related to the effects of parent speech on infant attention. Together, these results provide evidence of a trade‐off between quantity of speech and its effects, suggesting multiple pathways through which parents impact infants' multimodal attention to shape the moment‐by‐moment dynamics of an interaction.

     
    more » « less
  2. ; ; ; ( , Infancy)

    The identification of infants who are most susceptible to both negative and positive social environments is critical for understanding early behavioral development. This study longitudinally assessed the interactive effects of infant vagal tone (respiratory sinus arrhythmia;RSA) and maternal social support on behavioral problems and competence among 322 low‐income Mexican American mother–infant dyads (infants: 54.1% female) and explored sex differences. InfantRSAwas calculated from restingHRdata at 6 weeks of age. Mothers reported on general social support, partner support, and family support at 6 months, and infant behavioral problems and competence at 1 year. Two‐way interactions (RSA × support source) were evaluated to predict behavioral problems and competence, adjusting for covariates. Results indicated higher competence among infants with lowerRSAwhose mothers reported higher general support or higher partner support. Interactive effects on behavior problems ofRSAwith maternal partner or family support were only found for female infants: Girls with higherRSAshowed more behavior problems when mothers reported low support, but fewer problem levels in the context of high support. Our results suggest that infantRSAis an important moderator of the effects of the early social environment on early development.

     
    more » « less
  3. ; ; ; ( , American Journal of Biological Anthropology)
    Abstract Objectives

    Folate is an essential nutrient fundamental to human growth and development. Human milk maintains high folate content across the maternal folate status range, suggesting buffering of milk folate with prioritized delivery to milk at the expense of maternal depletion. We investigated whether and how the extent of this buffering may diminish under prolonged nutritional and/or disease stress, while taking into consideration infants' varying vulnerability to malnutrition‐related morbidity/mortality.

     Methods

    A cross‐sectional study analyzed milk specimens from northern Kenyan mothers (n = 203), surveyed during a historic drought and ensuing food shortage. Multiple regression models for folate receptor‐α (FOLR1) in milk were constructed. Predictors included maternal underweight (BMI < 18.5), iron‐deficiency anemia (hemoglobin <12 g/dl and dried‐blood‐spot transferrin receptor >5 mg/L), folate deficiency (hyperhomocysteinemia, homocysteine >12 or 14 μmol/L), inflammation (serum C‐reactive protein >5 mg/L), infant age and sex, and mother‐infant interactions.

     Results

    In adjusted models, milk FOLR1 was unassociated with maternal underweight, iron‐deficiency anemia and inflammation. FOLR1 was positively associated with maternal folate deficiency, and inversely associated with infant age. There was interaction between infant age and maternal underweight, and between infant sex and maternal folate deficiency, predicting complex changes in FOLR1.

     Conclusions

    Our results suggest that mothers buffer milk folate against their own nutritional stress even during a prolonged drought; however, the extent of this buffering may vary with infant age, and, among folate‐deficient mothers, with infant sex. Future research is needed to better understand this variability in maternal buffering of milk folate and how it relates to folate status in nursing infants.

     
    more » « less
  4. ; ( , PloS one)
    This study investigates the presence of dynamical patterns of interpersonal coordination in extended deceptive conversations across multimodal channels of behavior. Using a novel "devil’s advocate" paradigm, we experimentally elicited deception and truth across topics in which conversational partners either agreed or disagreed, and where one partner was surreptitiously asked to argue an opinion opposite of what he or she really believed. We focus on interpersonal coordination as an emergent behavioral signal that captures interdependencies between conversational partners, both as the coupling of head movements over the span of milliseconds, measured via a windowed lagged cross correlation (WLCC) technique, and more global temporal dependencies across speech rate, using cross recurrence quantification analysis (CRQA). Moreover, we considered how interpersonal coordination might be shaped by strategic, adaptive conversational goals associated with deception. We found that deceptive conversations displayed more structured speech rate and higher head movement coordination, the latter with a peak in deceptive disagreement conversations. Together the results allow us to posit an adaptive account, whereby interpersonal coordination is not beholden to any single functional explanation, but can strategically adapt to diverse conversational demands. 
    more » « less
  5. ; ; ( , Developmental Psychobiology)
    Abstract

    According to polyvagal theory, rapid modulation of the vagal brake develops early in infancy and supports social interactions. Despite being viewed as a dynamic system, researchers typically assess vagal regulation using global measures of respiratory sinus arrhythmia (RSA; an index of vagal tone). This study sought to capture the dynamic property of RSA and evaluate individual differences in within‐infant RSA responsivity during mother–infant interaction. RSA was evaluated in a sample of 135 6‐month‐old Mexican‐American infants during a 5‐min free play task. Mothers reported on their children's behavioral problems and competence at 18 months using the Brief Infant‐Toddler Social and Emotional Assessment. Time‐varying estimates of infant RSA during the interaction were obtained using a multiple window technique and spectrogram analysis. Using structural equation modeling, we evaluated whether within‐infantSDof RSA predicted infants’ behavioral problems and competence at 18 months, after adjusting for infants’ mean RSA and covariates. Greater within‐infantSDof RSA predicted more behavior problems at 18 months. This study demonstrates that assessing intra‐individual variability in RSA, or the extent to which infants fluctuate around their average level of RSA during a task, enhances our ability to test polyvagal theory's central tenet: vagal regulation supports well‐regulated social interaction.

     
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email