This study investigates the presence of dynamical patterns of interpersonal coordination in extended deceptive conversations across multimodal channels of behavior. Using a novel "devil’s advocate" paradigm, we experimentally elicited deception and truth across topics in which conversational partners either agreed or disagreed, and where one partner was surreptitiously asked to argue an opinion opposite of what he or she really believed. We focus on interpersonal coordination as an emergent behavioral signal that captures interdependencies between conversational partners, both as the coupling of head movements over the span of milliseconds, measured via a windowed lagged cross correlation (WLCC) technique, and more global temporal dependencies across speech rate, using cross recurrence quantification analysis (CRQA). Moreover, we considered how interpersonal coordination might be shaped by strategic, adaptive conversational goals associated with deception. We found that deceptive conversations displayed more structured speech rate and higher head movement coordination, the latter with a peak in deceptive disagreement conversations. Together the results allow us to posit an adaptive account, whereby interpersonal coordination is not beholden to any single functional explanation, but can strategically adapt to diverse conversational demands.
more »
« less
Automated measurement of head movement synchrony during dyadic depression severity interviews
With few exceptions, most research in automated assessment of depression has considered only the patient’s behavior to the exclusion of the therapist’s behavior. We investigated the interpersonal coordination (synchrony) of head movement during patient-therapist clinical interviews. Participants with major depressive disorder were recorded in clinical interviews (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. For each session, patient and therapist 3D head movement was tracked from 2D videos. Head angles in the horizontal (pitch) and vertical (yaw) axes were used to measure head movement. Interpersonal coordination of head movement between patients and therapists was measured using windowed cross-correlation. Patterns of coordination in head movement were investigated using the peak picking algorithm. Changes in head movement coordination over the course of treatment were measured using a hierarchical linear model (HLM). The results indicated a strong effect for patient-therapist head movement synchrony. Within-dyad variability in head movement coordination was higher than between-dyad variability, meaning that differences over time in a dyad were higher as compared to the differences between dyads. Head movement synchrony did not change over the course of treatment. To the best of our knowledge, this study is the first attempt to analyze the mutual influence of patient-therapist head movement in relation to depression severity.
more »
« less
- Award ID(s):
- 1721667
- PAR ID:
- 10100353
- Date Published:
- Journal Name:
- Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition
- Page Range / eLocation ID:
- 1-8
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).more » « less
-
Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination.more » « less
-
more » « less
Abstract Background/Objectives Transcranial direct current stimulation (tDCS) is a non-invasive brain stimulation intervention that shows promise as a potential treatment for depression. However, the clinical efficacy of tDCS varies, possibly due to individual differences in head anatomy affecting tDCS dosage. While functional changes in brain activity are more commonly reported in major depressive disorder (MDD), some studies suggest that subtle macroscopic structural differences, such as cortical thickness or brain volume reductions, may occur in MDD and could influence tDCS electric field (E-field) distributions. Therefore, accounting for individual anatomical differences may provide a pathway to optimize functional gains in MDD by formulating personalized tDCS dosage.
Methods To address the dosing variability of tDCS, we examined a subsample of sixteen active-tDCS participants’ data from the larger ELECT clinical trial (NCT01894815). With this dataset, individualized neuroimaging-derived computational models of tDCS current were generated for (1) classifying treatment response, (2) elucidating essential stimulation features associated with treatment response, and (3) computing a personalized dose of tDCS to maximize the likelihood of treatment response in MDD.
Results In the ELECT trial, tDCS was superior to placebo (3.2 points [95% CI, 0.7 to 5.5;
P = 0.01]). Our algorithm achieved over 90% overall accuracy in classifying treatment responders from the active-tDCS group (AUC = 0.90, F1 = 0.92, MCC = 0.79). Computed precision doses also achieved an average response likelihood of 99.981% and decreased dosing variability by 91.9%.Conclusion These findings support our previously developed precision-dosing method for a new application in psychiatry by optimizing the statistical likelihood of tDCS treatment response in MDD.
-
more » « less
Background Research to date has largely conceptualized irritability in terms of intraindividual differences. However, the role of interpersonal dyadic processes has received little consideration. Nevertheless, difficulties in how parent–child dyads synchronize during interactions may be an important correlate of irritably in early childhood. Innovations in developmentally sensitive neuroimaging methods now enable the use of measures of neural synchrony to quantify synchronous responses in parent–child dyads and can help clarify the neural underpinnings of these difficulties. We introduce the Disruptive Behavior Diagnostic Observation Schedule: Biological Synchrony (DB‐DOS:BioSync) as a paradigm for exploring parent–child neural synchrony as a potential biological mechanism for interpersonal difficulties in preschool psychopathology.
Methods Using functional near‐infrared spectroscopy (fNIRS) 4‐ to 5‐year‐olds (
N = 116) and their mothers completed the DB‐DOS:BioSync while assessing neural synchrony during mild frustration and recovery. Child irritability was measured using a latent irritability factor that was calculated from four developmentally sensitive indicators.Results Both the mild frustration and the recovery contexts resulted in neural synchrony. However, less neural synchrony during the recovery context only was associated with more child irritability.
Conclusions Our results suggest that recovering after a frustrating period might be particularly challenging for children high in irritability and offer support for the use of the DB‐DOS:BioSync task to elucidate interpersonal neural mechanisms of developmental psychopathology.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
