An official website of the United States government
This study investigates the presence of dynamical patterns of interpersonal coordination in extended deceptive conversations across multimodal channels of behavior. Using a novel "devil’s advocate" paradigm, we experimentally elicited deception and truth across topics in which conversational partners either agreed or disagreed, and where one partner was surreptitiously asked to argue an opinion opposite of what he or she really believed. We focus on interpersonal coordination as an emergent behavioral signal that captures interdependencies between conversational partners, both as the coupling of head movements over the span of milliseconds, measured via a windowed lagged cross correlation (WLCC) technique, and more global temporal dependencies across speech rate, using cross recurrence quantification analysis (CRQA). Moreover, we considered how interpersonal coordination might be shaped by strategic, adaptive conversational goals associated with deception. We found that deceptive conversations displayed more structured speech rate and higher head movement coordination, the latter with a peak in deceptive disagreement conversations. Together the results allow us to posit an adaptive account, whereby interpersonal coordination is not beholden to any single functional explanation, but can strategically adapt to diverse conversational demands.
more »
« less
Automated measurement of head movement synchrony during dyadic depression severity interviews
With few exceptions, most research in automated assessment of depression has considered only the patient’s behavior to the exclusion of the therapist’s behavior. We investigated the interpersonal coordination (synchrony) of head movement during patient-therapist clinical interviews. Participants with major depressive disorder were recorded in clinical interviews (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. For each session, patient and therapist 3D head movement was tracked from 2D videos. Head angles in the horizontal (pitch) and vertical (yaw) axes were used to measure head movement. Interpersonal coordination of head movement between patients and therapists was measured using windowed cross-correlation. Patterns of coordination in head movement were investigated using the peak picking algorithm. Changes in head movement coordination over the course of treatment were measured using a hierarchical linear model (HLM). The results indicated a strong effect for patient-therapist head movement synchrony. Within-dyad variability in head movement coordination was higher than between-dyad variability, meaning that differences over time in a dyad were higher as compared to the differences between dyads. Head movement synchrony did not change over the course of treatment. To the best of our knowledge, this study is the first attempt to analyze the mutual influence of patient-therapist head movement in relation to depression severity.
more »
« less
- Award ID(s):
- 1721667
- PAR ID:
- 10100353
- Date Published:
- Journal Name:
- Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition
- Page Range / eLocation ID:
- 1-8
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Observing how infants and mothers coordinate their behaviors can highlight meaningful patterns in early communication and infant development. While dyads often differ in the modalities they use to communicate, especially in the first year of life, it remains unclear how to capture coordination across multiple types of behaviors using existing computational models of interpersonal synchrony. This paper explores Dynamic Mode Decomposition with control (DMDc) as a method of integrating multiple signals from each communicating partner into a model of multimodal behavioral coordination. We used an existing video dataset to track the head pose, arm pose, and vocal fundamental frequency of infants and mothers during the Face-to-Face Still-Face (FFSF) procedure, a validated 3-stage interaction paradigm. For each recorded interaction, we fit both unimodal and multimodal DMDc models to the extracted pose data. The resulting dynamic characteristics of the models were analyzed to evaluate trends in individual behaviors and dyadic processes across infant age and stages of the interactions. Results demonstrate that observed trends in interaction dynamics across stages of the FFSF protocol were stronger and more significant when models incorporated both head and arm pose data, rather than a single behavior modality. Model output showed significant trends across age, identifying changes in infant movement and in the relationship between infant and mother behaviors. Models that included mothers’ audio data demonstrated similar results to those evaluated with pose data, confirming that DMDc can leverage different sets of behavioral signals from each interacting partner. Taken together, our results demonstrate the potential of DMDc toward integrating multiple behavioral signals into the measurement of multimodal interpersonal coordination.more » « less
-
Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).more » « less
-
Abstract The extent to which hand dominance may influence how each agent contributes to inter-personal coordination remains unknown. In the present study, right-handed human participants performed object balancing tasks either in dyadic conditions with each agent using one hand (left or right), or in bimanual conditions where each agent performed the task individually with both hands. We found that object load was shared between two hands more asymmetrically in dyadic than single-agent conditions. However, hand dominance did not influence how two hands shared the object load. In contrast, hand dominance was a major factor in modulating hand vertical movement speed. Furthermore, the magnitude of internal force produced by two hands against each other correlated with the synchrony between the two hands’ movement in dyads. This finding supports the important role of internal force in haptic communication. Importantly, both internal force and movement synchrony were affected by hand dominance of the paired participants. Overall, these results demonstrate, for the first time, that pairing of one dominant and one non-dominant hand may promote asymmetrical roles within a dyad during joint physical interactions. This appears to enable the agent using the dominant hand to actively maintain effective haptic communication and task performance.more » « less
-
Background & Purpose: Deformational plagiocephaly and brachycephaly (DPB) is a cranial condition manifested in 20% of infants in the US. DPB affects children and their families through psychological pressure, social stigma, and significant medical costs. If detected between 0-3 months of age, there is strong potential for correction via aggressive repositioning and/or physical therapy if congenital muscular torticollis is present. At later stages, DPB is most effectively treated by more expensive treatments like helmet therapy. Two cranial parameters that can help with the early detection and tracking of DPB are the cranial index (CI) and cranial vault asymmetry index (CVAI). Currently, these measurements are performed with a hand caliper by a specialist, i.e., nurse practitioner (CRNP) or physician assistant who specializes in cleft-craniofacial diagnosis, physical therapist, pediatric plastic/neurosurgeons, or orthotist. To make the measurements frequent, accessible, and accurate at the point of care, i.e., in pediatric offices, we developed and evaluated a mobile app called SoftspotTM to measure CI and CVAI, thus facilitating the early detection and monitoring of DPB. Method/Description: Sequences of bird’s eye-view head photos extracted from video were collected for 77 patients (aged 2 – 11 months, 51 females, 26 males) with an iPhone X (Apple Inc., Cupertino, CA). The head length, width, and diagonals were measured by a single CRNP via hand calipers at a large multidisciplinary cranio-facial center with IRB approval and patient consent. For each patient, five images were chosen by an analyst and segmented into head components, namely the head and nose, using quantitative imaging methods. For each image CI and CVAI were automatically measured, and these measurements were averaged for each patient. Automated CI and CVAI measurements were compared to values obtained by the caliper measurements in terms of mean absolute error (MAE), and outliers were excluded beyond 3 standard deviations away from the average MAE. Results were further analyzed by the Bland-Altman method and Spearman Correlation Coefficient. Results: MAE was 2.18 ± 1.60 for CI and 1.57 ± 1.03 for CVAI measurements. Spearman Correlation Coefficients between measurements and ground truth were 0.93 for CI (p<0.001) and 0.91 for CVAI (p<0.001). Bland-Altman analysis revealed limits of agreement for CI and CVAI as [-4.59, 5.76] (mean = 0.59) and [-3.91, 3.40] (mean = -0.25) respectively. Conclusions: Digital smartphone-based methods for DPB assessment are feasible, and this study demonstrated significant correlation between automated digital measurements and ground truth clinical values. Smartphone-based measurements of DPB can be performed at the point of care to improve the early detection and treatment of DPB.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
