skip to main content


Title: Multimodal and Multitask Approach to Listener’s Backchannel Prediction: Can Prediction of Turn-changing and Turn-management Willingness Improve Backchannel Modeling
Award ID(s):
1750439
NSF-PAR ID:
10317283
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents (IVA)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
  2. Participants in a conversation must carefully monitor the turn-management (speaking and listening) willingness of other conversational partners and adjust their turn-changing behaviors accordingly to have smooth conversation. Many studies have focused on developing actual turn-changing (i.e., next speaker or end-of-turn) models that can predict whether turn-keeping or turn-changing will occur. Participants' verbal and non-verbal behaviors have been used as input features for predictive models. To the best of our knowledge, these studies only model the relationship between participant behavior and turn-changing. Thus, there is no model that takes into account participants' willingness to acquire a turn (turn-management willingness). In this paper, we address the challenge of building such models to predict the willingness of both speakers and listeners. Firstly, we find that dissonance exists between willingness and actual turn-changing. Secondly, we propose predictive models that are based on trimodal inputs, including acoustic, linguistic, and visual cues distilled from conversations. Additionally, we study the impact of modeling willingness to help improve the task of turn-changing prediction. To do so, we introduce a dyadic conversation corpus with annotated scores of speaker/listener turn-management willingness. Our results show that using all three modalities (i.e., acoustic, linguistic, and visual cues) of the speaker and listener is critically important for predicting turn-management willingness. Furthermore, explicitly adding willingness as a prediction task improves the performance of turn-changing prediction. Moreover, turn-management willingness prediction becomes more accurate when this joint prediction of turn-management willingness and turn-changing is performed by using multi-task learning techniques. 
    more » « less
  3. Correction for ‘Single-conformation spectroscopy of cold, protonated D PG-containing peptides: switching β-turn types and formation of a sequential type II/II′ double β-turn’ by John T. Lawler et al. , Phys. Chem. Chem. Phys. , 2022, 24 , 2095–2109, https://doi.org/10.1039/D1CP04852J. 
    more » « less
  4. d -Proline ( D Pro, D P) is widely utilized to form β-hairpin loops in engineered peptides that would otherwise be unstructured, most often as part of a D PG sub-unit that forms a β-turn. To observe whether D PG facilitated this effect in short protonated peptides, conformation specific IR–UV double resonance photofragment spectra of the cold (∼10 K) protonated D P and L P diastereomers of the pentapeptide YAPGA was carried out in the hydride stretch (2800–3700 cm −1 ) and amide I/II (1400–1800 cm −1 ) regions. A model localized Hamiltonian was developed to better describe the 1600–1800 cm −1 region commonly associated with the amide I vibrations. The CO stretch fundamentals experience extensive mixing with the N–H bending fundamentals of the NH 3 + group in these protonated peptides. The model Hamiltonian accounts for experiment in quantitative detail. In the D P diastereomer, all the population is funneled into a single conformer which presented as a type II β-turn with A and D P in the i + 1 and i + 2 positions, respectively. This structure was not the anticipated type II′ β-turn across D PG that we had hypothesized based on solution-phase propensities. Analysis of the conformational energy landscape shows that both steric and charge-induced effects play a role in the preferred formation of the type II β-turn. In contrast, the L P isomer forms three conformations with very different structures, none of which were type II/II′ β-turns, confirming that L PG is not a β-turn former. Finally, single-conformation spectroscopy was also carried out on the extended peptide [YAA D PGAAA + H] + to determine whether moving the protonated N-terminus further from D PG would lead to β-hairpin formation. Despite funneling its entire population into a single peptide backbone structure, the assigned structure is not a β-hairpin, but a concatenated type II/type II′ double β-turn that displaces the peptide backbone laterally by about 7.5 Å, but leaves the backbone oriented in its original direction. 
    more » « less