A prerequisite for social coordination is bidirectional communication between teammates, each playing two roles simultaneously: as receptive listeners and expressive speakers. For robots working with humans in complex situations with multiple goals that differ in importance, failure to fulfill the expectation of either role could undermine group performance due to misalignment of values between humans and robots. Specifically, a robot needs to serve as an effective listener to infer human users’ intents from instructions and feedback and as an expressive speaker to explain its decision processes to users. Here, we investigate how to foster effective bidirectional human-robot communications in the context of value alignment—collaborative robots and users form an aligned understanding of the importance of possible task goals. We propose an explainable artificial intelligence (XAI)
system in which a group of robots predicts users’ values by taking in situ feedback into consideration while communicating their decision processes to users through explanations. To learn from human feedback, our XAI system integrates a cooperative communication model for inferring human values associated with multiple desirable goals. To be interpretable to humans, the system simulates human mental dynamics and predicts optimal explanations using graphical models. We conducted psychological experiments to examine the core components of the proposed computational framework. Our results show that real-time human-robot mutual understanding in complex cooperative tasks is achievable with a learning model based on bidirectional communication. We believe that this interaction framework can shed light on bidirectional value alignment in communicative XAI systems and, more broadly, in future human-machine teaming systems.
more »
« less
Simulated Language Learning from Communicative Goals and Linguistic Input
Children do not learn language from passively analyzing correlations between language and observations, but from interaction with caregivers or peers. The non-nativist approach claims that the main driver of language learning should be to achieve communicative goals. Imitation, on the other hand, is another natural desire that many argue influences language learning. However, there are still gaps in the research on what roles communicative goals and imitating linguistic input play in language acquisition, due to the difficulty of performing comprehensive experiments with human learners. In this paper, we propose a computational framework using simulated experiments that allows us to compare the roles of the two drivers. Specifically, we simulate a two-way communication game between a speaker, corresponding to a language learner, and a listener, corresponding to a caregiver or teacher. The speaker's communicative goals are modeled as rewards for successful completion of a referential game, and imitation is performed by mimicking feedback from the listener. The listener adaptively chooses to give feedback and makes choices based on the speaker's utterances. With empirical results on naturalistic visual and language data, we find that communicative goals play an important role in driving language learning, whereas imitation accelerates the learning process. We also find that (1) models trained with communicative goals tend to use minimal vocabulary and utterances and overextend them to concepts outside the original word meanings; (2) the strategy with which the listener provides feedback also influences the learning results and speed. Code and data for replicating the experiments are available (https://bit.ly/interactgym) to spur future research on models for computational studies of language learning.
more »
« less
- Award ID(s):
- 2141751
- NSF-PAR ID:
- 10356961
- Date Published:
- Journal Name:
- Proceedings of the Annual Meeting of the Cognitive Science Society
- Volume:
- 44
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Purpose: Stuttering-like disfluencies (SLDs) and typical disfluencies (TDs) are both more likely to occur as utterance length increases. However, longer and shorter utterances differ by more than the number of morphemes: They may also serve different communicative functions or describe different ideas. Decontextualized language, or language that describes events and concepts outside of the “here and now,” is associated with longer utterances. Prior work has shown that language samples taken in decontextualized contexts contain more disfluencies, but averaging across an entire language sample creates a confound between utterance length and decontextualization as contributors to stuttering. We coded individual utterances from naturalistic play samples to test the hypothesis that decontextualized language leads to increased disfluencies above and beyond the effects of utterance length. Method: We used archival transcripts of language samples from 15 preschool children who stutter (CWS) and 15 age- and sex-matched children who do not stutter (CWNS). Utterances were coded as either contextualized or decontextualized, and we used mixed-effects logistic regression to investigate the impact of utterance length and decontextualization on SLDs and TDs. Results: CWS were more likely to stutter when producing decontextualized utterances, even when controlling for utterance length. An interaction between decontextualization and utterance length indicated that the effect of decontextualization was greatest for shorter utterances. TDs increased in decontextualized utterances when controlling for utterance length for both CWS and CWNS. The effect of decontextualization on TDs did not differ statistically between the two groups. Conclusions: The increased working memory demands associated with decontextualized language contribute to increased language planning effort. This leads to increased TD in CWS and CWNS. Under a multifactorial dynamic model of stuttering, the increased language demands may also contribute to increased stuttering in CWS due to instabilities in their speech motor systems.more » « less
-
Responding to the need to teach remotely due to COVID-19, we used readily available computational approaches (and developed associated tutorials (https://mdh-cures-community.squarespace.com/virtual-cures-and-ures)) to teach virtual Course-Based Undergraduate Research Experience (CURE) laboratories that fulfil generally accepted main components of CUREs or Undergraduate Research Experiences (UREs): Scientific Background, Hypothesis Development, Proposal, Experiments, Teamwork, Data Analysis, Conclusions, and Presentation1. We then developed and taught remotely, in three phases, protein-centric CURE activities that are adaptable to virtually any protein, emphasizing contributions of noncovalent interactions to structure, binding and catalysis (an ASBMB learning framework2 foundational concept). The courses had five learning goals (unchanged in the virtual format),focused on i) use of primary literature and bioinformatics, ii) the roles of non-covalent interactions, iii) keeping accurate laboratory notebooks, iv) hypothesis development and research proposal writing, and, v) presenting the project and drawing evidence based conclusions The first phase, Developing a Research Proposal, contains three modules, and develops hallmarks of a good student-developed hypothesis using available literature (PubMed3) and preliminary observations obtained using bioinformatics, Module 1: Using Primary Literature and Data Bases (Protein Data Base4, Blast5 and Clustal Omega6), Module 2: Molecular Visualization (PyMol7 and Chimera8), culminating in a research proposal (Module 3). Provided rubrics guide student expectations. In the second phase, Preparing the Proteins, students prepared necessary proteins and mutants using Module 4: Creating and Validating Models, which leads users through creating mutants with PyMol, homology modeling with Phyre29 or Missense10, energy minimization using RefineD11 or ModRefiner12, and structure validation using MolProbity13. In the third phase, Computational Experimental Approaches to Explore the Questions developed from the Hypothesis, students selected appropriate tools to perform their experiments, chosen from computational techniques suitable for a CURE laboratory class taught remotely. Questions, paired with computational approaches were selected from Modules 5: Exploring Titratable Groups in a Protein using H++14, 6: Exploring Small Molecule Ligand Binding (with SwissDock15), 7: Exploring Protein-Protein Interaction (with HawkDock16), 8: Detecting and Exploring Potential Binding Sites on a Protein (with POCASA17 and SwissDock), and 9: Structure-Activity Relationships of Ligand Binding & Drug Design (with SwissDock, Open Eye18 or the Molecular Operating Environment (MOE)19). All involve freely available computational approaches on publicly accessible web-based servers around the world (with the exception of MOE). Original literature/Journal club activities on approaches helped students suggest tie-ins to wet lab experiments they could conduct in the future to complement their computational approaches. This approach allowed us to continue using high impact CURE teaching, without changing our course learning goals. Quantitative data (including replicates) was collected and analyzed during regular class periods. Students developed evidence-based conclusions and related them to their research questions and hypotheses. Projects culminated in a presentation where faculty feedback was facilitated with the Virtual Presentation platform from QUBES20 These computational approaches are readily adaptable for topics accessible for first to senior year classes and individual research projects (UREs). We used them in both partial and full semester CUREs in various institutional settings. We believe this format can benefit faculty and students from a wide variety of teaching institutions under conditions where remote teaching is necessary.more » « less
-
This work proposes a new computational framework for learning a structured generative model for real-world datasets. In particular, we propose to learn a Closed-loop Transcriptionbetween a multi-class, multi-dimensional data distribution and a Linear discriminative representation (CTRL) in the feature space that consists of multiple independent multi-dimensional linear subspaces. In particular, we argue that the optimal encoding and decoding mappings sought can be formulated as a two-player minimax game between the encoder and decoderfor the learned representation. A natural utility function for this game is the so-called rate reduction, a simple information-theoretic measure for distances between mixtures of subspace-like Gaussians in the feature space. Our formulation draws inspiration from closed-loop error feedback from control systems and avoids expensive evaluating and minimizing of approximated distances between arbitrary distributions in either the data space or the feature space. To a large extent, this new formulation unifies the concepts and benefits of Auto-Encoding and GAN and naturally extends them to the settings of learning a both discriminative and generative representation for multi-class and multi-dimensional real-world data. Our extensive experiments on many benchmark imagery datasets demonstrate tremendous potential of this new closed-loop formulation: under fair comparison, visual quality of the learned decoder and classification performance of the encoder is competitive and arguably better than existing methods based on GAN, VAE, or a combination of both. Unlike existing generative models, the so-learned features of the multiple classes are structured instead of hidden: different classes are explicitly mapped onto corresponding independent principal subspaces in the feature space, and diverse visual attributes within each class are modeled by the independent principal components within each subspace.more » « less
-
Contemporary approaches to perception, planning, estimation, and control have allowed robots to operate robustly as our remote surrogates in uncertain, unstructured environments. This progress now creates an opportunity for robots to operate not only in isolation, but also with and alongside humans in our complex environments. Realizing this opportunity requires an efficient and flexible medium through which humans can communicate with collaborative robots. Natural language provides one such medium, and through significant progress in statistical methods for natural-language understanding, robots are now able to interpret a diverse array of free-form navigation, manipulation, and mobile-manipulation commands. However, most contemporary approaches require a detailed, prior spatial-semantic map of the robot’s environment that models the space of possible referents of an utterance. Consequently, these methods fail when robots are deployed in new, previously unknown, or partially-observed environments, particularly when mental models of the environment differ between the human operator and the robot. This paper provides a comprehensive description of a novel learning framework that allows field and service robots to interpret and correctly execute natural-language instructions in a priori unknown, unstructured environments. Integral to our approach is its use of language as a “sensor”—inferring spatial, topological, and semantic information implicit in natural-language utterances and then exploiting this information to learn a distribution over a latent environment model. We incorporate this distribution in a probabilistic, language grounding model and infer a distribution over a symbolic representation of the robot’s action space, consistent with the utterance. We use imitation learning to identify a belief-space policy that reasons over the environment and behavior distributions. We evaluate our framework through a variety of different navigation and mobile-manipulation experiments involving an unmanned ground vehicle, a robotic wheelchair, and a mobile manipulator, demonstrating that the algorithm can follow natural-language instructions without prior knowledge of the environment.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
