An ideal integration of autonomous agents in a human world implies that they are able to collaborate on human terms. In particular, theory of mind plays an important role in maintaining common ground during human collaboration and communication. To enable theory of mind modeling in situated interactions, we introduce a fine-grained dataset of collaborative tasks performed by pairs of human subjects in the 3D virtual blocks world of Minecraft. It provides information that captures partners’ beliefs of the world and of each other as an interaction unfolds, bringing abundant opportunities to study human collaborative behaviors in situated language communication. As a first step towards our goal of developing embodied AI agents able to infer belief states of collaborative
partners in situ, we build and present results on computational models for several theory
of mind tasks.
more »
« less
Towards Collaborative Plan Acquisition through Theory of Mind Modeling in Situated Dialogue
Collaborative tasks often begin with partial task knowledge and incomplete initial plans from each partner. To complete these tasks, agents need to engage in situated communication with their partners and coordinate their partial plans towards a complete plan to achieve a joint task goal. While such collaboration seems effortless in a human-human team, it is highly challenging for human-AI collaboration. To address this limitation, this paper takes a step towards collaborative plan acquisition, where humans and agents strive to learn and communicate with each other to acquire a complete plan for joint tasks. Specifically, we formulate a novel problem for agents to predict the missing task knowledge for themselves and for their partners based on rich perceptual and dialogue history. We extend a situated dialogue benchmark for symmetric collaborative tasks in a 3D blocks world and investigate computational strategies for plan acquisition. Our empirical results suggest that predicting the partner's missing knowledge is a more viable approach than predicting one's own. We show that explicit modeling of the partner's dialogue moves and mental states produces improved and more stable results than without. These results provide insight for future AI agents that can predict what knowledge their partner is missing and, therefore, can proactively communicate such information to help their partner acquire such missing knowledge toward a common understanding of joint tasks.
more »
« less
- Award ID(s):
- 1949634
- NSF-PAR ID:
- 10472509
- Publisher / Repository:
- IJCAI 2023
- Date Published:
- Journal Name:
- 2023 International Joint Conferences on Artificial Intelligence
- Format(s):
- Medium: X
- Location:
- Macao, SAR
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Enabling efficient communication in artificial agents brings us closer to machines that can cooperate with each other and with human partners. Hand-engineered approaches have substantial limitations, leading to increased interest in methods for communication to emerge autonomously between artificial agents. Most of the research in the field explores unsituated communication in one-step referential tasks. The tasks are not temporally interactive and lack time pressures typically present in natural communication and language learning. In these settings, agents can successfully learn what to communicate but not when or whether to communicate. Here, we extend the literature by assessing emergence of communication between reinforcement learning agents in a temporally interactive, cooperative task of navigating a gridworld environment. We show that, through multi-step interactions, agents develop just-in-time messaging protocols that enable them to successfully solve the task. With memory—which provides flexibility around message timing—agent pairs converge to a look-ahead communication protocol, finding an optimal solution to the task more quickly than without memory. Lastly, we explore situated communication, enabling the acting agent to choose when and whether to communicate. With the opportunity cost of forgoing an action to communicate, the acting agent learns to solicit information sparingly, in line with the Gricean Maxim of quantity. Our results point towards the importance of studying language emergence through situated communication in multi-step interactions.more » « less
-
In an era of ubiquitous digital interfaces and systems, technology and design practitioners must address a range of ethical dilemmas surrounding the use of persuasive design techniques and how to balance shareholder and end-user needs [2], [5]. Similarly, the increasing user concerns about unethical products and services [1] is paralleling a rise in regulatory interests in enforcing ethical design and engineering practices among technology practitioners, surfacing a need for further support. Although various scholars have developed frameworks and methods to support practitioners in navigating these challenging contexts [3], [4], often, there is a lack of resonance between these generic methods and the situated ethical complexities facing the practitioner in their everyday work. In this project, we designed and implemented a three-hour cocreation workshop with designers, engineers, and technologists to support them to develop bespoke ethics-focused action plans that are resonant with the ethical challenges they face in their everyday practice. In developing the co-creation session, we sought to answer the following questions to empower practitioners: • How can we support practitioners in developing action plans to address ethical dilemmas in their everyday work? and • How can we empower designers to design more responsibly? Building on these questions as a guide, we employed Miro, a digital whiteboard platform, to develop the co-creation experience. The final c o-creation e xperience w as d esigned w ith the visual metaphor of a “house” with four floors and multiple rooms that allowed participants to complete different tasks per room, all aimed towards the overall goal of developing participants' own personalized action plan in an interactive and collaborative way. We invited participants to share their stories and ethical dilemmas to support their creation and iteration of a personal action plan that they could later use in their everyday work context. Across the six co-creation sessions we conducted, participants (n=26) gained a better understanding of the drivers for ethical action in the context of their everyday work and developed an action plan through the co-creation workshop that enabled them to constructively engage with ethical challenges in their professional context. At the end of the session, participants were provided the action plans they created to allow them to use it in their practice. Furthermore, the co-design workshops were designed such that practitioners could take them away (the house and session guide) and run them independently at their organization or another context to support their objectives. We describe the building and the activities conducted in each floor below and will provide a pictorial representation of the house with the different floors, rooms, and activities on the poster presentation. a) First floor-Welcome, Introduction, Reflection: The first floor of the virtual house was designed to allow participants to introduce themselves and to reflect on and discuss the ethical concerns they wished to resolve during the session. b) Second floor-Shopping for ethics-focused methods: The second floor of the virtual house was designed as a “shopping” space where participants selected from range of ethicsfocused building blocks that they wish to potentially adapt or incorporate into their own action plan. They were also allowed to introduce their own methods or tools. c) Third floor-DIY Workspace: The third floor was designed as a DIY workspace to allow the participants to work in small groups to develop their own bespoke action plan based on building blocks they have gathered from their shopping trip and by using any other components they wish. The goal here was to support participants in developing methods and action plans that were resonant with their situated ethical complexities. d) Fourth floor-Gallery Space: The fourth floor was designed as a gallery to allow participants to share and discuss their action plans with other participants and to identify how their action plans could impact their future practice or educational experiences. Participants were also provided an opportunity at this stage to reflect on their experience participating in the session and provide feedback on opportunities for future improvement.more » « less
-
Changing Electrical and Computer Engineering Department Culture from the Bottom Up: Action Plans Generated from Faculty Interviews We prefer a Lessons Learned Paper. In a collaborative effort between a RED: Revolutionizing Engineering and Computer Science Departments (RED) National Science Foundation grant awarded to an electrical and computer engineering department (ECpE) and a broader, university-wide ADVANCE program, ECpE faculty were invited to participate in focus groups to evaluate the culture of their department, to further department goals, and to facilitate long-term planning. Forty-four ECpE faculty members from a large Midwestern university participated in these interviews, which were specifically focused on departmental support and challenges, distribution of resources, faculty workload, career/family balance, mentoring, faculty professional development, productivity, recruitment, and diversity. Faculty were interviewed in groups according to rank, and issues important to particular subcategories of faculty (e.g., rank, gender, etc.) were noted. Data were analyzed by a social scientist using the full transcript of each interview/focus group and the NVivo 12 Qualitative Research Software Program. She presented the written report to the entire faculty. Based on the results of the focus groups, the ECpE department developed an action plan with six main thrusts for improving departmental culture and encouraging departmental change and transformation. 1. Department Interactions – Encourage open dialogue and consider department retreats. Academic areas should be held accountable for the working environment and encouraged to discuss department-related issues. 2. Mentoring, Promotion, and Evaluation – Continue mentoring junior faculty. Improve the clarity of P&T operational documents and seek faculty input on the evaluation system. 3. Teaching Loads – Investigate teaching assistant (TA) allocation models and explore models for teaching loads. Develop a TA performance evaluation system and return TA support to levels seen in the 2010 timeframe. Improvements to teaching evaluations should consider differential workloads, clarifying expectations for senior advising, and hiring more faculty for undergraduate-heavy areas. 4. Diversity, Equity, and Inclusion – Enact an explicit focus on diversity in hiring. Review departmental policies on inclusive teaching and learning environments. 5. Building – Communicate with upper administration about the need for a new building. Explore possibilities for collaborations with Computer Science on a joint building. 6. Support Staff – Increase communication with the department regarding new service delivery models. Request additional support for Human Resources, communications, and finance. Recognize staff excellence at the annual department banquet and through college/university awards.more » « less
-
We present GhostAR, a time-space editor for authoring and acting Human-Robot-Collaborative (HRC) tasks in-situ. Our system adopts an embodied authoring approach in Augmented Reality (AR), for spatially editing the actions and programming the robots through demonstrative role-playing. We propose a novel HRC workflow that externalizes user’s authoring as demonstrative and editable AR ghost, allowing for spatially situated visual referencing, realistic animated simulation, and collaborative action guidance. We develop a dynamic time warping (DTW) based collaboration model which takes the real-time captured motion as inputs, maps it to the previously authored human actions, and outputs the corresponding robot actions to achieve adaptive collaboration. We emphasize an in-situ authoring and rapid iterations of joint plans without an offline training process. Further, we demonstrate and evaluate the effectiveness of our workflow through HRC use cases and a three-session user study.more » « less
