An official website of the United States government
The main objective of this paper is to establish a framework to study the co-adaptation between humans and automation systems in a haptic shared control framework. We specifically used this framework to design control transfer strategies between humans and automation systems to resolve a conflict when co-steering a semi-automated ground vehicle. The proposed framework contains three main parts. First, we defined a modular structure to separate partner-specific strategies from task-dependent representations and use this structure to learn different co-adaption strategies. In this structure, we assume the human and automation steering commands can be determined by optimizing cost functions. For each agent, the costs are defined as a combination of a set of hand-coded features and vectors of weights. The hand-coded features can be selected to describe task-dependent representations. On the other hand, the weight distributions over these features can be used as a proxy to determine the partner-specific conventions. Second, to leverage the learned co-adaptation strategies, we developed a map connecting different strategies to the outputs of human-automation interactions by employing a collaborative-competitive game concept. Finally, using the map, we designed an adaptable automation system capable of co-adapting to human driver’s strategies. Specifically, we designed an episode-based policy search using the deep deterministic policy gradients technique to determine the optimal weights vector distribution of automation’s cost function. The simulation results demonstrate that the handover strategies designed based on co-adaption between human and automation systems can successfully resolve a conflict and improve the performance of the human automation teaming.
more »
« less
Cheater’s Bowl: Human vs. Computer Search Strategies for Open-Domain QA
For humans and computers, the first step in answering an open-domain question is retrieving a set of relevant documents from a large corpus. However, the strategies that computers use fundamentally differ from those of humans. To better understand these differences, we design a gamified interface for data collection—Cheater’s Bowl—where a human answers complex questions with access to both traditional and modern search tools. We collect a dataset of human search sessions, analyze human search strategies, and compare them to state-of-the-art multi-hop QA models. Humans query logically, apply dynamic search chains, and use world knowledge to boost searching. We demonstrate how human queries can improve the accuracy of existing systems and propose improving the future design of QA models.
more »
« less
- Award ID(s):
- 1822494
- PAR ID:
- 10451425
- Date Published:
- Journal Name:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Page Range / eLocation ID:
- 3627 to 3639
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Naturally-occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers to information-seeking questions. For instance, the question "When did Marie Curie discover Uranium?" cannot be answered as a typical when question without addressing the false assumption "Marie Curie discovered Uranium". In this work, we propose (QA)2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally-occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA)2, systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. We find that current models do struggle with handling questionable assumptions -- the best performing model achieves 59% human rater acceptability on abstractive QA with (QA)2 questions, leaving substantial headroom for progress.more » « less
-
Abstract Design researchers have struggled to produce quantitative predictions for exactly why and when diversity might help or hinder design search efforts. This paper addresses that problem by studying one ubiquitously used search strategy—Bayesian optimization (BO)—on a 2D test problem with modifiable convexity and difficulty. Specifically, we test how providing diverse versus non-diverse initial samples to BO affects its performance during search and introduce a fast ranked-determinantal point process method for computing diverse sets, which we need to detect sets of highly diverse or non-diverse initial samples. We initially found, to our surprise, that diversity did not appear to affect BO, neither helping nor hurting the optimizer’s convergence. However, follow-on experiments illuminated a key trade-off. Non-diverse initial samples hastened posterior convergence for the underlying model hyper-parameters—a model building advantage. In contrast, diverse initial samples accelerated exploring the function itself—a space exploration advantage. Both advantages help BO, but in different ways, and the initial sample diversity directly modulates how BO trades those advantages. Indeed, we show that fixing the BO hyper-parameters removes the model building advantage, causing diverse initial samples to always outperform models trained with non-diverse samples. These findings shed light on why, at least for BO-type optimizers, the use of diversity has mixed effects and cautions against the ubiquitous use of space-filling initializations in BO. To the extent that humans use explore-exploit search strategies similar to BO, our results provide a testable conjecture for why and when diversity may affect human-subject or design team experiments.more » « less
-
Rational decision-making is crucial in the later stages of engineering system design to allocate resources efficiently and minimize costs. However, human rationality is bounded by cognitive biases and limitations. Understanding how humans deviate from rationality is critical for guiding designers toward better design outcomes. In this paper, we quantify designer rationality in competitive scenarios based on utility theory. Using an experiment inspired by crowd-sourced contests, we show that designers employ varied search strategies. Some participants approximate a Bayesian agent that aimed to maximize its expected utility. Those with higher rationality reduce uncertainty more effectively. Furthermore, rationality correlates with both the proximity to optimal design and design iteration costs, with winning participants exhibiting greater rationality than losing participants.more » « less
-
Understanding and characterizing how people interact in information-seeking conversations will be a crucial component in developing effective conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled conversation dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intents on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.18653/v1/2022.findings-emnlp.266
