Task-oriented dialogue research has mainly focused
on a few popular languages like English
and Chinese, due to the high dataset creation
cost for a new language. To reduce the cost,
we apply manual editing to automatically translated
data. We create a new multilingual benchmark,
X-RiSAWOZ, by translating the Chinese
RiSAWOZ to 4 languages: English, French,
Hindi, Korean; and a code-mixed English-
Hindi language. X-RiSAWOZ has more than
18,000 human-verified dialogue utterances for
each language, and unlike most multilingual
prior work, is an end-to-end dataset for building
fully-functioning agents.
The many difficulties we encountered in creating
X-RiSAWOZ led us to develop a toolset
to accelerate the post-editing of a new language
dataset after translation. This toolset improves
machine translation with a hybrid entity
alignment technique that combines neural with
dictionary-based methods, along with many automated
and semi-automated validation checks.
We establish strong baselines for X-RiSAWOZ
by training dialogue agents in the zero- and
few-shot settings where limited gold data is
available in the target language. Our results
suggest that our translation and post-editing
methodology and toolset can be used to create
new high-quality multilingual dialogue agents
cost-effectively. Our dataset,
more »
« less
This content will become publicly available on May 1, 2024
Zero and Few-Shot Localization of Task-Oriented Dialogue Agents with a Distilled Representation
Task-oriented Dialogue (ToD) agents are
mostly limited to a few widely-spoken languages,
mainly due to the high cost of acquiring
training data for each language. Existing
low-cost approaches that rely on cross-lingual
embeddings or naive machine translation sacrifice
a lot of accuracy for data efficiency, and
largely fail in creating a usable dialogue agent.
We propose automatic methods that use ToD
training data in a source language to build a
high-quality functioning dialogue agent in another
target language that has no training data
(i.e. zero-shot) or a small training set (i.e. fewshot).
Unlike most prior work in cross-lingual
ToD that only focuses on Dialogue State Tracking
(DST), we build an end-to-end agent.
We show that our approach closes the accuracy
gap between few-shot and existing fullshot
methods for ToD agents. We achieve
this by (1) improving the dialogue data representation,
(2) improving entity-aware machine
translation, and (3) automatic filtering of noisy
translations.
We evaluate our approach on the recent bilingual
dialogue dataset BiToD. In Chinese to
English transfer, in the zero-shot setting, our
method achieves 46.7% and 22.0% in Task
Success Rate (TSR) and Dialogue Success
Rate (DSR) respectively. In the few-shot setting
where 10% of the data in the target language
is used, we improve the state-of-the-art
by 15.2% and 14.0%, coming within 5% of
full-shot training.
more »
« less
- Award ID(s):
- 1900638
- NSF-PAR ID:
- 10427011
- Date Published:
- Journal Name:
- Proceedings of the conference Association for Computational Linguistics European Chapter Conference
- ISSN:
- 1525-2450
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Robust state tracking for task-oriented dialogue systems currently remains restricted to a few popular languages. This paper shows that given a large-scale dialogue data set in one language, we can automatically produce an effective semantic parser for other languages using machine translation. We propose automatic translation of dialogue datasets with alignment to ensure faithful translation of slot values and eliminate costly human supervision used in previous benchmarks. We also propose a new contextual semantic parsing model, which encodes the formal slots and values, and only the last agent and user utterances. We show that the succinct representation reduces the compounding effect of translation errors, without harming the accuracy in practice. We evaluate our approach on several dialogue state tracking benchmarks. On RiSAWOZ, CrossWOZ, CrossWOZ-EN, and MultiWOZ-ZH datasets we improve the state of the art by 11%, 17%, 20%, and 0.3% in joint goal accuracy. We present a comprehensive error analysis for all three datasets showing erroneous annotations can lead to misguided judgments on the quality of the model. Finally, we present RiSAWOZ English and German datasets, created using our translation methodology. On these datasets, accuracy is within 11% of the original showing that high-accuracy multilingual dialogue datasets are possible without relying on expensive human annotations. We release our datasets and software open source.more » « less
-
Previous attempts to build effective semantic parsers for Wizard-of-Oz (WOZ) conversations suffer from the difficulty in acquiring a high-quality, manually annotated training set. Approaches based only on dialogue synthesis are insufficient, as dialogues generated from state-machine based models are poor approximations of real-life conversations. Furthermore, previously proposed dialogue state representations are ambiguous and lack the precision necessary for building an effective agent.This paper proposes a new dialogue representation and a sample-efficient methodology that can predict precise dialogue states in WOZ conversations. We extended the ThingTalk representation to capture all information an agent needs to respond properly. Our training strategy is sample-efficient: we combine (1) few-shot data sparsely sampling the full dialogue space and (2) synthesized data covering a subset space of dialogues generated by a succinct state-based dialogue model. The completeness of the extended ThingTalk language is demonstrated with a fully operational agent, which is also used in training data synthesis. We demonstrate the effectiveness of our methodology on MultiWOZ 3.0, a reannotation of the MultiWOZ 2.1 dataset in ThingTalk. ThingTalk can represent 98% of the test turns, while the simulator can emulate 85% of the validation set. We train a contextual semantic parser using our strategy, and obtain 79% turn-by-turn exact match accuracy on the reannotated test set.more » « less
-
Sentiment classification typically relies on a large amount of labeled data. In practice, the availability of labels is highly imbalanced among different languages, e.g., more English texts are labeled than texts in any other languages, which creates a considerable inequality in the quality of related information services received by users speaking different languages. To tackle this problem, cross-lingual sentiment classification approaches aim to transfer knowledge learned from one language that has abundant labeled examples (i.e., the source language, usually English) to another language with fewer labels (i.e., the target language). The source and the target languages are usually bridged through off-the-shelf machine translation tools. Through such a channel, cross-language sentiment patterns can be successfully learned from English and transferred into the target languages. This approach, however, often fails to capture sentiment knowledge specific to the target language, and thus compromises the accuracy of the downstream classification task. In this paper, we employ emojis, which are widely available in many languages, as a new channel to learn both the cross-language and the language-specific sentiment patterns. We propose a novel representation learning method that uses emoji prediction as an instrument to learn respective sentiment-aware representations for each language. The learned representations are then integrated to facilitate cross-lingual sentiment classification. The proposed method demonstrates state-of-the-art performance on benchmark datasets, which is sustained even when sentiment labels are scarce.more » « less
-
null (Ed.)Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.more » « less
