Children learn a striking amount of language in their first few years of life -thousands of sounds, words, grammatical categories, and how to combine them into meaningful utterances. Unlike most recent machine learning models, which learn language from static existing text or images (?, ?), very young children do not learn language purely from observing visuallinguistic co-occurrences, e.g. watching television (?, ?, ?, ?, ?), but rather from interacting with their parents in conversations regarding family members, body parts, animals, foods and clothing, directed by the interest of the child (?, ?, ?). The challenge is then to understand how this learning process works and what internal and external factors influence it. Figure 1: A child and adult playing shared-goal bidirectional communication games. The child learns from both communicative goals and the parent's feedback as linguistic input. On the left, the child uses incorrect word order to describe the shape in the middle, but the adult understands and gives corrective input. In contrast, on the right, the child uses "orange square" to describe the shape on the right, but the adult misinterprets and provides feedback for the shape on the left.
The most common and straightforward view is that children primarily use language as a tool to communicate (?, ?, ?). Just like learning to use other tools, one becomes more proficient via trial-and-error. Getting what they ask for, like asking for "applesauce" and receiving it instead of another object, reinforces the connection between entities and names. Conversely, failing to achieve a goal leads to a weaker connection or even negative reinforcement. Parents and adult members of the community also share intentions with children and respond to children's requests (?, ?), and thus children learn language from the use of language (?, ?). From this view, language is learned to convey meaning and reinforced by communicative goals (CG), providing pressure to learn at least semantics, and perhaps also syntax to allow for disambiguation of more difficult concepts (?, ?)
Another way children learn language from their parents is through imitation, which has been studied for centuries since ? (1787). Although parents do not always explicitly point out grammatical mistakes in children's language, they offer corrective linguistic input (LI) to children based on their understanding of the meaning of the children's utterances (?, ?, ?). As a part of social learning, children imitate the feedback from their parents and learn the correlation between feedback and the meaning they want to express. In this way, the fluency of a children's speech improves, but since the parents may not interpret the request correctly, the meaning of the feedback may align with the children's intent.
In this paper, we simulate this learning process in the context of language games, which have been the proving ground for various linguistics theories since their conception by ? (?).
Drawing an analogy to the child-parent conversation scenario, we model the child as a speaker agent, which generates utterances based on the target objects provided by the environment, and the parent as a listener agent, which responses to the child's utterances by choosing the objects and/or give corrective feedbacks. Based on this setup, we study the following research question Can we form a computational speaker model that learns to speak from a skilled language listener under the communication game setting? What role do communicative goals and linguistic input play in this process?
Our hypothesis is that communicative goals are the main driver of language learning in this formulation (Communication Games section), while linguistic input accelerates the learning process through syntax level supervision. To evaluate this, we use neural networks combined with heuristic rules to model the speakers and listeners (Model section). The learning process is implemented with a balance of reinforcement learning for CG and maximum likelihood estimation for LI (Learning Process section). We perform empirical experiments on communication games with MSCOCO (?, ?) images and captions (Experiment section). We find that CG contributes most to the game accuracy and also helps learning syntax as reflected by a fluency metric. We also find that different listener strategies also contribute to the success of language learning. Interesting, we also find that overextension in the resulting language of CG-driven models is very common, which is also common in early children speech (?, ?), while the same phenomenon does not often appear in models only trained with LI. Our results may provide evidence for usage-based language acquisition theory, the belief that language is acquired in the service of communicative functions (?, ?).
Following previous work on communicative agents learning to form communication pacts in referential games, we use asymmetric speaker-listener games (?, ?) with additional feedback channels.
A general goal-oriented language game provides an environment where the participants use language to communicate with each other to achieve the given goal. We consider the most basic setting of a collaborative referential game.
As illustrated in from the pool of images I, only visible to the speaker. N distractor images are randomly sampled from a distribution D N x . The target image and distractors are randomly shuffled before being shown to the listener to prevent any bias in the order. We denote the shuffled sequence of images as C ˜, and order of the goal as i g , i.e. C ˜ig = x and {C ˜i} i̸ =i g is a permutation of C.
The speaker (modeling the child) takes the first turn in each game by describing the image in English. The listener (modeling the parent) then takes one of two actions based on the utterance u: (1) choose an image i ˆor (2) perform no action i ˆ= noop (e.g. when they do not understand the utterance with enough confidence). Additionally, at the end of each game, the listener can choose to provide linguistic supervision to the speaker. At the end of each game, the speaker receives a reward based on the listener's action.
To model the communicative goals, we give positive rewards when the game is successful and negative rewards if the listener chooses the wrong image. In addition, we encourage the speaker to give unambiguous utterances by penalizing the noop action with a small negative reward -1 < w noop < 0.foot_0 :
As mentioned before, the participants consist of a speaker and a listener sending and receiving natural language messages. The speaker is a message-producing model defined by the vocabulary Σ; the space of observations I ; and a model f : I → Σ * . The listener is an instruction-follower defined by the same vocabulary Σ, observation space I N+1 , and space of actions [N + 1] as the speaker; and a model g :
Note that the listeners cannot directly observe the goal, so the speakers need to use instructions to inform the listeners about the goal of each game.
The speaker is an image captioning model (?, ?, ?), which first encodes the goal image x with a pretrained ResNet (?, ?), and generates the utterance u = u i M i=1 with an LSTM neural network (?, ?) in an auto-regressive fashion:
where w u i ∈ R d w is the word embedding of word u i .
A listener consists of two parts, a neural network-based ranker and a rule-based controller deciding whether to act and give language feedback to the speaker. Given the utterance u, the listener ranks the images C ˜by the dot product between the LSTM embeddings of u and image embeddings encoded by the same pretrained ResNet as the speaker, i.e. for each image C ˜i, the score for ranking is
where w u is a shorthand for the word embedding of all words in the sentence. Note that we use the same visual network for speakers and listeners, ignoring the differences between the visual perception of individuals, but the parameters language networks are not the same. We will discuss the method to acquire these parameters later in this section.
In human conversations, parents use a variety of techniques when giving feedback, including asking clarification questions and providing exemplar utterances. However, incorporating this open-ended feedback presents a huge challenge to the computational modeling of speakers. In this paper, we limit the feedback to full correct utterances for the goal image, which may be redundant or ineffective in many real world cases, but is general enough that most other kinds of feedback can be converted to it. We consider a listener controlled by both neural network rankers (as described above) and heuristic rules (Alg. ??) which makes a choice when its confidence is high enough and gives feedback to the speaker if it thinks the utterance is not articulated well. Following (?, ?), we use the probability of prediction as the indicator of confidence. Alg. ?? has two parameters θ 1 and θ 2 which control making choices and giving feedback respectively. We will show the dramatic effects brought by these two parameters on language learning in the experiments. The golden utterances for images U * are drawn from the captions provided in MSCOCO dataset (?, ?).
Listener Pretraining To model learning for a proficient language user, we need a good enough listener. Apart from θ 1 and θ 2 as well as the parameters in the ResNet, the parameters
If the confidence is too low, the listener will not make a choice or give feedback.
in the language network need pretraining. We use mini-batch stochastic gradient descent to optimize the following
Learning Process
The communicative goals and mimicking linguistic input can be modeled as two distinct learning objectives for the speaker network. Similar to children getting rewards from the environment if their request is fulfilled, and penalties otherwise, we use the expected game rewards as the objective for CG:
Note that the action space -the space of utterances -is discrete and non-differentiable, so we employ reinforcement learning to optimize the speaker policy π. Later in this section we give a brief introduction to PPO (?, ?), the RL method we used in the experiment. Children's language models are reinforced if they recover the parent's corrective linguistic input. We thus model this objective as an maximum likelihood objective which measures parents' language in children's models.
This part is optimized with stochastic gradient descent.
To study the joint effect of both objectives, we adopt a multitask learning objective:
where λ is the coefficient balancing the two objectives and correlates with the importance of the CG objective.
Reinforcement learning methods are often employed to optimize non-differentiable objectives. In this subsection, we use the short hands state s = {u i } t-1 i=1 , action a = u t at time step driven by communicative goals converge to a higher average reward than models with only linguistic input.
t for generating the utterance. The simplest RL method is policy gradient
As an on-policy method, PG optimizes its policy on the rollout data only once, which is inefficient. We can use importance sampling to reuse the data:
where r π old s,a (π) = π(a|s)/π old (a|s) is the likelihood ratio between the new policy π and the old policy π old used to sample data, A
is the advantage value function of the old policy π old . However, using this method does not guarantee a policy improvement. Therefore, TRPO (?, ?) and PPO (?, ?) are introduced with the basic of restricting the policy in a close distance from the old policy. PPO restricts the policy by a clipping function (?, ?)foot_1
where F CLIP is defined as
(1ε, 1 + ε) is called the clipping range, and 0 < ε < 1 is the parameter. Note that in theory most RL methods can be employed to optimize the CG objective. However, we use PPO here based on the trade-off of simplicity and relative good performance. We discuss other RL methods in the related works section.
We use conventional split of MS COCO (?, ?). All of our neural networks are trained or pretrained on the training set, and all the results below are calculated on the test set.
In each game, after sampling the goal image x, the distractors are sampled from either uniform distribution C ∼ U(I ) N (easy and default setting) or from a distribution skewed to the goal C ∼ D N x , where D x (y) ∝ e ∥x x x-y y y∥ 2 and x x x and y y y are embeddings from pretrained ResNet (?, ?) (hard setting).
We use two metrics in the following experiments: (1) accuracy: the frequency of the listener choosing the goal among images;
(2) fluency score, which reflects grammar quality of the sentence without considering semantics relatedness, following (?, ?)
We use GPT-2 large (?, ?) as p M and a unigram model as p U , both are fine-tuned/trained on MSCOCO.
The first question we want to investigate is which signal is more important in learning semantically correct descriptions for the target image. In this paper, we use the listener's accuracy as a proxy to examine the semantic quality of generated descriptions. As shown in Fig. ??, the accuracy of the LIonly model tops out at 60%, while models with the CG objective have significantly higher accuracy. However, the CGonly model needs about 400k steps to warm-up before dramatically improving on the similar performance of the combined model. With the help of LI, the CG+LI model (where λ = 0.01 is the best hyperparameter, used in all CG+LI models) not only has a faster improvement at the start of training, but also achieves higher accuracy then CG-only model. From this result, we can see that CG is the main driver for conveying accurate information. The communication goal signal steers the model to output pragmatical descriptions that help the listener choose the correct target. In the hard setting, the CG+LI model and CG-only model both achieve 74% accuracy while the LI-only model only reaches 59%, which is a similar trend as the easy setting, thus confirming our conclusion still holds even if more detailed descriptions are needed for the game.
The second question to investigate is which signal helps the speaker to learn to produce fluent language. Fig. ?? shows that LI is the main driver for learning to speak more fluently. The likely reason for the decreasing fluency of the CG-only model is the vocabulary shrinks and concentrates on a few words instead of all frequent ones in MSCOCO. In contrast, learning from linguistic inputs helps the model to fit the natural distribution of words. Later in this section, we will talk about the overextension of CG driven models. The improvement brought by LI may be the reason why CG+LI model does not need a warmup in Stage I in Fig. ??.
In all previous experiments, we present results with θ 1 = 0.4 and θ 2 = 0.9 as the the thresholds for the listener strategy. In Fig. ?? we show the influence of these two parameters on the model's final accuracy. We find that the performance is very sensitive to the listener strategy. A small 0.05 change results in the difference between the best result and failure.
Besides the experiments on CG and LI's influence on language learning, a signifcant difference between CG-driven models and LI-only model is overextension. To explore this, we randomly choose several nouns in the empirical vocabulary (words that exist in utterances) of CG+LI model. Most words exhibit intuitive cases of overextension. Some are based on color similarity, e.g., court; some are based on shape similarity, e.g. horse, giraffe, kite; while others are based on texture similarity, e.g. couch, pizza. We hypothesize there are a few possible reason for overextension in the model: (1) the shared visual perception -similar images to the speaker also look similar to the listener; (2) lack of linguistic inputs -with limited vocabulary, the RL model tests the acceptability of similar concepts; (3) the generality of the listener -the listener can understand the utterances just as we can interpret these errors. Although the models are making these errors, it may not necessarily be a bad thing. This phenomenon accounts for 40% of words used by 1;6 to 2;6 children (?, ?). This shows our formulation may be a good model of child language acquisition.
Without natural language annotations, this pressure for the speaker and listener enables language emergence. (?, ?) first uses the same recurrent neural networks as the speaker and the listener to conduct emergent communication in referential game. Following their lead, (?, ?) study how emergent languages are grounded to the input images, and (?, ?) studies multi-turn communication via negotiation. (?, ?, ?) study the compositionally and systematicity of emergent languages. (?, ?) also explore the setting of training speaker with both reinforcement learning and MLE in referential games. To build a model that can communicate with humans, they start with a pretrained language model and use ground truth data of games in the experiment. Whereas we start from randomly initialized speakers and do not allow listeners' access to the goal to study language learning from scratch in the communication games.
Different reinforcement learning methods have been applied to language generation. On-policy methods include REIN-FORCE (?, ?, ?), actor-critic(?, ?, ?), policy gradient (?, ?, ?, ?), and off-policy methods include importance weighted policy gradient (?, ?, ?, ?), Q-learning (?, ?), and soft Q-learning (?, ?). In this paper, we use the most commonly used onpolicy method PPO to optimize the CG objective. Experimenting with other methods is an interesting future direction.
In this paper, we propose a computational framework for language learning through communication games with both communicative goals and linguistic input objectives. We investigate the roles of CG and LI in general language learning in terms of conveying meaning and syntax learning. We also find that listener's strategy is important for language learning. This sheds light on child language learning -language usage may be the main driver, but without linguistic inputs language may be slow to acquire. Additionally, the adults' strategy in responding to the children's request is also important. Future work could further confirm this intuition by teaching human subjects (new) language with the best listener setting.
We use a tighter lower bound of w noop , so that a random choice is worse than no action: w > 1 N -1.
There are two variants of PPO: we refer to the one with clipping function as PPO, and refer to the one with adaptive KL penalty coefficient as PPO-penalty (?, ?).