<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Training Language Models with Language Feedback</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>05/20/2022</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10351046</idno>
					<idno type="doi"></idno>
					<title level='j'>ACL Workshop on Learning with Natural Language Supervision. 2022.</title>
<idno></idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Jun Shern Jon Ander Campos</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Pretrained language models often do not performtasks in ways that are in line with ourpreferences, e.g., generating offensive text orfactually incorrect summaries. Recent workapproaches the above issue by learning froma simple form of human evaluation: comparisonsbetween pairs of model-generated taskoutputs. Comparison feedback conveys limitedinformation about human preferences perhuman evaluation. Here, we propose to learnfrom natural language feedback, which conveysmore information per human evaluation.We learn from language feedback on modeloutputs using a three-step learning algorithm.First, we condition the language model on theinitial output and feedback to generate manyrefinements. Second, we choose the refinementwith the highest similarity to the feedback.Third, we finetune a language model tomaximize the likelihood of the chosen refinementgiven the input. In synthetic experiments,we first evaluate whether language models accuratelyincorporate feedback to produce refinements,finding that only large languagemodels (175B parameters) do so. Using only100 samples of human-written feedback, ourlearning algorithm finetunes a GPT-3 model toroughly human-level summarization ability.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Language Models (LMs) achieve strong performance across diverse NLP tasks, from summarization to question answering and conversational assistants <ref type="bibr">(Radford and Narasimhan, 2018;</ref><ref type="bibr">Radford et al., 2019;</ref><ref type="bibr">Brown et al., 2020;</ref><ref type="bibr">Rae et al., 2021, interalia)</ref>. A key problem with LMs is that they generate text that violates human preferences, such as LM-generated misinformation <ref type="bibr">(Lin et al., 2021)</ref>, offensive language <ref type="bibr">(Gehman et al., 2020)</ref>, and factually incorrect outputs such as summaries <ref type="bibr">(Stiennon et al., 2020)</ref>  highly according to human preferences, or a predictive model thereof <ref type="bibr">(Ziegler et al., 2019;</ref><ref type="bibr">Stiennon et al., 2020;</ref><ref type="bibr">Nakano et al., 2021;</ref><ref type="bibr">Ouyang et al., 2022)</ref>. In this line of work, human evaluators indicate their preferences by comparing text outputs. However, each comparison provides little information per evaluation about human preferences.</p><p>We propose to use natural language feedback, which contains more information per evaluation. We introduce a three-step learning algorithm, as shown in Figure <ref type="figure">1</ref>. First, we condition an LM on an input, model-generated output, and humanwritten feedback to sample many possible refinements of the output. Second, we choose the re-finement with the highest embedding-based similarity with the feedback. Third, we finetune an LM on the chosen refinements. Our algorithm departs from prior work, which uses reinforcement learning methods <ref type="bibr">(Ziegler et al., 2019, inter alia)</ref> or auxiliary losses <ref type="bibr">(Stacey et al., 2021)</ref> that cannot be straightforwardly generalized to use natural language feedback.</p><p>We validate our algorithm on a carefullycontrolled synthetic task of removing offensive words from a sentence with GPT-3-based models <ref type="bibr">(Brown et al., 2020;</ref><ref type="bibr">Ouyang et al., 2022)</ref>. We find that only the largest GPT-3-based models (175B parameters) accurately refine outputs. Using the above insight, we use the largest GPT-3 models to test our algorithm on text summarization, following <ref type="bibr">Stiennon et al. (2020)</ref>. A model trained with our algorithm generates summaries that human evaluators prefer to human reference summaries &#8764;51% of the time. We obtain these results when learning from only 100 samples of natural language feedback. Our analysis shows that LM-generated refinements typically do incorporate the feedback, especially when choosing the refinement with the highest similarity with the feedback. Overall, our results suggest that natural language feedback is a promising avenue for learning from human preferences.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Method</head><p>Here, we define our problem formulation more formally. Given an input x, we seek to generate an output y that is high quality according to human preference judgments. We assume access to natural language feedback f on an initial model-generated output y given the input x.</p><p>To tackle the above problem, we leverage the ability of pretrained LMs to follow instructions <ref type="bibr">(Radford et al., 2019;</ref><ref type="bibr">Sanh et al., 2021;</ref><ref type="bibr">Wei et al., 2022;</ref><ref type="bibr">Ouyang et al., 2022)</ref>. We assume access to an LM that takes an input (e.g., a task instruction) and produces a distribution over text completions (e.g., a task output). We instruct the LM to refine the initial output y given the input x and feedback f . We then sample N refinements y 1 , . . . , y N from the LM. Refinements may vary in quality, so we introduce a function S that scores refinements for how effectively they incorporate feedback. We choose the refinement with the highest score from S and finetune a model on all chosen y given x. We use the resulting model to generate Experimental Setup We instruct an LM to refine an automatically-generated sentence with &#8804; 10 offensive words by removing &#8804; 3 specific words (see Appendix B for a detailed explanation and examples). We evaluate how often the generated refinement exactly matches the target sentence, which we also automatically generate. For our LMs, we use differently-sized GPT-3 models <ref type="bibr">(Brown et al., 2020)</ref> and their finetuned, In-structGPT counterparts <ref type="bibr">(Ouyang et al., 2022)</ref>. <ref type="foot">1</ref> We report all hyperparameters used in Appendix E. We report mean and std. error for all results in our work.</p><p>Results Table <ref type="table">1</ref> shows the results. We observe that only the largest GPT-3 and InstructGPT models (175B parameters) incorporate feedback a nonnegligible amount of time. Using this insight, we only use the 175B parameter (Davinci) models in the rest of our experiments.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Text Summarization</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.1">Experimental Setup</head><p>Generating Refinements We now evaluate our algorithm on the real-world task of text summarization. We follow prior work on learning from human preferences <ref type="bibr">(Stiennon et al., 2020)</ref> and learn to summarize Reddit posts from <ref type="bibr">V&#246;lske et al. (2017)</ref>.</p><p>We take 100 samples from the Reddit data subset used in <ref type="bibr">Stiennon et al. (2020)</ref>. We use InstructGPT (175B) to generate initial summaries and refinements, using the instructions in Appendix F. We then write feedback f on the initial summary y given the Reddit post x, and we generate possible refinements y 1 , . . . , y 20 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Scoring Refinements</head><p>We choose a refinement with a scoring function S that scores refinements for how effectively they incorporate feedback. For S, we use contrastive pre-trained text embedding function E <ref type="bibr">(Neelakantan et al., 2022)</ref> to embed the feedback f and refinements y 1 , . . . , y 20<ref type="foot">foot_1</ref> . We then choose the refinement with the highest cosine similarity score with the feedback. We opted for high similarity with the feedback because feedback often describes what the ideal or improved text would look like. We refer to refinements generated with the above algorithm as REFINEMENT WITH FEEDBACK + BEST OF N.</p><p>Finetuning We finetune <ref type="bibr">GPT-3 (175B;</ref><ref type="bibr">Brown et al., 2020)</ref> <ref type="foot">foot_2</ref> on refinements generated by REFINE-MENT WITH FEEDBACK + BEST OF N. We compare against finetuning on INITIAL SUMMARIES generated with InstructGPT. We also compare against summaries generated directly by Instruct-GPT and GPT-3 (175B). We use the same instructions as for INITIAL SUMMARIES (in Appendix F) and provide the post and its title.</p><p>Evaluation We test on 100 unseen Reddit posts from the same dataset and conduct human evaluations for all experiments. <ref type="foot">4</ref> Evaluators rank the summaries according to the rubric in Appendix C, with ties allowed. We show the win rate of an algorithm, counting ties as a half win, similar to Kendall rank correlation. <ref type="foot">5</ref> . We refer to Appendix C for a description of all human evaluation and feedback annotation procedures and Appendix D for more details about the ranking scheme. N generates summaries on par with human summaries, with a win rate of 51.0 &#177; 5.0% over human summaries. In contrast, all baselines underperform human summaries, with win rates of 19.0 &#177; 3.9% (GPT-3), 35.0 &#177; 4.8% (InstructGPT), 44.0 &#177; 5.0% (finetuning on INITIAL SUMMARIES)). In particular, our approach achieves a win rate of 57.0&#177;5.0% over the strongest baseline, finetuning on INITIAL SUMMARIES. Our result suggests that our learning algorithm produces higher-quality summaries by finetuning on the higher-quality targets (our refinements). Overall, we achieve strong results on summarization while learning from only 100 samples of human-written feedback.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.2">Main Results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.3">Analysis</head><p>We now aim to examine the importance of various aspects of our algorithm for generating highquality refinements (before finetuning). We evaluate REFINEMENT WITH FEEDBACK, which randomly chooses a refinement &#8712; y 1 , . . . , y 20 . This ablation helps to evaluate the importance of choosing a refinement with our scoring function S. We also evaluate REFINEMENT WITHOUT FEEDBACK, which instructs the LM to refine the initial summary but without feedback. This ablation helps to evaluate the importance of using the feedback.  <ref type="figure">4</ref> shows win rates of refinements generated with various methods against HUMAN SUM-MARIES and Appendix Fig. <ref type="figure">6</ref> shows that refinements are more helpful when the initial summary is of lower quality. We also refer to Appendix G for 10 random examples of INITIAL SUMMARIES, feedback, and refinements from various methods. Overall, using feedback and scoring refinements are both important steps for generating high-quality refinements of the initial output.</p><p>Here, we examine whether refinements are of higher quality because they incorporate the feedback, rather than by improving the summary in other ways. To do so, we evaluate how often the refinements incorporate the human-written feedback. We evaluate (1) how often &#8805; 1 point mentioned in the feedback is incorporated in the refinement, (2) how often &gt; 1 point is incorporated, and (3) how often all of the feedback is incorporated. In Fig. <ref type="figure">3</ref> (right), we see that our algorithm incorporates &#8805; 1 feedback point 72.0 &#177; 4.5% of the time, showing that LMs are able to incorporate feedback with high accuracy. REFINEMENTS WITHOUT FEED-BACK only incorporates at least one feedback point 15.0 &#177; 3.6% of the time. Our results suggest that refinements are high-quality because they incorporate specific points in the feedback.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Additional Related Work</head><p>Existing work in NLP primarily investigates using explanations for labeled outputs to classification tasks. In contrast, we do not assume access to goldlabeled outputs, and we study the more general text generation setting, which classification tasks can be formulated as <ref type="bibr">(Radford et al., 2019;</ref><ref type="bibr">Raffel et al., 2020;</ref><ref type="bibr">Brown et al., 2020)</ref>. Explanations describe why a labeled output is correct, while feedback describes how to improve a candidate output. Prior work explores ways of using explanations to train text classification models, with mixed results <ref type="bibr">(Camburu et al., 2018;</ref><ref type="bibr">Stacey et al., 2021;</ref><ref type="bibr">Pruthi et al., 2021;</ref><ref type="bibr">Wiegreffe et al., 2021;</ref><ref type="bibr">Hase and Bansal, 2021;</ref><ref type="bibr">Lampinen et al., 2022, inter alia)</ref>. A few prior works also learn from language feedback, for the purpose of ranking candidate outputs rather than generating outputs <ref type="bibr">(Weston, 2016;</ref><ref type="bibr">Li et al., 2016;</ref><ref type="bibr">Hancock et al., 2019;</ref><ref type="bibr">Li et al., 2022)</ref>. Matiana et al. ( <ref type="formula">2021</ref>) learn text embeddings of language feedback, where improvements could benefit the refinement-scoring step of our algorithm.</p><p>Outside of text domains, there is abundant work in reinforcement learning that leverages language in various ways (see <ref type="bibr">Luketina et al., 2019, for an overview)</ref>. Prior work uses language to specify the task ("instruction following" <ref type="bibr">Chaplot et al., 2017;</ref><ref type="bibr">Mahmoudieh et al., 2022;</ref><ref type="bibr">Ouyang et al., 2022, inter alia)</ref>, drive exploration <ref type="bibr">(Tam et al., 2022)</ref>, infer reward functions <ref type="bibr">(Lin et al., 2022;</ref><ref type="bibr">Sumers et al., 2021;</ref><ref type="bibr">Fidler et al., 2017, inter alia)</ref>, and train the model via strong supervision <ref type="bibr">(Andreas et al., 2017;</ref><ref type="bibr">Kaplan et al., 2017)</ref>, reward shaping <ref type="bibr">(Goyal et al., 2019)</ref>, or purely with language by providing descriptions of trajectories <ref type="bibr">(Nguyen et al., 2021)</ref>. In contrast, we use language to correct faulty behavior. Other work uses language feedback at test time to correct mistakes in a model's behavior, for e.g. image segmentation <ref type="bibr">(Rupprecht et al., 2018)</ref> or code generation <ref type="bibr">(Elgohary et al., 2020;</ref><ref type="bibr">Austin et al., 2021)</ref>. In contrast, we use feedback to train models, and our approach does not require human intervention at test time.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Conclusion</head><p>In this work, we proposed an algorithm for training LMs to behave in line with human preferences, by learning from natural language feedback. We validated our approach on a carefully-controlled word-removal task, showing that only large LMs (175B parameters) accurately incorporate feedback. Using this insight, we then tested our algorithm on the real-world task for text summarization. Our finetuning algorithm brought a GPT-3 model to roughly human-level summarization ability, using only 100 samples of human feedback. Language feedback is a natural form of communicating with models which may make it easier for many people to provide informative, high-quality feedback. In the long run, our work suggests many exciting avenues for future work, e.g., in guiding models with language feedback in other domains from code generation to conversational assistance. .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>A Additional Results</head><p>Here, we report additional results. In Fig. <ref type="figure">4</ref>, we report the win rate of our various refinement methods over HUMAN SUMMARIES. In Fig. <ref type="figure">5</ref>, we report the win rate of our learning algorithm over InstructGPT (used to generate the INITIAL SUMMARIES). Fig. <ref type="figure">6</ref> shows the win rate of various refinement methods by the rank of the INITIAL SUMMARIES relative to summaries from other methods. RE-FINEMENT WITH FEEDBACK (+ BEST OF N) have high win rates relative to other methods when theINITIAL SUMMARY is poorly ranked. When the INITIAL SUMMARY rank is 4, REFINEMENT WITH FEEDBACK has a win rate of 83.0 &#177; 3.9% vs. 49.0 &#177; 5.4% for REFINEMENT WITHOUT FEED-BACK. On the other hand, when the INITIAL SUM-MARY is higher quality (rank 2), the win rate of REFINEMENT WITH FEEDBACK is only 7.8&#177;4.0% vs. 31.25 &#177; 5.8% for REFINEMENTS WITHOUT FEEDBACK. The result is intuitive, as feedback on a bad summary should be more helpful than feedback on a good summary since there is more room for improvement on a bad summary. We observe that the worse the initial summaries, the better the refinements are.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B Targeted Word Removal Details</head><p>Below is an example of how we instruct or "prompt" an LM to remove specific, offensive words from a sentence.</p><p>"In this text, many toxic and offensive words are used: You are such a jerk, and a nice person, and an idiot. The ideal text should remove the word jerk, but otherwise be unchanged: You are"</p><p>Here, the target completion is " such a nice person and an idiot." More formally, we sample offensive sentences by using k offensive words from a fixed set of 25 offensive words, drawn uniformly at random (without replacement). Each offensive sentence also includes the words "nice person" in addition to all the offensive words. For each k &#8712; {1, . . . , 10}, we sample 50 offensive sentences. The task is then to remove l &#8712; [1, 2, 3] offensive words from a given sentence, with k &#8805; l. Since we include the words "nice person" in the offensive sentence, we can remove l = k offensive words and still have a target sentence that intuitively makes sense.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C Human Feedback and Evaluation</head><p>Automated metrics such as ROUGE <ref type="bibr">(Lin, 2004)</ref> do not correlate well with human preferences for summarization <ref type="bibr">(Paulus et al., 2018;</ref><ref type="bibr">Stiennon et al., 2020)</ref>, so we conduct human evaluations to evaluate various methods. We show all instructions given to annotators and evaluators here. Two of our authors wrote feedback on the INITIAL SUM-MARIES. To annotate feedback, they had access to the title, post, and the INITIAL SUMMARY. One author conducted the human evaluation for how often refinement incorporates feedback (Fig. <ref type="figure">3</ref>; right). For this evaluation, we provided the title, post, INI-TIAL SUMMARY, feedback, and summaries generated by various methods. Lastly, two authors not involved in feedback annotation conducted the human evaluation of generated refinements (Fig. <ref type="figure">3</ref> left, Fig. <ref type="figure">4</ref>, and Fig. <ref type="figure">6</ref>). The annotators had access to title, post, and summaries generated with various methods. See Appendix D for more detail about our ranking procedure. One author conducted the human evaluation for Figs. <ref type="figure">2</ref> and<ref type="figure">5</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D Details about Ranking Procedure</head><p>We use a standard ranking scheme where each of 5 summaries is given a rank between 1 and 5 (inclusive). The method REFINEMENT WITHOUT FEED-BACK often generates refinements that are exact copies of the INITIAL SUMMARIES, so we allow for ties in our ranking scheme. We assign the rank r to all summaries ranked in a tie, where r = r+(r+n-1) 2</p><p>, r is the rank of the tied elements, and n is the number of ties at the rank. For example, we map a ranking of (1, 2, 2, 4, 5) &#8594; (1, 2.5, 2.5, 4, 5) and a ranking of (1, 2, 3, 3, 3) &#8594; (1, 2, 4, 4, 4).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E Hyperparameters E.1 Generating Refinements</head><p>For the targeted word removal experiments ( &#167;3.1), we use greedy decoding until 200 tokens or / n is generated. For all summarization experiments ( &#167;3.2), we sample up to 48 tokens (as in <ref type="bibr">Stiennon et al., 2020)</ref> with nucleus sampling <ref type="bibr">(Holtzman et al., 2019)</ref> with p = 0.9. We strip nonalphanumeric characters (e.g., newlines) from the beginning of sampled summaries. Due to the maximum token length, sampled summaries sometimes end with an incomplete sentence. Thus, we remove ending sentences that do not end in ".", "!", or "?".</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E.2 Finetuning</head><p>We finetune GPT-3 with 175B parameters on RE-FINEMENTS WITH FEEDBACK + BEST OF N and INITIAL SUMMARIES. We use a batch size of 1 and the default of 4 epochs, as recommended by the OpenAI API. For other hyperparameters, we conduct a hyperparameter search using 5-fold crossvalidation on our train dataset of 100 examples. We use OpenAI's default hyperparameter settings from the OpenAI API and search over the learning rate multiplier, the multiplier on the pretraining learning rate used to obtain the fine-tuning learning rate. We sweep over [0.005, 0.01, 0.025, 0.05, 0.1, 0.2] and choose 0.05. We also use OpenAI's default hyperparameter settings and sweep over the prompt loss weight or the weight used for language modeling loss on the prompt <ref type="bibr">(Radford and Narasimhan, 2018)</ref>. We sweep over [0.01, 0.025, 0.05, 0.1, 0.2] and choose 0.01. We then finetune a new model on the full 100 examples with the chosen hyperparameters. We refer to the API documentation for a more detailed description of the adjustable hyperparameters provided by OpenAI.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>F Prompt Templates</head><p>Table <ref type="table">2</ref> shows the prompts used in our experiments.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>G Examples</head><p>Table <ref type="table">3</ref> shows summaries of 10 random Reddit posts from various methods: initial modelgenerated summaries and refinement methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Methods</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Format</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>INITIAL SUMMARY</head><p>Write an excellent summary of a given text. An excellent summary is coherent, accurate, concise, and detailed, and it follows human preferences about summaries.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>TITLE: {title}</head><p>Text: {text} TL;DR: REFINEMENT WITH FEEDBACK Given a text, an initial summary, and feedback on the initial summary, write an improved summary that incorporates the feedback on the initial summary and is better than the initial summary. The improved summary is coherent, accurate, concise, and detailed, and it follows human preferences about summaries. Most importantly, the improved summary incorporates the feedback. Given a text and an initial summary, write an improved summary that is better than the initial summary. The improved summary is coherent, accurate, concise, and detailed, and it follows human preferences about summaries.  </p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0"><p>Via the OpenAI API. OpenAI does not disclose the size of the provided models, so we use estimates from Eleuther.</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1"><p>We use OpenAI's API to access the embeddings.</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_2"><p>InstructGPT cannot yet be finetuning via OpenAI's API.</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_3"><p>We plan to conduct larger-scale human evaluations in the future, to confirm our initial findings.</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="5" xml:id="foot_4"><p>Kendall Rank correlation</p></note>
		</body>
		</text>
</TEI>
