Deep reinforcement learning (RL) has emerged as a powerful method whereby agents learn complex behaviors on their own through trial and error (Kohl & Stone, 2004; Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021. Copyright 2021 by the author(s). Kober & Peters, 2011;Kober et al., 2013;Silver et al., 2017;Andrychowicz et al., 2020;Kalashnikov et al., 2018;Vinyals et al., 2019). Scaling RL to many applications, however, is yet precluded by a number of challenges. One such challenge lies in providing a suitable reward function.
For example, while it may be desirable to provide sparse rewards out of ease, they are often insufficient to train successful RL agents. Thus, to provide adequately dense signal, real-world problems may require extensive instrumentation, such as accelerometers to detect door opening (Yahya et al., 2017), thermal cameras to detect pouring (Schenck & Fox, 2017) or motion capture for object tracking (Kormushev et al., 2010;Akkaya et al., 2019;Peng et al., 2020). Despite these costly measures, it may still be difficult to construct a suitable reward function due to reward exploitation. That is, RL algorithms often discover ways to achieve high returns by unexpected, unintended means. In general, there is nuance in how we might want agents to behave, such as obeying social norms, that are difficult to account for and communicate effectively through an engineered reward function (Amodei et al., 2016;Shah et al., 2019;Turner et al., 2020). A popular way to avoid reward engineering is through imitation learning, during which a learner distills information about its objectives or tries to directly follow an expert (Schaal, 1997;Ng et al., 2000;Abbeel & Ng, 2004;Argall et al., 2009). While imitation learning is a powerful tool, suitable demonstrations are often prohibitively expensive to obtain in practice (Calinon et al., 2009;Pastor et al., 2011;Akgun et al., 2012;Zhang et al., 2018).
In contrast, humans often learn fairly autonomously, relying on occasional external feedback from a teacher. Part of what makes a teacher effective is their ability to interactively guide students according to their progress, providing corrective or increasingly advanced instructions as needed. Such an interactive learning process is also alluring for artificial agents since the agent's behavior can naturally be tailored to one's preference (avoiding reward exploitation) without requiring extensive engineering. This approach is only feasible if the feedback is both practical for a human to provide and sufficiently high-bandwidth. As such, human-in-theloop (HiL) RL (Knox & Stone, 2009;Christiano et al., 2017;MacGlashan et al., 2017) has not yet been widely adopted.
Figure 1. Illustration of our method. First, the agent engages in unsupervised pre-training during which it is encouraged to visit a diverse set of states so its queries can provide more meaningful signal than on randomly collected experience (left). Then, a teacher provides preferences between two clips of behavior, and we learn a reward model based on them. The agent is updated to maximize the expected return under the model. We also relabel all its past experiences with this model to maximize their utilization to update the policy (right).
In this work, we aim to substantially reduce the amount of human effort required for HiL learning. To this end, we present PEBBLE: unsupervised PrE-training and preference-Based learning via relaBeLing Experience, a feedback-efficient RL algorithm by which learning is largely autonomous and supplemented by a practical number of binary labels (i.e. preferences) provided by a supervisor. Our method relies on two main, synergistic ingredients: unsupervised pre-training and off-policy learning (see Figure 1). For generality, we do not assume the agent is privy to rewards from its environment. Instead, we first allow the agent to explore using only intrinsic motivation (Oudeyer et al., 2007;Schmidhuber, 2010) to diversify its experience and produce coherent behaviors. Collecting a breadth of experiences enables the teacher to provide more meaningful feedback, as compared to feedback on data collected in an indeliberate manner. The supervisor then steps in to teach the agent by expressing their preferences between pairs of clips of the agent's behavior (Christiano et al., 2017). The agent distills this information into a reward model and uses RL to optimize this inferred reward function.
Leveraging unsupervised pre-training increases the efficiency of the teacher's initial feedback; however, RL requires a large enough number of samples such that supervising the learning process is still quite expensive for humans. It is thus especially critical to enable off-policy algorithms that can reuse data to maximize the agent's, and thereby human's, efficiency. However, on-policy methods have typically been used thus far for HiL RL because of their ability to mitigate the effects of non-stationarity in reward caused by online learning. We show that by simply relabeling all of the agent's past experience every time the reward model is updated, we can make use and reuse of all the agent's collected experience to improve sample and feedback efficiency by a large margin. Source code and videos are available at https://sites.google. com/view/icml21pebble.
We summarize the main contributions of PEBBLE:
• For the first time, we show that unsupervised pre-training and off-policy learning can significantly improve the sample-and feedback-efficiency of HiL RL.
• PEBBLE outperforms prior preference-based RL baselines on complex locomotion and robotic manipulation tasks from DeepMind Control Suite (DMControl; Tassa et al. 2018;2020) and Meta-world (Yu et al., 2020).
• We demonstrate that PEBBLE can learn behaviors for which a typical reward function is difficult to engineer very efficiently.
• We also show that PEBBLE can avoid reward exploitation, leading to more desirable behaviors compared to an agent trained with respect to an engineered reward function.
Learning from human feedback. Several works have successfully utilized feedback from real humans to train agents where it is assumed that the feedback is available at all times (Pilarski et al., 2011;MacGlashan et al., 2017;Arumugam et al., 2019). Due to this high feedback frequency, these approaches are difficult to scale to more complex learning problems that require substantial agent experience.
Better suited to learning in complex domains is to learn a reward model so the agent can learn without a supervisor's perpetual presence. One simple yet effective direction in reward learning is to train a classifier that recognizes task success and use it as basis for a reward function (Pinto & Gupta, 2016;Levine et al., 2018;Fu et al., 2018;Xie et al., 2018). Positive examples may be designated or reinforced through human feedback (Zhang et al., 2019;Singh et al., 2019;Smith et al., 2020). Another promising direction has focused on simply training a reward model via regres-sion using unbounded real-valued feedback (Knox & Stone, 2009;Warnell et al., 2018), but this has been challenging to scale because it is difficult for humans to reliably provide a particular utility value for certain behaviors of the RL agent.
Much easier for humans is to make relative judgments, i.e., comparing behaviors as better or worse. Preference-based learning is thus an attractive alternative because the supervision is easy to provide yet information-rich (Akrour et al., 2011;Pilarski et al., 2011;Akrour et al., 2012;Wilson et al., 2012;Sugiyama et al., 2012;Wirth & Fürnkranz, 2013;Wirth et al., 2016;Sadigh et al., 2017;Biyik & Sadigh, 2018;Leike et al., 2018;Biyik et al., 2020). Christiano et al. (2017) scaled preference-based learning to utilize modern deep learning techniques-they learn a reward function, modeled with deep neural networks, that is consistent with the observed preferences and use it to optimize an agent using RL. They choose on-policy RL methods (Schulman et al., 2015;Mnih et al., 2016) since they are more robust to the non-stationarity in rewards caused by online learning. Although they demonstrate that preference-based learning provides a fairly efficient (requiring feedback on less than 1% of the agent's experience) means of distilling information from feedback, they rely on notoriously sampleinefficient on-policy RL, so a large burden can yet be placed on the human. Subsequent works have aimed to improve the efficiency of this method by introducing additional forms of feedback such as demonstrations (Ibarz et al., 2018) or non-binary rankings (Cao et al., 2020). Our proposed approach similarly focuses on developing a more sample-and feedback-efficient preference-based RL algorithm without adding any additional forms of supervision. Instead, we enable off-policy learning as well as utilize unsupervised pre-training to substantially improve efficiency.
Unsupervised pre-training for RL. Unsupervised pretraining has been studied for extracting strong behavioral priors that can be utilized to solve downstream tasks efficiently in the context of RL (Daniel et al., 2016;Florensa et al., 2018;Achiam et al., 2018;Eysenbach et al., 2019;Sharma et al., 2020). Specifically, agents are encouraged to expand the boundary of seen states by maximizing various intrinsic rewards, such as prediction errors (Houthooft et al., 2016;Pathak et al., 2017;Burda et al., 2019), count-based state novelty (Bellemare et al., 2016;Tang et al., 2017;Ostrovski et al., 2017), mutual information (Eysenbach et al., 2019) and state entropy (Hazan et al., 2019;Lee et al., 2019;Hao & Pieter, 2021). Such unsupervised pre-training methods allow learning diverse behaviors without extrinsic rewards, effectively facilitating accelerated learning of downstream tasks. In this work, we show that unsupervised pre-training enables a teacher to provide more meaningful signal by showing them a diverse set of behaviors.
Reinforcement learning. We consider a standard RL framework where an agent interacts with an environment in discrete time. Formally, at each timestep t, the agent receives a state s t from the environment and chooses an action a t based on its policy π. The environment returns a reward r t and the agent transitions to the next state s t+1 .
The return R t = ∞ k=0 γ k r t+k is the discounted sum of rewards from timestep t with discount factor γ ∈ [0, 1). RL then maximizes the expected return from each state s t .
Soft Actor-Critic. SAC (Haarnoja et al., 2018) is an offpolicy actor-critic method based on the maximum entropy RL framework (Ziebart, 2010), which encourages exploration and greater robustness to noise by maximizing a weighted objective of the reward and the policy entropy. To update the parameters, SAC alternates between a soft policy evaluation and a soft policy improvement. At the soft policy evaluation step, a soft Q-function, which is modeled as a neural network with parameters θ, is updated by minimizing the following soft Bellman residual:
where τ t = (s t , a t , s t+1 , r t ) is a transition, B is a replay buffer, θ are the delayed parameters, and α is a temperature parameter. At the soft policy improvement step, the policy π φ is updated by minimizing the following objective:
SAC enjoys good sample-efficiency relative to its on-policy counterparts by reusing its past experiences. However, for the same reason, SAC is not robust to a non-stationary reward function.
Reward learning from preferences. We follow the basic framework for learning a reward function r ψ from preferences in which the function is trained to be consistent with observed human feedback (Wilson et al., 2012;Christiano et al., 2017). In this framework, a segment σ is a sequence of observations and actions {s k , a k , ..., s k+H , a k+H }. We elicit preferences y for segments σ 0 and σ 1 , where y is a distribution indicating which segment a human prefers, i.e., y ∈ {(0, 1), (1, 0), (0.5, 0.5)}. The judgment is recorded in a dataset D as a triple (σ 0 , σ 1 , y). By following the Bradley-Terry model (Bradley & Terry, 1952), we model a preference predictor using the reward function r ψ as follows:
where σ i σ j denotes the event that segment i is preferable to segment j. Intuitively, this can be interpreted as assuming the probability of preferring a segment depends exponentially on the sum over the segment of an underlying reward function. While r ψ is not a binary classifier, learning r ψ amounts to binary classification with labels y provided by a supervisor. Concretely, the reward function, modeled as a neural network with parameters ψ, is updated by minimizing the following loss:
In this section, we present PEBBLE: unsupervised PrEtraining and preference-Based learning via relaBeLing Experience, an off-policy actor-critic algorithm for HiL RL. Formally, we consider a policy π φ , Q-function Q θ and reward function r ψ , which are updated by the following processes (see Algorithm 2 for the full procedure):
•
Step 0 (unsupervised pre-training): We pre-train the policy π φ only using intrinsic motivation to explore and collect diverse experiences (see Section 4.1).
•
Step 1 (reward learning): We learn a reward function r ψ that can lead to the desired behavior by getting feedback from a teacher (see Section 4.2).
•
Step 2 (agent learning): We update the policy π φ and Q-function Q θ using an off-policy RL algorithm with relabeling to mitigate the effects of a non-stationary reward function r ψ (see Section 4.3).
Step 1 and Step 2.
In our setting, we assume the agent is given feedback in the form of preferences between segments. In the beginning of training, though, a naive agent executes a random policy, which does not provide good state coverage nor coherent behaviors. The agent's queries are thus quite limited and likely difficult for human teachers to judge. As a result, it requires many samples (and thus queries) for these methods to show initial progress. Recent work has addressed this issue by means of providing demonstrations; however, this is not ideal since these are notoriously hard to procure (Ibarz et al., 2018). Instead, our insight is to produce informative queries at the start of training by utilizing unsupervised pretraining to collect diverse samples solely through intrinsic motivation (Oudeyer et al., 2007;Schmidhuber, 2010).
Specifically, we encourage our agent to visit a wider range of states by using the state entropy H(s) = Algorithm 1 EXPLORE: Unsupervised exploration 1: Initialize parameters of Q θ and π φ and a replay buffer B ← ∅ 2: for each iteration do 3: for each timestep t do 4:
Collect st+1 by taking at ∼ π φ (at|st) 5:
Compute intrinsic reward r int t ← r int (st) as in (5) 6:
Store transitions B ← B ∪ {(st, at, st+1, r int t )} 7:
end for 8:
for each gradient step do 9:
Sample minibatch { sj, aj, sj+1, r int j } B j=1 ∼ B 10:
Optimize L SAC critic in (1) and L SAC act in (2) with respect to θ and φ 11:
end for 12: end for 13: return B, π φ -E s∼p(s) [log p(s)] as an intrinsic reward (Hazan et al., 2019;Lee et al., 2019;Hao & Pieter, 2021;Seo et al., 2021). By updating the agent to maximize the sum of expected intrinsic rewards, it can efficiently explore an environment and learn how to generate diverse behaviors. However, this intrinsic reward is intractable to compute in most settings. To handle this issue, we employ the simplified version of particle-based entropy estimator (Beirlant et al., 1997;Singh et al., 2003) (see the supplementary material for more details):
where H denotes the particle-based entropy estimator and s k i is the k-th nearest neighbor (k-NN) of s i . This implies that maximizing the distance between a state and its nearest neighbor increases the overall state entropy. Inspired by this, we define the intrinsic reward of the current state s t as the distance between s t and its k-th nearest neighbor by following the idea of Hao & Pieter (2021) that treats each transition as a particle:
In our experiments, we compute k-NN distances between a sample and all samples in the replay buffer and normalize the intrinsic reward by dividing it by a running estimate of the standard deviation. The full procedure of unsupervised pre-training is summarized in Algorithm 1.
As previously mentioned, we learn our reward function by modeling the probability that a teacher prefers one sampled segment over another as proportional to the exponentiated sum of rewards over the segment (see Section 3). Ideally, one should solicit preferences so as to maximize expected value of information (EVOI; Savage 1972): the improvement of an agent caused by optimizing with respect to the resulting reward model (Viappiani, 2012;Akrour et al., 2012). if iteration % K == 0 then 9:
for m in 1 . . . M do 10:
(σ 0 , σ 1 ) ∼ SAMPLE() (see Section 4.2) 11:
Query instructor for y 12:
Store preference D ← D ∪ {(σ 0 , σ 1 , y)} 13:
end for 14:
for each gradient step do 15:
Sample minibatch {(σ 0 , σ 1 , y)j} D j=1 ∼ D 16:
Optimize L Reward in (4) with respect to ψ 17:
end for 18:
Relabel entire replay buffer B using r ψ 19:
end if 20:
for each timestep t do 21:
Collect st+1 by taking at ∼ π φ (at|st) 22:
Store transitions B ← B ∪ {(st, at, st+1, r ψ (st))} 23:
end for 24:
for each gradient step do 25:
Sample random minibatch {(τj)} B j=1 ∼ B 26:
Optimize L SAC critic in (1) and L SAC act in (2) with respect to θ and φ, respectively 27: end for 28: end for Computing the EVOI is intractable since it involves taking an expectation over all possible trajectories induced by the updated policy. To handle this issue, several approximations have been explored by prior works to sample queries that are likely to change the reward model (Daniel et al., 2014;Christiano et al., 2017;Ibarz et al., 2018). In this work, we consider the sampling schemes employed by Christiano et al. (2017): (1) uniform sampling and (2) ensemble-based sampling, which selects pairs of segments with high variance across ensemble reward models. We explore an additional third method, entropy-based sampling, which seeks to disambiguate pairs of segments nearest the decision boundary. That is, we sample a large batch of segment pairs and select pairs that maximize H(P ψ ). We evaluate the effects of these sampling methods in Section 5.
Once we learn a reward function r ψ , we can update the policy π φ and Q-function Q θ using any RL algorithm. A caveat is that the reward function r ψ may be non-stationary because we update it during training. Christiano et al. (2017) used on-policy RL algorithms, TRPO (Schulman et al., 2015) and A2C (Mnih et al., 2016), to address this issue. However, their poor sample-efficiency leads to poor feedbackefficiency of the overall HiL method. In this work, we use an off-policy RL algorithm, which provides for sample-efficient learning by reusing past experiences that are stored in the replay buffer. However, the learning process of off-policy RL algorithms can be unstable because previous experiences in the replay buffer are labeled with previous learned rewards. To handle this issue, we relabel all of the agent's past experience every time we update the reward model. We find that this simple technique stabilizes the learning process and provides large gains in performance (see Figure 5(a) for supporting results). The full procedure ofPEBBLE is summarized in Algorithm 2.
We design our experiments to investigate the following:
1. How does PEBBLE compare to existing methods in terms of sample and feedback efficiency?
2. What is the contribution of each of the proposed techniques in PEBBLE?
3. Can PEBBLE learn novel behaviors for which a typical reward function is difficult to engineer?
4. Can PEBBLE mitigate the effects of reward exploitation?
We evaluate PEBBLE on several continuous control tasks involving locomotion and robotic manipulation from Deep-Mind Control Suite (DMControl; Tassa et al. 2018;2020) and Meta-world (Yu et al., 2020). In order to verify the efficacy of our method, we first focus on having an agent solve a range of tasks without being able to directly observe the ground truth reward function. Instead, similar to Christiano et al. (2017) and Ibarz et al. (2018), the agent learns to perform a task only by getting feedback from a scripted teacher that provides preferences between trajectory segments according to the true, underlying task reward. Because this scripted teacher's preferences are immediately generated by a ground truth reward, we are able to evaluate the agent quantitatively by measuring the true average return and do more rapid experiments. For all experiments, we report the mean and standard deviation across ten runs.
We also run experiments with actual human trainers (the authors) to show the benefits of human-in-the-loop RL. First, we show that human trainers can teach novel behaviors (e.g., waving a leg), which are not defined in original benchmarks. Second, we show that agents trained with the handengineered rewards from benchmarks can perform the task in an undesirable way (i.e., the agent exploits a misspecified reward function), while agents trained using human feedback can perform the same task in the desired way. For all experiments, each trajectory segment is presented to the human as a 1 second video clip, and a maximum of one hour of human time is required.
For evaluation, we compare to Christiano et al. (2017), which is the current state-of-the-art approach using the same type of feedback. The primary differences in our method are (1) the introduction of unsupervised pre-training, (2) the accommodation of off-policy RL, and (3) entropy-based sampling. We re-implemented Christiano et al. ( 2017) using the state-of-the-art on-policy RL algorithm: PPO (Schulman et al., 2017). We use the same reward learning framework and ensemble disagreement-based sampling as they proposed. We refer to this baseline as Preference PPO.
As an upper bound, since we evaluate against the task reward function, we also compare to SAC (Haarnoja et al., 2018) and PPO using the same ground truth reward. For our method, we pre-train an agent for 10K timesteps and include these pre-training steps in all learning curves. We do not alter any hyperparameters of the original SAC algorithm and use an ensemble of three reward models. Unless stated otherwise, we use entropy-based sampling. More experimental details including model architectures, sampling schemes, and reward learning are in the supplementary material.
Locomotion tasks from DMControl. Figure 3 shows the learning curves of PEBBLE with 1400, 700 or 400 pieces of feedbackfoot_0 and that of Preference PPO with 2100 or 1400 pieces of feedback on three complex environments: Cheetahrun, Walker-walk and Quadruped-walk. Note that we explicitly give preference PPO an advantage by providing it with more feedback. We find that given a budget of 1400 queries, PEBBLE (green) reaches the same performance as SAC (pink) while Preference PPO (purple) is unable to match PPO (black). That PEBBLE requires less feedback than Preference PPO to match its respective oracle performance corroborates that PEBBLE is indeed more feedbackefficient. These results demonstrate that PEBBLE can enable the agent to solve the tasks without directly observing the ground truth reward function.
For further analysis, we incorporated our pre-training with Preference PPO (red) and find that it improves performance for Quadruped and Walker. We emphasize that our insight of using pre-training is able to improve both methods in terms of feedback-efficiency and asymptotic performance, but PEBBLE is uniquely positioned to benefit as it is able to utilize unsupervised experience for policy learning.
Robotic manipulation tasks from Meta-world. One application area in which HiL methods could have significant real-world impact is robotic manipulation, since learning often requires extensive engineering in the real world (Yahya et al., 2017;Schenck & Fox, 2017;Kormushev et al., 2010;Rusu et al., 2017;Akkaya et al., 2019;Peng et al., 2020). However, the common approach is to perform goalconditioned learning with classifiers (Singh et al., 2019), which can only capture limited information about what goal states are, and not about how they can be achieved. To study how we can utilize preference-based learning to perform more complex skills, we also consider six tasks covering a range of fundamental robotic manipulation skills from Meta-world (see Figure 2). As shown in Figure 4, PEB-BLE matches the performance of SAC using the ground truth reward and outperforms Preference PPO, given comparable (and more) feedback, on every task. By demonstrating the applicability of PEBBLE to learning a variety of robotic manipulation tasks, we believe that we are taking an important step towards anyone (non-experts included) being able to teach robots in real-world settings.
Contribution of each technique. In order to evaluate the individual effects of each technique in PEBBLE, we incre- mentally apply unsupervised pre-training and relabeling.
Figure 5(a) shows the learning curves of PEBBLE with 1400 queries on Quadruped-walk. First, we remark that relabeling significantly improves performance because it enables the agent to be robust to changes in its reward model. By additionally utilizing unsupervised pre-training, both sample-efficiency and asymptotic performance of PEB-BLE are further improved because showing diverse behaviors to a teacher can induce a better-shaped reward. This shows that PEBBLE's key ingredients are fruitfully wed, and their unique combination is crucial to our method's success.
Effects of sampling schemes. We also analyze the effects of different sampling schemes to select queries. Figure 5(b) shows the learning curves of PEBBLE with three different sampling schemes: uniform sampling, disagreement sampling and entropy sampling on Quadruped-walk. For this complex domain, we find that the uncertainty-based sampling schemes (using ensemble disagreement or entropy) are superior to the naive uniform sampling scheme. However, we note that they did not lead to extra gains on relatively simple environments, like Walker and Cheetah, similar to observations from Ibarz et al. (2018) (see the supplementary material for more results).
Comparison with step-wise feedback. We also measure the performance of PEBBLE by varying the length of segments. Figure 5(c) shows that feedback from longer segments (green curve) provide more meaningful signal than step-wise feedback (red curve). We believe that this is because longer segments can provide more context in reward learning.
Novel behaviors. We show that agents can perform various novel behaviors based on human feedback using PEBBLE in Figure 6. Specifically, we demonstrate (a) the Cart agent swinging a pole (using 50 queries), (b) the Quadruped agent waving a front leg (using 200 queries), and (c) the Hopper performing a backflip (using 50 queries). We note that the human is indeed able to guide the agent in a controlled way, as evidenced by training the same agent to perform several variations of the same task (e.g., waving different legs or spinning in opposite directions). The videos of all behaviors and examples of selected queries are available in the supplementary material.
Reward exploitation. One concern in utilizing handengineered rewards is that an agent can exploit unexpected sources of reward, leading to unintended behaviors. Indeed, we find that the Walker agent learns to walk using only one leg even though it achieves the maximum scores as shown in Figure 7(b). However, using 200 human queries, we were able to train the Walker to walk in a more natural, humanlike manner (using both legs) as shown in Figure 7(a). This result clearly shows the advantage of HiL RL to avoid reward exploitation.
In this work, we present PEBBLE, a feedback-efficient algorithm for HiL RL. By leveraging unsupervised pre-training and off-policy learning, we show that sample-and feedbackefficiency of HiL RL can be significantly improved and this framework can be applied to tasks of higher complexity than previously considered by previous methods, including a variety of locomotion and robotic manipulation skills. Additionally, we demonstrate that PEBBLE can learn novel behaviors and avoid reward exploitation, leading to more desirable behaviors compared to an agent trained with respect to an engineered reward function. We believe by making preference-based learning more tractable, PEBBLE may facilitate broadening the impact of RL beyond settings in which experts can carefully craft reward functions to those in which laypeople can likewise utilize the advances of robot learning in the real world.
Figures 8 and9 show the learning curves of PEBBLE with various sampling schemes. For Quadruped, we find that the uncertainty-based sampling schemes (using ensemble disagreement or entropy) are superior to the naive uniform sampling scheme. However, they did not lead to extra gains on relatively simple environments, like Walker and Cheetah, similar to observations from Ibarz et al. (2018). Similarly, on the robotic manipulation tasks, we find little difference in performance for simpler tasks (Drawer Close, Window Open). However, we find that the uncertainty-based sampling schemes generally fare better on the other environments.
Figure 10, 11 and 12 show some examples from the selected queries to teach the agents.
One piece of feedback corresponds to one preference query.