As large pretrained models have become increasingly performant, the problem of aligning them with human preferences has risen to the forefront of research. This alignment is especially difficult when larger datasets inevitably include suboptimal behaviors. Reinforcement learning from human feedback (RLHF) has emerged as a popular solution to this problem. Using human preferences, RLHF techniques discriminate between desirable and undesirable behaviors with the goal of refining a learned policy. This paradigm has shown promising results when applied to finetuning large language models (LLMs) (Ouyang et al., 2022), improving image generation models (Lee et al., 2023), and adapting robot policies (Christiano et al., 2017) -all from suboptimal data. For most RLHF algorithms, this process includes two phases. First, a reward model is trained from user preference data. And second, that reward model is optimized by an off-the-shelf reinforcement learning (RL) algorithm.
Unfortunately, this two-phase paradigm is founded on a flawed assumption. Algorithms that learn reward models from preference data require that human preferences are distributed according to the discounted sum of rewards or partial return of each behavior segment. However, recent work (Knox et al., 2022) calls this into question, positing that humans instead provide preferences based on the regret of each behavior under the optimal policy of the expert's reward function. Intuitively, a human's judgement is likely based on optimality, instead of which states and actions have higher quantity for reward. As a result, the correct quantity to learn from feedback might not be the reward, but instead the optimal advantage function or, in other words, the negated regret.
In their second phase, two-phase RLHF algorithms optimize the reward function learned from the first phase with RL. In practice, RL algorithms suffer from a suite of optimization challenges stemming from temporal credit assignment, such as the high-variance of policy gradients (Marbach & Tsitsiklis, 2003) or instability of approximate dynamic programming (Van Hasselt et al., 2018). Thus, past works limit their scope to circumvent these issues. For instance, RLHF techniques for LLMs assume a contextual bandit formulation (Ouyang et al., 2022), where the policy receives a single reward value in response to a given query to the user. While this reduces the need for long-horizon credit assignment, and consequently the high variance of policy gradients, in reality user interactions with LLMs are multi-step and sequential, violating the single-step bandit assumption. As another example, RLHF has been applied to low-dimensional state-based robotics problems (Christiano et al., 2017;Sikchi et al., 2023a), a setting where approximate dynamic programming excels, but not yet scaled to more realistic high-dimensional continuous control domains with image inputs. Broadly, RLHF methods not only incorrectly assume that the reward function alone drives human preferences, but also require mitigating the optimization challenges of RL by making restrictive assumptions about the sequential nature of problems or dimensionality.
In this work, we introduce a new family of RLHF methods that use a regret-based model of preferences, instead of the commonly accepted partial return model that only considers the sum of rewards. Unlike the partial return model, the regret-based model directly provides information about the optimal policy. A fortunate outcome of this is that it completely eliminates the need for RL, allowing us to solve RLHF problems in the general MDP framework with high-dimensional state and action spaces. Our key insight is to combine the regret-based preference framework with the principle of Maximum Entropy (MaxEnt), resulting in a bijection between advantage functions and policies. By exchanging optimization over advantages for optimization over policies, we are able to derive a purely supervised learning objective whose optimum is the optimal policy under the expert's reward. We refer to our approach as Contrastive Preference Learning due to its resemblance with commonly accepted contrastive learning objectives.
CPL has three key benefits over prior work. First, CPL can scale as well as supervised learning because it uses only supervised objectives to match the optimal advantage without any policy gradients or dynamic programming. Second, CPL is fully off-policy, enabling effectively using any offline suboptimal data source. Finally, CPL can be applied to arbitrary Markov Decision Processes (MDPs), allowing for learning from preference queries over sequential data. To our knowledge, no prior methods for RLHF simultaneously fulfill all three of these tenets. To demonstrate CPL's adherence to the three aforementioned tenets, we show its effectiveness on sequential decision making problems with sub-optimal and high-dimensional off-policy data. Notably, we show that CPL can effectively use the same RLHF fine tuning procedure as dialog models to learn temporally extended manipulation policies in the MetaWorld Benchmark. Specifically, we pretrain policies using supervised learning from high-dimensional image observations, before fine tuning them with preferences. Without dynamic programming or policy gradients, CPL is able to match the performance of prior RL based methods. At the same time, it is 1.6× faster and four times as parameter efficient. When using denser preference data, CPL is able to surpass the performance of RL baselines on 5 out of 6 tasks.
We consider the general reinforcement learning from human feedback (RLHF) problem within a reward-free MDP M/r = (S, A, p, γ) with state space S, action space A, transition dynamics p(s t+1 |s t , a t ), and discount factor γ. We assume all states are reachable by some policy. The goal of RLHF is to learn a policy π(a|s) that maximizes an expert user's reward function r E (s, a). However, since the reward function is not given in an MDP /r, it must be inferred from the expert's preferences. Typically, a user preference orders two behavior segments. A length-k segment is denoted σ = (s 1 , a 1 , s 2 , a 2 , . . . , s k , a k ). We use σ + ≻ σ -to indicate that segment σ + was preferred to σ - by the user without loss of generality and assume we are given a dataset
Phase 1 Reward Learning Phase 2 RL 𝑟 𝜃 𝜎 - 𝜎 + RL Algorithm 𝜋 𝜃 (𝑎|𝑠) Contrastive Preference Learning Contrastive Learning 𝜎 - 𝜎 + 𝜎 + log 𝜋 𝜃 (𝑎 𝑡 + |𝑠 𝑡 + ) 𝜎 - log 𝜋 𝜃 (𝑎 𝑡 -|𝑠 𝑡 -) Regret-based Preferences 𝐿 𝐶𝑃𝐿 = -𝔼 log 𝑃 log 𝜋 𝜃 𝜎 + ≻ 𝜎 - 𝑎 + 𝑎 - 𝜋 𝜃 (𝑎|𝑠) 𝑃 𝐴 * 𝜎 + ≻ 𝜎 -= 𝑒 σ 𝜎 + 𝐴 * (𝑠 𝑡 + ,𝑎 𝑡 + )
𝑒 σ 𝜎 + 𝐴 * (𝑠 𝑡 + ,𝑎 𝑡 + ) + 𝑒 σ 𝜎 -𝐴 * (𝑠 𝑡 -,𝑎 𝑡 -)
Figure 1: While most RLHF algorithms use a two-phase reward learning, then RL approach, CPL directly learns a policy using a contrastive objective. This is enabled by the regret preference model.
Maximum Entropy Reinforcement Learning. The aim of maximum-entropy reinforcement learning is to learn a policy π that maximizes its causal entropy in addition to the cumulative discounted return, leading to the objective:
where α is a temperature parameter. Augmenting the reward function with an additional negated log µ(a|s) term for reference distribution µ(a|s) yields the KL-constrained objective used in offline RL (Levine & Koltun, 2013;Garg et al., 2023) and prominent RLHF approaches for LLMs (Ziegler et al., 2019;Ouyang et al., 2022). Though we adopt the standard maximum entropy framework, our approach easily extends to the constrained setting. Under policy π and reward function r, we denote the state-value function by V π r (s) and state-action value function by
, measures how much worse taking action a is than acting according to π. We use π * as short-hand for the solution to Eq. ( 1) with reward function r E , and write its corresponding corresponding value functions as V * (s) and Q * (s, a) instead of V π * r E and Q π * r E . We measure the optimality of behavior directly by using the advantage function of π * , A * (s, a).
The Regret (or Advantage) Preference Model. Learning π * requires characterizing how preferences are generated according to a preference model P E [σ + ≻ σ -], or the probability the expert prefers σ + to σ -. Typically, the preference model is chosen to be the Boltzmann rational distribution over each segment's discounted partial return, ∑︁ k t=1 γ t r E (s t , a t ), where r E is the expert's hidden reward function. However, such models have been shown to be inconsistent with real human preferences (Knox et al., 2022). For instance, consider a sparse reward r E (s, a) = 1{s = g}. Two segments that do not reach the goal would have the same partial returns even if one moved towards the goal g while the other moved away from it. This inconsistency is resolved by considering preferences to be distributed according to the Boltzmann rational distribution over the negated discounted regret under r E , or -
In this framework, a user's preference indicates that a segment has lower regret with respect to their intended optimal policy. Leveraging the equivalence of negated regret and the discounted sum of optimal advantages, we equivalently write the regret-based preference model as
where we use the shorthand "+" and "-" as indexing the states and actions of segments σ + and σ -. In the next section, we use the regret preference model in combination with the principle of maximum causal entropy to derive CPL.
Though recent work has shown that human preferences are better modeled by the optimal advantage function or regret, most existing RLHF algorithms assume otherwise. By learning a reward function with a mistaken model of preference and then applying RL, traditional RLHF approaches incur a vast, unnecessary computational expense (Knox et al., 2023). Our aim is to derive simple and scalable RLHF algorithms that are purpose-built for the more accurate regret model of human preferences.
Modeling human preferences with regret is not new, but past work suffers from a number of shortcomings. Specifically, existing algorithms using the regret preference model are brittle, as they rely on estimating gradients with respect to a moving reward function, which thus far has only been approximated by computing successor features and assuming a correct linear or tabular representation of the expert reward function r E (Knox et al., 2022;2023). Consequently, these algorithms appear unsuitable for complex scenarios beyond the simplistic grid world environments in which they have been tested.
The key idea of our approach is simple: we recognize that the advantage function, used in regret preference model, can easily be replaced with the log-probability of the policy when using the maximum entropy reinforcement learning framework. The benefit of this simple substitution is however immense. Using the log-probability of the policy circumvents the need to learn the advantage function or grapple with optimization challenges associated with RL-like algorithms. In sum, this enables us to not only embrace a more closely aligned regret preference model, but also to exclusively rely on supervised learning when learning from human feedback.
In this section, we first derive the CPL objective and show that it converges to the optimal policy for r E with unbounded data. Then, we draw connections between CPL and other supervised-learning approaches. Finally, we provide recipes for using CPL in practice. Our algorithms are the first examples of a new class of methods for sequential decision making problems which directly learn a policy from regret based preferences without RL, making them far more efficient.
Under the regret preference model, our preference dataset D pref contains information about the optimal advantage function A * (s, a), which can intuitively be seen as a measure of how much worse a given action a is than an action generated by the optimal policy at state s. Therefore, actions that maximize the optimal advantage are by definition an optimal actions and learning the optimal advantage function from preferences should intuitively allow us to extract the optimal policy.
Naïve approach. When presented with D pref , one might naïvely follow the standard RLHF reward modeling recipe, but with advantages. This would equate to optimizing a parameterized advantage A θ to maximize the log likelihood of D pref given the preference model in Eq. (2), or
where P A θ is the preference model induced by the learned advantage function. Once an advantage function that aligns with the preference data is learned, it could be distilled into a parameterized policy. At first glance, it seems like this simple two-step approach could be used to recover the optimal policy from preference data. However, it turns out that learning a Bellman-consistent advantage function is non-trivial in both standard and MaxEnt RL, making learning a valid intermediate advantage function not only unnecessary, but also harder in practice.
Eliminating the need to learn advantage. In maximum entropy RL, Ziebart (2010) has shown that the following relationship between the optimal advantage function and optimal policy holds:
This means that in order for a learned advantage function to be optimal, it must be normalized, that is ∫︁ A e A * (s,a)/α da = 1. Enforcing this constraint is intractable, particularly in continuous spaces with large neural networks, making naïvely learning A θ via maximum likelihood estimation difficult.
However, one might instead notice that the above equation establishes a bijection between the advantage function A * r and the policy π * , namely that the optimal advantage function is proportional to the optimal policy's log-likelihood:
This means that instead of learning the optimal advantage function, we can directly learn the optimal policy. Given preferences are distributed according to the optimal advantage function for the expert reward function r E , we can write the preference model in terms of the optimal policy π * by substituting Eq. (3) into Eq. (2) as follows,
.
(4)
Thus, the maximum entropy framework has led to a model of human preferences that is solely in terms of the optimal policy π * . Using this equivalent form of the advantage-based preference model, we can directly optimize a learned policy π θ to match the preference model via maximum likelihood with the following convex objective:
Assuming sufficient representation power, at convergence π θ will perfectly model the user's preferences, and thus exactly recover π * under the advantage-based preference model given an unbounded amount of preference data. Specifically, in Appendix A, we prove the following Theorem: Theorem 1. Assume an unbounded number of preferences generated from a noisy rational regretpreference model with expert advantage function A * . CPL recovers the optimal policy π * corresponding to reward r E .
This proof relies on the bijection between optimal advantage functions and policies in maximum entropy RL and the fact that the regret preference model is identifiable (Knox et al., 2022), meaning the objective can achieve a loss of zero.
Benefits of directly learning the policy. Directly learning π in this manner has several benefits, both practical and theoretical. Perhaps most obviously, directly learning the policy circumvents the need for learning any other functions, like a reward function or value function. This makes CPL extremely simple in comparison to prior work. When scaling to larger models, only learning the policy reduces both complexity and computational cost. Second, as pointed out by prior works (Christiano et al., 2017;Hejna & Sadigh, 2023), reward learning can be harmed by the invariance of Boltzmann rational preference models (Eq. (2)) to shifts; i.e., adding a constant to each exponent does not change
In CPL the distributional constraint of the policy (π θ (a|s) ≥ 0 for all a and ∫︁ A π θ (a|s)da = 1) remedies this issue automatically, since adding a constant makes ∫︁ A π θ (a|s)da ̸ = 1. Finally, per previous arguments, the policy's distributional constraint guarantees that ∫︁ A e A θ (s,a)/α da = 1. Thus, it can be shown that CPL's learned implicit advantage function is always the optimal advantage function for some reward function. We call this property, defined below, consistency and prove the following Proposition in Appendix A. Definition 1. An advantage function A(s, a) is consistent if there exists some reward function r(s, a) for which A is the optimal advantage, or A(s, a) = A * r (s, a). Proposition 1. CPL learns a consistent advantage function.
The consequences of this are that no matter the amount of preference data used, CPL will always learn the optimal policy for some reward function, and adding additional preference data only improves the implicit estimate of r E .
Connections to Contrastive Learning. When deriving CPL, we intentionally chose to denote preferred and unpreferred behavior segments by "+" and "-" to highlight the similarities between CPL and contrastive learning approaches. Though some two-phase RLHF approaches have drawn connections between their reward learning phase and contrastive learning (Kang et al., 2023), CPL directly uses a contrastive objective for policy learning. Specifically, Eq. ( 5) is an instantiation of the Noise Constrastive Estimation objective (Gutmann & Hyvärinen, 2010) where a segment's score is its discounted sum of log-probabilities under the policy, the positive example being σ + and the negative σ -. In the appendix we show that when applied to ranking data using a Plackett-Luce Model, CPL recovers the InfoNCE objective from Oord et al. (2018) where the negative examples are all the segments ranked below the positive segment. Effectively, CPL has fully exchanged the reinforcement learning objective for a supervised, representation learning objective while still converging to the optimal policy. As marked success has been achieved applying contrastive learning objectives to large-scale datasets and neural networks (Chen et al., 2020;He et al., 2020;Radford et al., 2021), we expect CPL to scale competitively to RLHF methods that use traditional RL algorithms.
The Contrastive Preference Learning framework provides a general loss function for learning policies from advantage-based preferences, from which many algorithms can be derived. In this section, we detail practical considerations for one particular instantiation of the CPL framework which we found to work well in practice. In the appendix, we include several instantiations of CPL for different types of data and conservative regularizers.
CPL with Finite Offline Data. Though CPL converges to the optimal policy with unbounded preference data, in practice we are often interested in learning from finite offline datasets. In this setting, policies that extrapolate too much beyond the support of the dataset perform poorly as they take actions leading to out of distribution states. Like many other preference-based objectives, CPL's objective is not strictly convex (Appendix A.3). Thus, many policies, even those with a high weight on actions not in the dataset, can achieve the same optima of Eq. ( 5). We demonstrate this by formulating CPL as a logistic regression problem. Let the policy be represented by a one-dimensional vector π ∈ R |S×A| . The difference between positive and negative segments,
) can be re-written as a dot-product between π and a "comparison" vector x, whose values are either γ t , -γ t , or 0 indicating membership to the comparison σ + ≻ σ -. Using the logistic function, logistic(z) = 1 1+e -z , we re-write the CPL objective in the finite case as
denotes the tth timestep of the preferred segment from the ith comparison in D pref . We can reason about the set of all policies that yield the same CPL loss by assembling all comparison vectors into a matrix X, where the ith row of X is the vector x i for the ith comparison in the dataset. Any changes to log π in the null space of X have no effect on the logits of the logistic function, and consequently no effect on the loss. In practice, |S × A| >> n, making the null space of X often nontrivial such that there are multiple minimizers of the CPL loss, some of which potentially place a high probability on state-action pairs not in the dataset. In Appendix A.3 we provide constructions of X where this is true. Next, we show how this problem can be resolved by incorporating regularization into the CPL objective.
Regularization. In finite settings, we want to choose the policy that minimizes the CPL loss function while placing higher likelihood on actions in the dataset. To accomplish this, we modify Eq. ( 5) with a conservative regularizer that assigns lower loss when the policy has higher likelihood on actions in D pref , keeping it in-distribution. Though there are many possible choices of regularizers, we use an asymmetric "bias" regularizer adapted from An et al. (2023) as it performed best in our experiments. Within our objective, the bias regularizer down-weights negative segments by λ ∈ (0, 1) as so:
If the policy places more weight on actions in the dataset, log π θ (a|s) will increase. In the standard Boltzmann model, increasing the log-probabilities of both the positive and negative segments by the same amount would have no effect on the loss. The bias, however, weighs the increased logprobabilities of the negative segments less, which ultimately decreases the loss. Thus, while a minimizer of the vanilla CPL loss function could place a high probability on unseen actions, Eq. ( 6) is minimized with a higher weight on in-distribution actions. This is formally captured by the following proposition, which shows that, for a fixed policy, L CPL(λ) is lower when the policy places a higher likelihood on actions in the dataset versus other comparisons with the same CPL Loss. Proposition 2. Consider a comparison σ + ≻ σ -from D pref and an arbitrary comparison
Essentially, this shows that the bias regularizer breaks ties in the CPL loss function by penalizing lower likelihoods. We prove this, along with a more general version, in Appendix A.4. In Appendix B we also consider CPL variants with other forms of conservative regularization.
Pretraining. Pre-training the policy π θ with behavior cloning (BC) is a common practice in RLHF (Ouyang et al., 2022). Thus, before fine-tuning with preferences using the CPL loss, we trained the policy using the standard maximum likelihood BC objective, min θ E (s,a)∼D [log π θ (a|s)]. Though D could be any dataset, we chose D pref . We found that this helped performance in some cases, but hurt it others (Appendix C). We posit that pre-training helps when a policy closer to the data distribution is desirable, particularly when out-of-distribution actions are detrimental.
In this section, we address the following questions about CPL: First, is CPL effective at fine-tuning policies from regret-based preferences? Second, does CPL scale to high-dimensional control problems and larger networks? Third, what ingredients of CPL are important for attaining high performance? And finally, how does CPL perform with limited real-human preference data? Additional experiments and details are included in the appendix.
Preference Data. We evaluate CPL's ability to learn policies for general MDPs from sub-optimal offpolicy rollout data and preferences. In particular, we consider the training procedure commonly used for large foundation models: supervised learning followed by fine-tuning with RLHF. To do this, we use six tasks from the MetaWorld robotics benchmark (Yu et al., 2020). First, we train baseline policies until they approximately reach a 50% success rate. Then, we rollout 2500 episodes for each suboptimal stochastic policy. We then form synthetic preference datasets D pref of different sizes by sampling segments of length 64 uniformly from the rollout data. We estimate regret-based preference labels using the Q-function and policy of an oracle Soft Actor-Critic (SAC) (Haarnoja et al., 2018) model trained to 100% success on a combination of the suboptimal rollout and online data. In practice, we consider two main types of preference datasets: dense, where we label comparisons between every sampled segment (effectively ranking all segments), and sparse, where we label only one comparison per segment.
Baseline Methods. We consider four strong baselines. First, supervised fine-tuning (SFT) trains a policy with BC on all segments, then fine-tunes on only the preferred segments, i.e., all σ + in D pref .
The second baseline is Preference IQL (P-IQL), which learns a reward function from D pref assuming the partial return model, then subsequently learns a policy to maximize it with Implicit Q-Learning (Kostrikov et al., 2022), a state-of-the-art offline RL algorithm. Though P-IQL was first used with the partial return model, here it uses an approximation of A * r E as its reward, which as we show in Appendix A's Corollary 1 preserves the optimal policy. In fact, P-IQL should be more performant with regret-based labels, since A * r E is a highly shaped reward function for r E Ng et al. (1999); Knox et al. (2023). We use Hejna & Sadigh (2023)'s implementation of P-IQL as it outperformed several contemporary baselines. This baseline is also equivalent to TREX on pairwise preferences (Brown et al., 2019). Third, we consider PPO with a KL-constrained reward as commonly used for RLHF with LLMs (Ziegler et al., 2019). This is not a fair comparison, as PPO requires 3.84 million additional online state-action pairs to estimate the policy gradient, totalling 25× the data for CPL 2.5K Dense and 4× the data for CPL 20K Sparse. Unlike the contextual bandit setting used for LLMs, PPO here require full trajectory rollouts. Finally, to demonstrate CPL's ability to extrapolate beyond the best performance in the data, we compare to %BC, where a policy is trained with behavior cloning on the top X% of rollouts according to the ground truth r E .
How does CPL perform with state-based observations? Our main state-based results can be found in rows 1 and 3 of Table 1. When using sparser comparison data (row 3), CPL outperforms prior methods in 5 of 6 environments, often by a substantial margin of over P-IQL, particularly in the Button Press, Bin Picking, and Sweep Into environments. When applied to datasets with more dense comparisons (row 1), CPL performs even better. Though the dense-comparison datasets have less state-action coverage, they contain substantially more comparisons than the sparse-comparison datasets. We posit that more comparisons per segment is more beneficial to CPL than to P-IQL because of its contrastive objective -more comparison-rich datasets are likely to have more informative positive-negative pairs that help shape the policy. PPO is unable to consistently perform better than CPL despite access to vast amounts of online data from the environment. We found PPO to be very sensitive to the KL-constraint coefficient on reward, which makes it difficult to tune as observed in prior work (Touvron et al., 2023). This problem is likely exacerbated by the instability of the KL-divergence in continuous spaces and the high variance of both value estimation and policy gradients with longer horizons in robotics tasks versus the LLM contextual-bandit setting. We find that CPL consistently outperforms %BC, indicating the CPL is indeed exhibiting policy improvement beyond the best behaviors in the dataset.
To test how CPL's supervised objectives scale to high-dimensional continuous control problems, we render the MetaWorld datasets discussed above to 64 × 64 images. We use the network architecture from DrQv2 (Yarats et al., 2022) and Bin Picking Button Press Door Open Drawer Open Plate Slide Sweep Into State 2.5k Dense PPO 83.7 ± 3.7 22.7 ± 1.9 79.3 ± 1.2 66.7 ± 8.2 51.5 ± 3.9 55.3 ± 6.0 SFT 66.9 ± 2.1 21.6 ± 1.6 63.3 ± 1.9 62.6 ± 2.4 41.6 ± 3.5 51.9 ± 2.1 P-IQL 70.6 ± 4.1 16.2 ± 5.4 69.0 ± 6.2 71.1 ± 2.3 49.6 ± 3.4 60.6 ± 3.6 CPL 80.0 ± 2.5 24.5 ± 2.1 80.0 ± 6.8 83.6 ± 1.6 61.1 ± 3.0 70.4 ± 3.0 Image 2.5k Dense SFT 74.7 ± 4.8 20.8 ± 2.4 62.9 ± 2.3 64.5 ± 7.6 44.5 ± 3.2 52.5 ± 2.5 P-IQL 83.7 ± 0.4 22.1 ± 0.8 68.0 ± 4.6 76.0 ± 4.6 51.2 ± 2.4 67.7 ± 4.4 CPL 80.0 ± 4.9 27.5 ± 4.2 73.6 ± 6.9 80.3 ± 1.4 57.3 ± 5.9 68.3 ± 4.8 State 20k Sparse PPO 68.0 ± 4.3 24.5 ± 0.8 82.8 ± 1.6 63.2 ± 6.6 60.7 ± 4.2 58.2 ± 0.6 SFT 67.0 ± 4.9 21.4 ± 2.7 63.6 ± 2.4 63.5 ± 0.9 41.9 ± 3.1 50.9 ± 3.2 P-IQL 75.0 ± 3.3 19.5 ± 1.8 79.0 ± 6.6 76.2 ± 2.8 55.5 ± 4.2 73.4 ± 4.2 CPL 83.2 ± 3.5 29.8 ± 1.8 77.9 ± 9.3 79.1 ± 5.0 56.4 ± 3.9 81.2 ± 1.6 Image 20k Sparse SFT 71.5 ± 1.9 22.3 ± 2.9 65.2 ± 2.2 67.5 ± 1.1 41.3 ± 2.8 55.8 ± 2.9 P-IQL 80.0 ± 2.3 27.2 ± 4.1 74.8 ± 5.8 80.3 ± 1.2 54.8 ± 5.8 72.5 ± 2.0 CPL 78.5 ± 3.1 31.3 ± 1.6 70.2 ± 2.1 79.5 ± 1.4 61.0 ± 4.2 72.0 ± 1.8 State % BC 10% 62.6 ± 2.6 18.9 ± 1.7 57.5 ± 3.0 61.5 ± 3.7 39.1 ± 2.5 49.3 ± 2.1 5% 64.6 ± 4.1 18.2 ± 0.6 59.8 ± 1.6 61.3 ± 1.8 38.6 ± 2.5
49.2 ± 1.9
Table 1: Success rates (in percent) of all methods across six tasks on the MetaWorld benchmark on different datasets. The leftmost column contains the observation modality (state or image), the number of segments in the dataset, and the means of labeling comparisons (dense or sparse). Dense refers to labeling every possible pairwise comparison and sparse refers to labeling only one comparison for every two segments, e.g., 10k comparisons for 20k segments. We run four seeds for state and three seeds for images. We report the maximum average performance across seeds over an 8-checkpoint, 200 episode evaluation window (details in Appendix D). Bolded values are within 1% of the top-performing method. The bottom section shows oracle performance of %BC with access to ground-truth reward. State-spaces results include PPO (delinated with a dashed line) which is not a 1-1 comparison as it uses 3.84 million extra online transitions.
the same hyper-parameters as our state-based experiments. We additionally use random shift augmentations, which drastically improve the performance of RL from images (Laskin et al., 2020).
Our image-based results can be found in rows 2 and 4 of Table 1. Interestingly, we find that performance moderately increases for SFT but substantially for P-IQL. We posit that this is because data-augmentation, which is inapplicable in state, plays a key role in improving value representation for P-IQL. Despite this, when learning from denser preference data (row 2), CPL still outperforms P-IQL in 4 of 6 environments and ties on Sweep Into. When learning from sparser comparisons (row 4), CPL and P-IQL perform comparably on most tasks, even though CPL is drastically simpler than P-IQL. Again, the gap in performance between CPL and P-IQL is higher with denser comparison data, underscoring the importance of informative negatives.
Method Params Runtime P-IQL 9.6m 16.5 hrs CPL 2.1m 10.2 hrs Table 2: Computational efficiency of each method when learning from pixels for 200k training steps on a single TitanRTX GPU.
These results are more impressive considering CPL's significant reduction in complexity. P-IQL must learn a reward function, a Q-function, a value function, and a policy. CPL avoids all of this, and only learns a policy, drastically reducing training time and parameter count. As we can see in Table 2, CPL runs 1.62× faster than P-IQL on images and has less than a quarter of the the parameters. As networks get larger and larger, the performance gain from using CPL would only increase.
As alluded to in previous sections, we find that the gap in performance between CPL and baselines is higher for datasets with denser comparisons. This is consistent with prior works in contrastive Kim et al. (2023). "PT" denotes the original Preference Transformer baseline which uses the IQL algorithm after learning a transformer-based reward model. learning (Robinson et al., 2021). To study this effect, evaluate CPL's performance as we increase the number of comparisons sampled per segment over a fixed dataset of 5000 segments. We show results of this for Drawer Open with state-based observations on the left of Fig. 2 and include the rest in Appendix C.3 in addition to dense data scaling. Overall, we find that CPL benefits from an increasing number of comparisons per segment in all tasks except Plate Slide. P-IQL is less affected, though sometimes performs worse with more comparisons, which we suspect is due to reward under-fitting. This highlights another drawback of P-IQL -due to its higher number of components, it has more hyperparameters and is consequently more sensitive to changes in the dataset. We tuned hyperparameters for all methods with 10K comparisons, then left them the same for scaling experiments.
Finally, we ablate both of CPL's hyperparameters -the temperature value α and bias regularizer λfor Drawer Open on the right of Fig. 2. While CPL generally performs well with all values, we find that higher performance could have been attained with further hyper-parameter tuning, particularly for λ. In the Appendix B we ablate more design decisions, like the choice of conservative regularizer.
To demonstrate both the applicability of the regret preference model and performance of CPL, we perform additional experiments with real-human preferences. Specifically, we adopt the benchmarks from Kim et al. (2023) which use either 100 (expert) or 500 (replay) human preferences on datasets from the D4RL benchmark (Fu et al., 2020). To faciliate learning from such a limited number of queries, we modified CPL by first learning a logistic regression model to predict the users preference P [σ + ≻ σ -] following An et al. (2023). Critically, this is not a reward or advantage function as it takes entire segments as inputs, not single state-action pairs. We then use this model to relabel the offline D4RL data with dense preferences for CPL. We borrow the architecture from Kim et al. (2023) and do not use BC pretraining. Our results can be found in Table 3. CPL has the best performance by a large margin in Hopper-Medium-Exp, but performs worse in Walker-Medium-Replay. We posit that this is because the preferences for this dataset may not closely follow the regret-based model as per discussions with the authors of Kim et al. (2023) they were collected by a single user with a pre-planned rules-based approach. Nonetheless, CPL is still able to perform well on these tasks with only 100 or 500 human preferences, and no value or Q-function learning.
Though RLHF has recently surged in popularity, learning policies from human preferences has been a long-studied problem, referred to as preference-based RL (PbRL).
PbRL methods typically start by learning a reward function, usually from pairwise comparisons, then use an RL algorithm for policy optimization (Fürnkranz et al., 2012). While Akrour et al. (2012;2011); Wilson et al. (2012) were some of the first examples of PbRL, more recently several works have shown that, provided thousands of queries or sufficient pretraining, PbRL can train deep neural-network policies for control using comparisons (Christiano et al., 2017;Lee et al., 2021;Ibarz et al., 2018;Brown et al., 2020;Hejna & Sadigh, 2022;Shin & Brown, 2021) or rankings (Brown et al., 2019;Bıyık et al., 2019;Sikchi et al., 2023a). These approaches, however, are generally demonstrated only on low-dimensional state-based control because of the challenges RL faces when scaling to larger inputs and networks (Ota et al., 2021). In the past, removing RL has lead to effective algorithms for goal-conditioned RL from images (Hejna et al., 2023;Eysenbach et al., 2022). CPL does the same but for PbRL. Other works address the problem of selecting feedback (Sadigh et al., 2017;Biyik et al., 2020;Daniel et al., 2015), which we consider complementary because CPL can benefit from higher quality data elicitation.
To scale RLHF, recent approaches for refining LLMs have ignored the temporal component of RL, and instead treated text-generation as a contextual bandits problem (Ziegler et al., 2019). While this approach has proven effective at tasks like summarization (Stiennon et al., 2020;Wu & Hu, 2018), instruction following (Ouyang et al., 2022;Nakano et al., 2021), and even image generation (Lee et al., 2023;Black et al., 2023), it fundamentally ignores the fact that interaction with users is often sequential, spanning multiple turns. Unlike these methods, CPL works with general MDPs. CPL's unique ability to learn from sequence data with only supervised objectives makes it a prime candidate for scaling to more complex problems. In fact, Direct Preference Optimization (DPO) (Rafailov et al., 2023) recently demonstrated that a supervised objective similar to CPL can work as well as RL in the contextual bandits setting. We show in Appendix A that DPO can be derived as a special case of CPL in which segments are length 1 and start at the same state. This parallels Knox et al. (2023), who show that the common contextual bandit-approach is a special case of the naïve approach from Section 3.
To derive CPL's objective, we leverage knowledge from works building on the principle of maximum entropy in control (Ziebart et al., 2008;Ziebart, 2010;Haarnoja et al., 2017). The resulting contrastive update directly learns the optimal policy with fully off-policy data. This is unlike many RL-based RLHF algorithms in both langauge (Ziegler et al., 2019) or control (Christiano et al., 2017) which require on policy rollouts and additional learned components that have been shown to increase variance (Hejna & Sadigh, 2023). Similar contrastive learning objectives have shown to be effective for temporal representation learning (Ma et al., 2023), even with preference data (Kang et al., 2023;Xu et al., 2023;An et al., 2023).
In this work we introduce CPL, a novel framework for RLHF using the regret preference model. Theoretically, we proved that CPL always learns a consistent advantage function and converges to the optimal policy for the expert's reward function. Practically, we showed that CPL's supervised objective is able to outperform RL baselines when learning complex manipulation policies from dense preference data while being simpler and 1.6× faster.
Limitations. CPL, like other RLHF approaches, assumes knowledge of the human rater's temporal discounting (i.e., of the discount factor γ), which in practice would be difficult to communicate. As CPL's loss function is computed over segments, it requires a substantial amount of GPU memory for large segment sizes. Finally, no model of human behavior is perfect.
Future Directions. Several exciting research directions remain. First is scaling CPL to larger datasets and architectures where we believe its benefits will be more pronounced. One potentially exciting application is LLMs, where CPL enables fine-tuning on multiple steps of turn-based dialogue. To our knowledge, no multi-step preferences dataset currently exists for LLMs. Second, our work only considers offline data generated by suboptimal policies. An online version of CPL could be developed that works with online human feedback, allowing policies to continually improve. Lastly, to relax the assumption that γ is known, one might instead include it in the expressivity of CPL or other RLHF approaches.
We first prove a lemma about the consistency of CPL as it is used when proving convergence.
Lemma 1. Any function A(s, a) that satisfies ∫︁ A e A(s,a)/α da = 1 ∀s ∈ S is a consistent advantage function under some reward function r in the MaxEntRL setting.
Idea. Given advantage A(s, a), we want to show that there exists a reward function r for which A is the optimal advantage function.
∫︁ A e A(s,a)/α da = 1, consider the corresponding policy π A (a|s) = e A(s,a)/α . Let the reward function be the advantage, or r(s, a) = A(s, a) = α log π A (a|s). We can determine the optimal policy π * for this reward according to Eq. ( 1):
Thus, the objective is point-wise maximized if and only if π A (•|s) = π(•|s) ∀ s ∈ S. Therefore, π A is the optimal policy for reward function r(s, a) = A(s, a).foot_0 Under this reward function, π * = π A = e A , which implies that A is a consistent advantage function.
Proposition 1. CPL learns a consistent advantage function.
Optimization via CPL fits a valid policy π subject to ∫︁ A π(a|s)da = 1 ∀s ∈ S, with corresponding MaxEnt Advantage function A(s, a) = α log π(a|s). Corollary 1. The reward function r and the reward function defined as the optimal advantage function for r, A * r , have the same optimal MaxEnt policy.
This corollary can be seen by examining the proof of Lemma 1. According to the MaxEnt RL objective for reward r the optimal policy is π * r = e A * r /α (Ziebart, 2010). Therefore A * r = α log π * r . Repeating the steps of Lemma 1 by setting r ′ = A * r = α log π * r , we get the following objective for the optimal policy π * r ′ with respect to r ′ :
Since the final expression above is minimized only when π = π * r , then π * r ′ = π * r . In other words, the reward function r and reward function r ′ = A * r have the same optimal MaxEnt policy. Implication for our P-IQL baseline. With regret-based preferences, an algorithm that learns a reward function while assuming the partial return preference model and then conducts RL on that learned reward function-including P-IQL-is actually using an approximation of A * r as the reward function. This corollary therefore implies that if that approximation is error-free, then P-IQL is using a reward function that preserves the optimal policy of the expert user's reward function r E . This application of the corollary extends the similar insight of Knox et al. (2023) to the MaxEnt RL setting. Furthermore, as shown in Lemma 1, when π = π * , E ρπ [r(s, a) -α log π(a|s)] = 0, implying that V * (s) = 0 as the reward and entropy regularization over the occupancy measure from any state is exactly the value function. Thus, as originally pointed out by (Ng et al., 1999), using A * r as the reward function results in a high amount of shaping, so much so that a horizon of one transition is sufficient to determine the optimal action in each state (Knox et al., 2023).
Theorem 1. Assume an unbounded number of preferences generated from a noisy rational regretpreference model with expert advantage function A * . CPL recovers the optimal policy π * .
Proof. Without loss of generality we let α = 1. For the purposes of this proof only, let σ k denote a segment of length k where the state-actions in the segment are denoted by σ k = (s 0 , a 0 , s 1 , a 1 , ..., s k-1 , a k-1 ). Let y be the label indicating whether the expert regret preference model prefers σ 1 k to σ 0 k , i.e., y
. Let A ˆ= log π ˆbe the implicit estimate of A * learned by CPL. For brevity, we will use the shorthand A(σ k ) = ∑︁ σ k γ t A(s t , a t ) to denote the discounted sum of advantages of a segment σ.
e A ˆ(σ 1 ) +e A ˆ(σ 0 ) ) The cross-entropy CPL loss function can be re-written as follows:
The KL divergence is optimized only when the two distributions are exactly equal. Because the preference model is rational and we assume sufficient representation power and unbounded data, it is possible for the loss to converge to zero by pointwise matching KL-divergence for each two comparisons (See Knox et al. (2022) for more information specific to the identifiability of regret based preferences). Thus, under the assumption of unbounded data, for all possible segments σ 1 k , σ 0 k we must have that
Rearranging, we get:
) where p ∈ {0, 1}. We will show that the above equality also holds for all sequences of length k -1. Consider the last action for the segment σ 1 k denoted as a 1 k-1 , then:
Now, we will use the consistency of CPL. Per Ziebart (2010), for the optimal value function A * , ∑︁ a∈A e A * (s,a) = 1, ∀s. Because CPL is consistent (Proposition 1), we also have that ∑︁ a∈A e A ˆ(s,a) = 1, ∀s. We use this, in combination with the fact that all possible dynamically feasible segments of length k -1 are a subset of dynamically feasible segments of length k to arrive at:
Inductively we have that:
Applying the same argument again, this time for σ 0 k , we have
which is equivalent to A * (s, a) = A ˆ(s, a) ∀s, a.
CPL is convex, but not strictly convex. Here we show that the CPL loss function is convex in log π.
Consider the logistic regression interpretation of CPL for finite data
where x i is the "comaprison" vector for the ith comparison in D pref . We can re-write this using matrix notation as:
The hessian of this objective (logistic regression) with respect to log π is X ⊤ DX, where D is the diagonal matrix such that
As X ⊤ DX is symmetric, it is guaranteed to be positive semi-definite making the objective function convex. The distributional constraint of CPL, that ∀s ∈ S, ∫︁ A e log π(a|s) da = 1, is also convex as e log π is convex in log π. Thus, the overall objective is convex. However, this does not imply strict convexity, or that there is a unique solution. X ⊤ DX is only positive definite if it is full rank, which is unlikely to happen in practice as usually |S × A| >> n. This means that the objective is likely not strictly convex in practice, as there can exist more than one minimizer of the objective, formally denoted π ˆ= arg min π L CPL (π, D pref ). To prove that CPL is not always strictly convex, we construct another policy π ˆ′ such that
First, we demonstrate this on a simple single-state MDP and then provide a general construction for arbitrary MDPs with discrete actions.
A simple example. Consider a single state MDP with three actions a 1 , a 2 , a 3 and expert reward function r E (s, a i ) = r i where i indexes the actions. It can be shown that, due to the single state nature of this simple MDP, the optimal maximum entropy advantage function is A * (s, a i ) = r i . Consider a preference dataset D pref consisting only of comparisons between segments (s, a 1 ) and (s, a 2 ). According to the regret preference model, the expert labels these preferences according to Bern (︁ exp r 1 /(exp r 1 + exp r 2 ) )︁ and thus we expect some labels in the preference matrix X to conflict. The finite CPL loss becomes
where c 1 and c 2 are the number of comparisons where a 1 and a 2 were preferred respectively. By taking the gradient of this objective, it can be shown that the loss is optimized only when logistic
Intuitively, this makes sense, as the logits are optimized to produce the same ratio of preferences as found in the dataset. However, when we consider the unseen action a 3 , to which we can assign arbitrary probability, the existence of multiple optimizers π ˆbecomes clear. For example, take c 1 = c 2 . By the conditions above its straightforward to see that π ˆ= [0.5, 0.5, 0.0] is an optimum of the CPL loss function. However, π ˆ= [0.1, 0.1, 0.8] achieves the same loss as its difference in log probabilities log π(a 1 |s) -log π(a 2 |s) is the same. If c 2 = 0, or we have no conflicting preferences, π ˆ= [1, 0, 0] and the implied A ˆ= log π ˆis undefined, implying some of the reward values are infinite. This means we do not have enough data to accurately fit π * . Next, we provide a construction for more general MDPs in the presence of OOD actions.
A more general construction. Let the expert reward function r E to be bounded. We will interpret the finite preference dataset as a matrix X as described in Section 3.2 and use N (X) to denote the null space of X. For a vector u ∈ R |S×A| we use u(s, a) to index the vector at state s and action a.
Assume the following about X:
1. There exists a vector u ∈ N (X) such that for state action s, a contained in X, u(s, a) ̸ = 0.
In other words, the null space is non-trival on the support of D pref .
2. For every state in the dataset where there is an action such that u(s, a) ̸ = 0, there exists at least one out-of-distribution (OOD) action a OOD not in the dataset. The indicator vector for s, a OOD is thus a basis vector for N (X).
Let π ˆbe the minima of the CPL loss function. We will construct π ˆ′ as follows. Select a vector u ∈ N (X) that is non-zero for at least one s, a pair in X. As u ∈ N (X), we have that L CPL (π ˆ, D pref ) = L CPL (e log π ˆ+u , D pref ). However, e log π ˆ+u violates the policy constraint as it may not integrate to one. We can fix this problem by adding or removing probability mass from the OOD actions we have assumed exist at states where u is non-zero. We do this by constructing another vector v ∈ N (X) by choosing one a OOD at each state without loss of generality. By examining the total sum of probabilities of the modified policy, ∀s ∈ X, ∑︂ As v is formed from a linear combination of basis vectors of N (X), v ∈ N (X). Consequently, L CPL (π ˆ, D pref ) = L CPL (e log π ˆ+u+v , D pref ) and by the above construction π ˆ′ = π ˆeu+v is a valid policy. This completes the construction.
We have shown that an infinite number of policies can attain the same optima, just by shifting the amount of probability assigned to OOD actions. For some of these solutions, the entire mode of the policy is potentially out-of-distribution. In the offline setting, the pessimism principle dictates that we should discourage modes that are out-of-distribution. We fix this by introducing regularization.
CPL loss translates a relative weighting between preferences to a policy, but does not employ any mechanism to ensure the learned policy is close to the dataset. In the offline setting, this can be detrimental if the learned policy incorrectly extrapolates to out of distribution actions. A similar approach, under the name of pessimism or conservatism, is commonly seen in offline RL literature (Levine et al., 2020;Jin et al., 2021;Sikchi et al., 2023b). As expalined in Section 3.2, we want to learn policies that have a high-probability on the dataset. However, there are many datasets that potentially have the the same loss, as L CPL depends only on the difference in probability for each preference comparison, or ∑︁ σ + γ t log π(a t |s t ) -∑︁ σ -γ t log π(a t |s t ), and thus constants added to the log probabilities of each segment cancel. However, we would prefer that a higher loss is given when the policy is assigns lower probability to actions in the dataset.
To remedy this, we introduced bias regularizer λ ∈ (0, 1) in Section 3, which leads to the modified preference loss:
]︃ .
Next, we prove that this loss discourages the policy from learning modes that are out-of-distribution, starting with the proposition from the main text.
Proposition 2. Consider a comparison σ + ≻ σ -from D pref and an arbitrary comparison
Succinctly, this proposition states that if preference comparisons each achieve the same loss, the less likely comparisons under the policy (in this case σ ′+ ≻ σ ′-), will have higher regularized CPL loss.
Essentially, this shows that the regularized objective encourages the policy to have higher likelihood on the provided comparisons than any other potential comparison that exists. Proof. By the stated assumptions, it must be that ∑︁ σ ′+ γ t log π(a t |s t ) + δ = ∑︁ σ + γ t log π(a t |s t ) for some δ > 0. As the two comparisons also have the same CPL Loss, their logits must be the same, or
Consequently, the same δ must hold for the negative segments, or ∑︁ σ ′-γ t log π(a t |s t ) + δ = ∑︁ σ -γ t log π(a t |s t ). We can then examine the regularized CPL loss under each comparison. First, we evaluate the finite regularized loss for σ + ≻ σ -, algebraically simplified for clarity:
log π(a t |s t ) exp ∑︁ σ + γ t α log π(a t |s t ) + exp λ ∑︁ σ -γ t α log π(a t |s t ) = -log logistic (︄ ∑︂ σ + γ t log π(a t |s t ) -λ ∑︂ σ - γ t log π(a t |s t ) )︄ = log (︄ 1 + exp (︄ λ ∑︂ σ - γ t log π(a t |s t ) -∑︂ σ + γ t log π(a t |s t )
)︄)︄
We can then compare this to the regularized loss for σ ′+ ≻ σ ′-.
The key step in the above is substituting the relationship between the log probabilities of the comparisons. As δ > 0 and 0 < λ < 1, it can easily be seen that the loss is lower for σ + ≻ σ -, letting us conclude that
We can extend this proposition to the regularized CPL loss over entire datasets as follows:
For a fixed policy π, consider two preference datasets
for all i and strictly for at least one i,
The proof of this amounts to first noticing that, because the preference losses are the same for every ordered subset, the losses for the ith datapoints in D ′ n and D n must be the same. Then, we can repeatedly apply Proposition 2. Since the inequality is strict at at-least one datapoint, the regularized loss will be strictly lower.
We can construct datasets for which this is applicable. For example, consider a dataset D containing a total ordering over n segments, σ 1 ⪰ σ 2 ⪰ ... ⪰ σ n . The unregularized loss for this policy and dataset is L CPL (π, D). We can construct another dataset D ′ over a different set of totally ordered segments from anywhere in the state space σ ′1 ⪰ σ ′2 ⪰ .. ⪰ σ ′n such that:
for all i = 1, 2, ..., n and some δ ≥ 0.
We can derive a version of CPL for ranking data using a Plackett-Luce model (Plackett, 1975). We denote the chosen ranking as a permutation τ :
where K is the number of segments presented, σ 1 , ..., σ K . The Plackett-Luce model under regret based preferences is:
This model generalizes to Bradley-Terry (Bradley & Terry, 1952) when K = 2. To learn the optimal policy, we maximize the log likelihood of the above and make the same substitution as CPL, α log π * (a|s) = A * (s, a). This gives us the CPL loss function for rankings, which can be seen as a version of the InfoNCE objective. Without loss of generality, we order the permutations τ such that
]︄
Except for the sum over k, this is the exact objective from Oord et al. (2018) where the scores are the discounted sum of log probabilities over the segments.
Reduction via Maximum Entropy Advantage. Note that by the Bellman equation,
DPO (Rafailov et al., 2023) assumes the contextual-bandits setting, thus the MDP terminates after a single step and there is no next state s ′ . As we can see from the above, in this setting, A * (s, a) = r E (s, a) -V * (s). DPO also assumes that all preferences start from the same state s, and thus only actions a + and a -differ. This is consistent with RLHF on LLMs as humans score "responses" to fixed prompts.
The regret preference model becomes:
which is the same preference model used in DPO. From here the same conservative derivation as DPO can be applied by noting that, for KL-constrained contextual bandits, π * (a|s) = µ(a|s)e Q * (s,a)-V * (s) = µ(a|s)e r E (s,a)-V * (s) for reference distribution µ. Solving for r E , we can perform a substitution just like in CPL to arrive at the DPO objective.
CPL under Constrained Regret Preferences. We can also consider a setting where users provide preferences constrained to a reference distribution µ. This might arise in scenarios where users are only shown a fixed set of behaviors, and do not extrapolate far beyond them. Though we do not believe this premise has previously been considered, it leads to an interesting result. Assume preferences to be distributed according to the KL constrained advantage function. In this setting, π * (a|s) = µ(a|s)e A * (s,a) and by substitution the CPL loss becomes
which is essentially a multi-step generalization of DPO which has not previously been considered. In the next section, we expand on this as a variant of CPL.
In the main body of the paper, we presented the version of CPL which we found to consistently attain good performance. In some of our experiments, we also considered two other variants of CPL. We detail these below.
BC-Regularized CPL. Instead of using our biased conservative regularization from An et al. ( 2023), we consider using a simple BC regularizer. This can be derived by considering the objective: We refer to this method as CPL (BC).
KL Constrained CPL. We can also consider the setting where preferences are assumed to be distributed according to the constrained advantage function. Though in practice we sample preferences according to the maximum entropy advantage function, we found this approach to still work in many settings. First, we learn the reference distribution µ using behavior cloning. Then, we use constrained CPL with bias regularization, making the final loss function:
We refer to this method as CPL (KL).
CPL with Dense Preferences. When learning from "dense" preference data, it is possible to augment the batch to include more comparisons using the transitive property. Specifically, given a batch of b segments, we compute all possible pairwise comparisons within a batch:
This provides as much contrastive signal per-batch as possible. We applied this technique to our CPL experiments with images, and found that it lead to a slight increase in performance for some tasks.
In this section we provide our full experimental results:
1. Learning curves from state for CPL, baselines, and variants described in Appendix B.
2. Learning curves from images for CPL and baselines.
3. Scaling results for CPL and P-IQL with different sized dense datasets and fixed sparse datasets with a varying number of comparisons.
4. Results when varying the number of comparisons for a fixed dataset.
5. More Hyper-parameter ablations, include temperature and pretraining. 100000 200000 300000 400000 500000 0.0 0.2 0.4 0.6 0.8 Success Rate Bin Picking 100000 200000 300000 400000 500000 0.0 0.1 0.2 0.3 0.4 Button Press 100000 200000 300000 400000 500000 0.5 0.6 0.7 0.8 0.9 Door Open 100000 200000 300000 400000 500000 Training Steps 0.5 0.6 0.7 0.8 0.9 Success Rate Drawer Open 100000 200000 300000 400000 500000 Training Steps 0.3 0.4 0.5 0.6 Plate Slide 100000 200000 300000 400000 500000 Training Steps 0.4 0.5 0.6 0.7 0.8 Sweep Into MetaWorld State 20K Sparse SFT P-IQL PPO CPL (BC) CPL (KL) CPL We find that for these results, P-IQL's performance sometimes goes down. Inspecting our training logs reveals that this is likely due to the reward function underfitting. For example, on Door Open, the reward modeling loss is near zero with only 2 comparisons per segment (10K comparisons total, which is the amount we tuned our hyper-parameters for). With 16 comparisons per segment, the loss ends near 0.16. This highlights an additional limitation of P-IQL: it requires tuning an addition reward model, which can be very sensitive to its training parameters. CPL removes this additional complexity, and is thus easier to scale.
Note that we assume that all states are reachable and therefore have support in ρ t π (s) for any optimal MaxEnt policy.