Parameterized policy representation is an important component for policy-based deep reinforcement learning (DRL) (Sutton & Barto, 2018;Arulkumaran et al., 2017;Dong et al., 2020). Prior works have developed a variety of policy parameterization methods. For discrete action space, existing policy parameterization includes Softmax action preferences (Sutton & Barto, 2018), Gumbel-Softmax for categorical distributions (Jang et al., 2016), decision trees (Frosst & Hinton, 2017;Ding et al., 2020), etc. For continuous action space, the most typical choice is unimodal Gaussian distribution. However, in practice the demonstration dataset often encompasses samples from a mixture of behavior policies. To capture the multi-modality in data distribution, Gaussian mixture model (GMM) (Jacobs et al., 1991;Ren et al., 2021), variational auto-encoders (VAE) (Kingma & Welling, 2013;Kumar et al., 2019), denoising diffusion probabilistic model (DDPM) (Ho et al., 2020;Song et al., 2020;Wang et al., 2022;Chi et al., 2023;Hansen-Estruch et al., 2023;Venkatraman et al., 2023) are adopted as policy representation.
The desiderata for policy representation in DRL includes: 1. The strong expressiveness of the function class is found to be critical for modeling multi-modal data distribution in offline RL (Wang et al., 2022) or imitation learning (IL) (Chi et al., 2023); 2. Differentiability of the model is usually required for ease of optimization with stochastic gradient descent; 3. Computational and time efficiency for sampling can be essential for RL agents learning from interactions with environments. Previous works with action diffusion models (i.e., diffusion policy) testify the expressiveness of diffusion models for multi-modal action distributions (Wang et al., 2022;Chi et al., 2023;Hansen-Estruch et al., 2023;Janner et al., 2022;Ajay et al., 2022). Although GMM and VAE also capture multi-modality, diffusion models with large sampling steps are found to be more expressive for IL and offline RL scenarios (Wang et al., 2022;Chi et al., 2023). However, it is known that the diffusion model with progressive denoising over a large number of steps can lead to slow sampling speed. The action inference can be a critical bottleneck for online RL heavily depends on sampling from environments. A direct usage of diffusion policies for online settings with policy gradient for optimization requires backpropagating through the diffusion networks for the number of sampling steps, which is not scalable for its large time consumption and memory occupancy. Consistency models (Song et al., 2023) based on probability flow ordinary differential equation (ODE) is proposed as a rescue with comparable performances as diffusion models but much less computational time, which allows few-step generation process thus significantly reduce the time consumption at inference stage.
Offline-to-online RL usually suffers from a catastrophic degraded performance at initial online training stage, due to the distribution shift of training samples. Previous research has studied online fine-tuning with offline data or pre-trained policies, including Hybrid Q learning (Song et al., 2022), RLPD (Ball et al., 2023), Cal-QL (Nakamoto et al., 2023), Action-free Guide (Zhu et al., 2023), Actor-Critic Alignment (ACA) (Yu & Zhang, 2023) and Lee et al. (2022).
Score-based Generative Model for RL. For policy representation in RL, recent work also uses Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020;Song et al., 2020), which we loosely refer to as the diffusion model (original diffusion model traces back to Sohl-Dickstein et al. (2015)) in this paper, to capture the multi-modal distributions in offline dataset. Diffusion QL (Wang et al., 2022) uses diffusion model for policy representation in the Q-learning+BC approach. Implicit Diffusion Q-learning (IDQL) (Hansen-Estruch et al., 2023) is a variant of IQL using diffusion policy. Diffusion policies (Chi et al., 2023) applies diffusion models for policy representation under imitation learning settings in robotics domain. Diffuser (Janner et al., 2022) and Decision Diffuser (Ajay et al., 2022) combines decision transformer architecture with diffusion models for model-based reinforcement learning from offline dataset. Diffusion policies are also used for goal-conditioned imitation learning (Reuss et al., 2023) and human behavior imitation (Pearce et al., 2023). Q-guided Policy Optimization (QGPO) (Lu et al., 2023) proposes a new formulation for intermediate guidance in diffusion sampling process. Latent Diffusion-Constrained Q-Learning (LDCQ) (Venkatraman et al., 2023) proposes to apply latent diffusion model with a batch-constrained Q value to handle the stitching issue and the extrapolation errors for offline dataset.
For RL, we define a Markov decision process (S,A,R,T ,ρ 0 ,γ), where S is the state space, A is the action space, R(s,a) : S ×A → R is the reward function, T (s ′ |s,a) : S ×A → Pr(S) is the stochastic transition function, ρ 0 (s 0 ) : S → Pr(S) is the initial state distribution, and γ ∈ [0,1] is the discount factor for value estimation. A stochastic policy π(a|s) : S → Pr(A) determines the action a ∈ A for the agent to take given its current state s ∈ S, and the optimization objective for the policy is its discounted cumulative reward: E π [ ∞ t=0 γ t r(s t ,a t )]. For offline RL, there exist a dataset D = {(s,a,r,s ′ ,done)} collected with some behavior policies π b , and the current policy π is set to be optimized with D. For online RL, the agent is allowed collect samples through interacting with the environment to compose an online training dataset D for optimizing its policy. We consider parameterized policy representation as π θ .
The diffusion model (Ho et al., 2020;Song et al., 2020) solves the multi-modal distribution matching problem with a stochastic differential equation (SDE), while the consistency model (Song et al., 2023) solves an equivalent probability flow ordinary differential equation (ODE): dxτ dτ = -τ ∇logp τ (x) with p τ (x) = p data (x) ⊗ N (0,τ 2 I) for time period τ ∈ [0,T ], where p data (x) is the data distribution. The reverse process along the solution trajectory {x τ } τ ∈[ϵ,T ] of this ODE is the data generation process from initial random samples xT ∼ N (0,T 2 I), with ϵ as a small constant close to 0 for handling numerical problem at the boundary. For speeding up the sampling process from a diffusion model, consistency model shrinks the required number of sampling steps to a much smaller value than the diffusion model, without hurting the model generation performance much. Specifically, it approximates a parameterized consistency function f θ : (x τ ,τ ) → x ϵ , which is defined as a map from the noisy sample x τ at step τ back to the original sample x ϵ , instead of applying a step-by-step denoising function p θ (x τ -1 |x τ ) as the reverse diffusion process in diffusion model. The training and inference details of consistency model refer to Appendix B. For modeling the conditional distribution with condition variable c, the consistency function is changed to be f θ (c,x τ ,τ ), which is sightly different from original consistency model.
The consistency model as policy representation in RL can be formulated in the following way. To map the consistency model to a policy in MDP, we set:
where p D (a|s) is the action-state conditional distribution from offline dataset D.
Consistency Action Inference. By setting the condition variable c as state s and generated variable x as action a, the consistency function f θ can be used for generating actions from states following the conditional distribution of the dataset, i.e., a behavior RL policy. The parameterized policy π θ is defined implicitly in terms of f θ , with which an action a conditioned on state s can be generated following the Consistency Inference as Alg. 1 with predetermined {τ n |n ∈ [N ]} sequence.
During the inference process, a trained consistency model f θ (s,â τn ,τ n ) iteratively predicts denoised samples from the noisy inputs âτn = a + τ 2 n -ϵ 2 z along the probability flow ODE trajectory at step n ∈ [N ], with Gaussian noise z ∼ N (0,I). {τ n |n ∈ [N ]} is a sub-sequence of time points on a certain time period [ϵ,T ] with τ 1 = ϵ,τ N = T . For inference, the sub-sequence is a linspace of [ϵ,T ] with (N -1) sub-intervals. A single-step version of Consistency Inference can be achieved by just set {τ n |n = 0,1} = {ϵ,T }. Notice that T here is the time horizon for denoising process in the consistency model instead of the episode length of the sample trajectory.
Consistency Behavior Cloning. With the offline dataset D, the conditional consistency model as policy can be trained with loss by adapting the original (Song et al., 2023):
where λ(
•) is a step-dependent weight function, a τn = a+τ n z and d(•,•) is the distance metric. f θ ⊺ is exponential moving average of f θ for stabilizing the target estimation in training. In classical actor-critic algorithm, there exists the same delayed update of the policy network π θ ⊺ (i.e., f θ ⊺ ) for estimating target Q-values, which is set to coincide with the target in estimating the consistency loss. The setting for τ n is detailed in Appendix B. Pseudo-code of Consistency BC refers to Alg. 2. Algorithm 1 Consistency Action Inference Input s,f θ ,N,{τ n
â τn ,τ n ) end for return a Algorithm 2 Consistency Behavior Cloning Input offline dataset D Initialize consistency policy π θ , target θ ⊺ ← θ for iterations k = 1,...,K do Update policy π θ (with model f θ ) using loss L c (θ) as Eq. 2; Update target: θ ⊺ ← αθ ⊺ +(1-α)θ end for Algorithm 3 Offline Consistency Actor-Critic Input offline dataset D Initialize consistency policy network π θ , critic networks Q ϕ1 ,Q ϕ2 Initialize target network parameters: θ ⊺ ← θ, ϕ ⊺ 1 ← ϕ 1 ,ϕ ⊺ 2 ← ϕ 2 for policy training iterations k = 1,...,K do Sample minibatch B = {(s,a,r,s ′ )} ⊆ D; % Q-value Update Update Q ϕ1 ,Q ϕ2 with Eq. 3; % Policy Update Update policy π θ (with model f θ ) via Eq. 4; % Target Update Update target:
Consistency Actor-Critic. As as estimation of the state-action value of current policy, the parameterized Q ϕ (s,a) function can be learned with the double Q-learning loss (Fujimoto et al., 2018) with batched data B ⊆ D:
for stabilizing training. The regularized policy π θ on offline dataset is learned with a combination of policy gradient through maximizing the expected Q ϕ (s,a) function and a behavior cloning regularization with consistency loss L c (θ):
where L q (θ) = -E s∼B,a∼π θ (s) Q ϕ (s,a)
(5) where a ∼ π θ (s) is action inference from the consistency policy as Alg.1. It can be noticed that the actions generated with N denoising steps will produce the policy gradients through the Q ϕ (s,a) in above equation, thus it also backpropagates through f θ for N times in the gradient descent procedure, which can lead to additional time consumption apart from the multi-step model inference. Therefore, reducing the denoising steps N can be critical for the speed of this type of models as RL policies. The consistency actor-critic (Consistency-AC) algorithm is provided in pseudo-code Alg. 3.
Loss Scaling. The consistency loss as Eq. 2 matches the denoised predictions from two consecutive timesteps τ n and τ n+1 . Due to the usage of N (k) schedule (detailed in Appendix B), their difference |τ n+1 -τ n | decreases as the training iteration k increases (thus N (k) also increases), which allows the consistency model to have a coarse-to-fine matching process across different time scales. This also leads to a decreasing loss value L(θ;k) as k increases from 1 to K since the predictions from smaller time intervals are easier to match. Actually, the loss L c (θ) changes drastically across several magnitudes within an epoch, which leads to severe imbalance with the second loss term L q (θ) in Eq. 4. The coefficient η is a constant hyperparameter independent of k, so it cannot help to alleviate this issue. Although original consistency model applies λ(τ n ) ≡ 1 for image generation, we empirically find that in offline RL this imbalance of two loss terms can hurt the effect of policy regularization in some tasks, as evidenced by ablation studies in Sec. 5.2. To solve this issue, we propose a k-dependent weighting mechanism to balance the values of two loss terms. This is found to improve the performances of this policy regularization method with consistency model on offline RL. Specifically, λ(•) in Eq. 2 is chosen to be:
where ξ is set according to tasks (or absorbed in η). The denominator captures the loss scale at iteration k conveniently.
To evaluate the expressiveness and computational efficiency of the proposed consistency policy and corresponding algorithms, we conduct experiments on four task suites (Gym, AntMaze, Adroit, Kitchen) in D4RL benchmarks under three canonical RL settings: offline (Sec. 5.1, Sec. 5.2), offline-to-online and online (Sec. 5.3). It is known that the D4RL offline dataset can exhibit multi-modality since the samples may be collected with a mixture of polices or along various sub-optimal trajectories, which makes the expressiveness of policy representation critical (Fu et al., 2020;Wang et al., 2022). For offline RL, the generative models as policies are evaluated with both behavior cloning (Consistency-BC, Diffusion-BC) and actor-critic type (Consistency-AC, Diffusion-QL) algorithms, in terms of both performances and computational time. The Diffusion-QL is also an actor-critic algorithm though with name QL. Variants of Consistency-AC are compared as ablation studies and the best performances are reported. For offlineto-online and online RL settings, the learning curves and final results are compared for different methods. For evaluation, each model is evaluated over 10 episodes for Gym tasks and 100 episodes for other tasks, following the settings in previous work (Wang et al., 2022). By default, the consistency policy applies the number of denoising steps N = 2 with a saturated performances on most of D4RL tasks, while diffusion policy uses N = 5 (Wang et al., 2022). Effects of different choices of N are discussed in Sec. 5.2.
Empirical finding 1: By behavior cloning alone (without any RL component), using an expressive policy representation with multi-modality like the consistency or diffusion model achieves performances comparable to many existing popular offline RL methods. Learning consistency policy requires much less computation than learning diffusion policy.
The proposed method Consistency-BC follows Alg. 2 with consistency policy for behavior cloning, and Diffusion-BC is by replacing the policy representation with a diffusion model and replace the policy loss with the diffusion model training loss L d (θ) as specified in paper (Wang et al., 2022). Results for classic BC with Gaussian policies and previous offline RL baselines, including AWAC (Nair et al., 2020), Diffuser (Janner et al., 2022), MoRel (Kidambi et al., 2020), Onestep RL (Brandfonbrener et al., 2021), TD3+BC (Fujimoto & Gu, 2021), Decision Transformer (DT) (Chen et al., 2021), BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019) and REM (Agarwal et al., 2020), are adopted from previous paper (Wang et al., 2022). SAC (Haarnoja et al., 2018) is the Results from Tab. 1 show the advantage of using multi-modal policy representation for offline RL even only with the BC method. For reference purpose, the values in the brackets allow for online evaluation to achieve the best model selection from the set of trained models, which serve as the maximal possible values for the standard offline selection without leveraging online evaluation. Compared with vanilla BC using the Gaussian distribution for policies, Consistency-BC with multi-modality outperforms it on 14/20 tasks, and Diffusion-BC has better or equivalent performance as BC for 20/20 tasks. Through leveraging multi-modal representation in BC, the improvement of normalized scores averaged over tasks is significant, and this is mainly caused by the multi-modality within the offline dataset by mixing over policies. Moreover, compared with previous offline RL baselines, which do not just apply BC, the Consistency-BC and Diffusion-BC show comparable performances, and even superior performances for tasks like walker2d-medium-v2, hopper-medium-replay-v2, walker2dmedium-replay-v2, walker2d-medium-expert-v2 in Gym tasks, most of AntMaze, Adroit and Kitchen tasks. The consistency policy is slightly less expressive than the diffusion policy, which is within our expectation due to its heavy reduction on the sampling steps. However, the consistency policy shows higher computational efficiency than diffusion policy as compared in Fig. 1, with an average reduction of 42.97% computational time across 20 tasks. Detailed computational time for each task is provided in Appendix C.2 Tab. 6.
Table 2: The performance of Consistency-AC and SOTA baselines on D4RL Gym, AntMaze, Adroit and Kitchen tasks for offline RL setting. For Consistency-AC and Diffusion-QL, each cell has two values: one for offline model selection and another (in brackets) for online model selection. The bold values are the highest among each row. Tasks CQL IQL X -QL ARQ IDQL-A Diffusion-QL Consistency-AC halfcheetah-m 44.0 47.4 48.3 45 ± 0.3 51.0 51.1 ± 0.5 (51.5 ± 0.3) 69.1 ± 0.7 (71.9 ± 0.8) hopper-m 58.5 66.3 74.2 61 ± 0.4 65.4 90.5 ± 4.6 (96.6 ± 3.4) 80.7 ±10.5 (99.7 ±2.3) walker2d-m 72.5 78.3 84.2 81 ± 0.7 82.5 87.0 ± 0.9 (87.3 ± 0.5) 83.1 ±0.3 (84.1 ±0.3) halfcheetah-mr 45.5 44.2 45.2 42 ± 0.3 45.9 47.8 ± 0.3 (48.3 ± 0.2) 58.7 ±3.9 (62.7 ±0.6) hopper-mr 95.0 94.7 100.7 81 ± 24.2 92.1 101.3 ± 0.6 (102.0 ± 0.4) 99.7 ± 0.5 (100.4 ± 0.6) walker2d-mr 77.2 73.9 82.2 66 ± 7.0 85.1 95.5 ± 1.5 (98.0 ± 0.5) 79.5 ± 3.6 (83.0 ± 1.5) halfcheetah-me 91.6 86.7 94.2 91 ± 0.7 95.9 96.8 ± 0.3 (97.2 ± 0.4) 84.3 ± 4.1 (89.2 ± 3.3) hopper-me 105.4 91.5 111.2 110 ± 0.9 108.6 111.1 ± 1.3 (112.3 ± 0.8) 100.4 ± 3.5 (106.0 ± 1.3) walker2d-me 108.8 109.6 112.7 109 ± 0.5 112.7 110.1 ± 0.3 (111.2 ± 0.9) 110.4 ± 0.7 (111.6 ± 0.7) Average 77.6 77.0 83.7 76.2 82.1 87.9 (89.3) 85.1 (89.8) antmaze-u 74.0 87.5 93.8 97 ± 0.8 94.0 93.4 ± 3.4 (96.0 ± 3.3) 75.8 ± 1.6 (85.6 ± 3.9) antmaze-ud 84.0 62.2 82.0 62 ± 12.1 80.2 66.2 ± 8.6 (84.0 ± 10.1) 77.6 ± 6.3 (82.4 ± 3.4) antmaze-mp 61.2 71.2 76.0 80 ± 8.3 84.5 76.6 ± 10.8 (79.8 ± 8.7) 56.8 ± 30.1 (71.6 ± 14.5) Average 73.1 73.6 83.9 79.7 86.2 78.7 (86.6) 70.1 (79.9) pen-human-v1 35.2 71.5 -45 ± 5.2 (v0) -72.8 ± 9.6 (75.7 ± 9.0) 63.4 ± 7.7 (67.9 ± 5.3) pen-cloned-v1 27.2 37.3 -50 ± 7.1 (v0) -57.3 ± 11.9 (60.8 ± 11.8) 50.1 ± 2.2 (53.7 ± 3.4) Average 31.2 54.4 -47.5 -65.1 (68.3) 56.8 (60.8) kitchen-c 43.8 62.5 82.4 37 ± 14.2 -84.0 ± 7.4 (84.5 ± 6.1) 51.9 ± 6.0 (67.6 ± 2.7) kitchen-p 49.8 46.3 73.7 50 ± 5.0 -60.5 ± 6.9 (63.7 ± 5.2) 38.2 ± 1.8 (39.8 ± 1.6) kitchen-m 51.0 51.0 62.5 39 ± 9.4 -62.6 ± 5.1 (66.6 ± 3.3) 45.8 ± 1.5 (46.7 ± 0.9) Average 48.2 53.3 72.9 42.0 -69.0 (71.6) 45.3 (51.4) Total Average 66.2 69.6 -67.4 -80.3 (83.2) 72.1 (77.9)
Empirical finding 2: Replacing diffusion model with consistency model in TD3-BC type algorithm for offline RL will lead to speed up of model training and inference, with slightly worse performances while still outperforming some other baselines.
For offline RL, the proposed method Consistency-AC follows Alg. 3 with consistency model for policy representation, and the consistency policy is embedded in an actor-critic algorithm with BC policy regularization to avoid generating out-of-distribution actions. Results for previous baselines, including CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2021), X -QL (Garg et al., 2023), ARQ (Goo & Niekum, 2022), IDQL-A (Hansen-Estruch et al., 2023) and Diffusion-QL (Wang et al., 2022) are adopted from results reported in corresponding papers. Detailed hyperparameters for Consistency-AC and Diffusion-QL refer to Appendix C.1. Results from Tab. 2 show the average normalized scores of different methods across five random seeds, with standard deviations reported for Diffusion-QL and Consistency-AC. The results in Tab. 2 are directly comparable with the results in Tab. 1 since they follow the same offline RL setting. Tab. 2 shows that although Consistency-AC achieves a slightly lower average score (72.1) than Diffusion-QL (80.3), it outperforms the other baselines in most of the tasks, like Gym and Adroit.
The AntMaze tasks are found to be hard for the Consistency-AC method, we conjecture that this is potentially caused by the sparse reward signals (as evidenced by Appendix A Fig. 5) in dataset, which makes the difficulty of Q-learning become more of a bottleneck than modeling the multi-modal distributions with behavior cloning. The conservativeness of the Q-value estimation might be important but orthogonal to the proposed consistency policy. Considering the reduction of denoising steps in the training and inference stages of Consistency-AC, it can be regarded as a trade-off between the computational efficiency and the approximation accuracy of multi-modal distribution, which will be discussed as following.
Method N Training Time (s per epoch) Inference Time (ms per sample) Avg. Norm Score Diffusion-QL 50 206.44±16.70 30.65±2.10 -20 108.65±2.85 13.04±0.90 109.2±1.1 (111.1±1.9) 10 76.54±10.74 6.87±0.55 108.6±0.6 (112.5±0.2) 5 57.06±19.16 3.76±0.29 108.2±5.6 (112.3±0.2) 2 31.59±10.32 1.96±0.10 53.6±16.6 (103.5±10.0) 1 30.23±8.75 1.37±0.09 2.8±1.5 (13.1±12.5) Consistency-AC 50 150.84±31.02 26.50±1.92 -20 76.22±9.92 11.12±0.77 101.3±6.3 (107.3±0.2) 10 54.04±4.43 5.95±0.44 98.4±4.3 (107.1±4.0) 5 40.79±2.79 3.39±0.29 101.4±4.7 (110.1±1.6) 2 31.94±1.55 1.84±0.21 102.4±3.0 (106.2±1.6) 1 28.51±1.78 1.23±0.11 6.2±5.4 (19.1±9.3)
Table 3: Comparison of computational time for two methods with different denoising steps N on the task hopper-medium-expert-v2. The gray lines apply default N values for two models. Saturates with smaller N (=2) Larger time gap while yielding a slightly better performance (1.057×). The "-" in the table with N = 50 indicates a missing value of the average score due to exceeding the limited time (72 hours) for the job. Moreover, as shown in Fig. 2, Consistency-AC has better scaling laws than Diffusion-QL for both training and inference in time consumption with increasing N , which is further testified by the linear fitting results in Appendix. C.2 Fig. 6. Ablation Studies. Hansen-Estruch et al. (2023) proposes to use residual networks with layer normalization for network parameterization in diffusion policy, namely LN-Resnet, which is also tested for consistency policy in our experiments. As an ablation study, we compare different variants of Consistency-AC for offline RL setting, including (1) Consistency-BC by setting η = 0 and without using loss scaling (λ(τ n ) ≡ 1 in Eq. 2); (2) only without loss scaling; (3) the standard setting with multi-layer perceptrons (MLP) networks for the parameterization of f θ ; (4) the LN-Resnet parameterization of f θ . These variants can be regarded as various hyperparameters or training settings for the proposed Consistency-AC algorithm, and the reported results in Tab. 2 are the best choices among these variants. The comparison results for four variants across four task domains are summarized in Fig. 3. Detailed results for this ablation study are shown in Appendix C.3. We find that LN-Resnet does not consistently improve over MLP across tasks for the consistency model but benefits mainly for the Adroit tasks.
Without loss scaling, the performance degrades significantly (by 37.8% on average) for most tasks, although for some specific tasks (e.g., AntMaze) it may improve the performance a bit without loss scaling. For most tasks except for AntMaze, Consistency-BC cannot achieve the best performances and the Q-learning loss L q (θ) (as Eq. 5) with proper scaling helps to further improve the scores.
Figure 3: Comparison of variants of Consistency-AC across tasks in offline RL setting. 5.3 OFFLINE-TO-ONLINE AND ONLINE RL Empirical finding 3: Consistency policy has a close but slightly worse performance than diffusion policy for offline-to-online RL, but a significant improvement of computational efficiency.
For online RL, we consider both online learning from scratch and the offline-to-online setting with the model trained on offline dataset as an initialization for online fine-tuning. As discussed in previous Sec. 5.2, the offline model can be selected in either an online or offline manner, respectively by model evaluation with or without online experience. Both types of models are used for initializing the policy and value models at the beginning of online fine-tuning. For online fine-tuning, it follows the standard actor-critic algorithm, that the Q value is updated with Eq. 3 using online data, and the policy is updated with the Q-learning loss L q (θ) only as Eq. 5. The algorithms use ϵ-greedy for exploration with a decaying schedule. Pseudo-codes for offline-to-online and online Consistency-AC are provided in Appendix D.1. Hyperparameters for training refer to Appendix D.2. Tab. 4 summarizes the quantitative results for average scores achieved with Consistency-AC and Diffusion-QL across five random seeds for two settings over 9 Gym tasks, as well as offline-to-online baseline methods SAC, AWAC and ACA (Yu & Zhang, 2023). Both the Consistency-AC and Diffusion-QL are pre-trained on the offline dataset for 2000 epochs. Each model is trained for one million steps for online fine-tuning or learning from scratch. The results for SAC, AWAC and ACA are adopted from the paper (Yu & Zhang, 2023) with each model fine-tuned for 100k online steps. Each cell has two values: one for offline model selection as initialization and another (in brackets) for online model selection as initialization. The normalized scores are slightly lower for Consistency-AC compared with Diffusion-QL in offline-to-online settings, but higher in online RL from scratch. On average, the two methods achieve lower values in online setting than the offline-to-online setting, which testifies the improvement of learning efficiency by initializing with pre-trained generative policy models. However, since the training is set to have a fixed overall timesteps and using the same learning rate 1 × 10 -5 , the purely online models do not converge to its optimal performances yet. The learning rate is chosen for the fine-tuning setting, and the purpose is not to show online RL can achieve scores higher than 100 with sufficient training but to compare with the offline-to-online setting, for demonstrating the effectiveness of initialized models with offline pre-training.
Empirical finding 4: Consistency policy could outperform diffusion policy for online RL setting mostly in computational efficiency and sometimes in sample efficiency, especially for hard tasks. For most tasks, the consistency policy has comparable performances with the diffusion policy and a significantly shorter time to finish the entire online training. The offline-to-online methods are usually more sample efficient than the online methods except for three hopper tasks, which are relatively easy to learn a good policy. For the online setting, the consistency policies demonstrate significantly more efficient learning than the diffusion policies, especially for more complex tasks like halfcheetah. Consistency policies show a sharper score-increasing slope for 8/9 tasks than the diffusion policies. Our conjecture is that the expressiveness of a model is more essential in offline setting than online setting. For a deterministic optimal policy in MDP, overly expressive policy models like diffusion may hinder the convergence in online setting by being too explorative. For offline-to-online setting, this advantage is less obvious presumably due to the lower initial performances of consistency policies from the offline pre-training. We refer to Appendix D.3 Tab. 9 and Fig. 7 for more analysis of the training time for two methods.
The consistency model as a RL policy strikes a balance between the computational efficiency and the modeling accuracy of multi-modal distribution on offline RL dataset, and achieves comparable performances with the diffusion policies but significant speedup in three typical RL settings. The proposed Consistency-AC algorithm leverages a novel policy representation with policy regularization for offline RL, orthogonal to other offline RL techniques. Future directions include combining the consistency policy with other techniques like conservative Q-value estimation for offline RL, better alignment and initialization from offline to online fine-tuning, advanced exploration methods for online RL, etc. Scaling up the task complexity, where more sampling steps are required for the generative models, will reveal a greater potential for consistency policy to show its benefits in retaining expressiveness while reducing the computational cost.
C.2 COMPUTATIONAL TIME Overall Training Time. Tab. 6 shows the comparison of Diffusion-BC and Consistency-BC in terms of the computational time during training for D4RL Gym, AntMaze, Adroit and Kitchen tasks. Each result is averaged over five random seeds with standard deviations reported. Since different environments are trained for various numbers of total epochs, the comparison is based on per-epoch time consumption. The two methods use the same batch size and number of iterations within each epoch, as well as the same network architecture. Consistency-AC algorithm. The average scores for five random seeds over D4RL Gym, AntMaze, Adroit and Kitchen are shown in Tab. 7. Table 7: The performance of Consistency-AC variants on D4RL Gym, AntMaze, Adroit and Kitchen tasks for offline RL setting. Each cell has two values: one for offline model selection and another (in brackets) for online model selection. Each result is averaged over five random seeds with standard deviations reported. D.4 MORE RESULTS Table 10: Comparison of scores (unnormalized, maximum over epochs) for methods in offline-to-online and online RL settings.
https://github.com/quantumiracle/Consistency Model For Reinforcement Learning
Our experiments use constants ϵ = 0.002,T = 80;ρ = 7;s0 = 2,s1 = 150 followingSong et al. (2023)