Symmetry is a powerful inductive bias that can be used to improve generalization and data efficiency in deep learning. One way to leverage symmetry Figure 1: An approximately equivariant policy π on a Reacher domain, where the goal is to determine the torques (green, magenta) to apply on each joint for the fingertip to reach the target (red). Due to wear, the first joint is more responsive to positive torques. When the state is flipped, the policy also flips the actions but can learn to adjust for symmetry breaking factors.
is through equivariant neural networks, which are model classes constrained to respect the symmetry of a known ground truth. Equivariant neural networks have successfully been applied to image classification (Cohen and Welling, 2016;Worrall et al., 2017), particle physics (Bogatskiy et al., 2020), molecular biology (Satorras et al., 2021;Thomas et al., 2018), and robotic manipulation (Wang et al., 2022b). Empirical studies have demonstrated that equivariant networks require much fewer data than their standard network counterparts (Winkels and Cohen, 2018;Wang et al., 2022b), can have fewer parameters (Weiler and Cesa, 2019;He et al., 2022), and can generalize better to unseen data (Wang et al., 2020;Fuchs et al., 2020).
However, equivariant neural networks crucially assume that the data is perfectly symmetric in both the inputs and outputs, which may not be true in real-world data such as fluid dynamics (Wang et al., 2022c) or finan-cial data (Black, 1986). By relaxing the strict equivariance constraints, approximately equivariant networks can outperform exactly equivariant and unconstrained networks in the presence of asymmetry. While various approaches to achieve approximate equivariance have been proposed (Wang et al., 2022c;van der Ouderaa et al., 2022;McNeela, 2023;Kim et al., 2023), they focus on vision-based tasks or dynamics modeling.
One area where symmetry has been especially useful is in reinforcement learning (RL), where equivariant networks greatly improve sample efficiency (Wang et al., 2022b;Zhu et al., 2022), a key challenge in RL. However, most works consider exact symmetry and use exactly equivariant networks, which cannot address symmetry breaking in the reward or transition functions or noise in the observations. In this work, we employ relaxed group and steerable convolutional neural networks for RL (Wang et al., 2022c); they are flexible enough to adapt to approximate equivariance but also have improved efficiency and robustness.
In this paper, we theoretically and empirically investigate approximately equivariant reinforcement learning. Our key contributions are to:
• formalize the notion of approximately equivariant MDPs and prove the (optimal) value function in such MDPs exhibits approximate equivariance, motivating the use of approximately equivariant RL, • introduce a novel approximately equivariant RL architecture using relaxed group convolutions, • demonstrate improved sample efficiency and robustness to noise for our approximately equivariant RL compared to other baselines with or without symmetry biases, • successfully apply approximate equivariant RL to real-world financial data.
Equivariant Reinforcement Learning Early works explored equivalence classes in reinforcement learning from the lens of abstractions by defining MDP homomorphisms (Ravindran and Barto, 2002;Zinkevich and Balch, 2001). More recently, several approaches have combined function approximation with RL with equivariant neural networks ( Van der Pol et al., 2020;Wang et al., 2022b;Mondal et al., 2020) with significantly improved sample efficiency. However, all of these works considered perfectly symmetric domains where the policy is constrained to be exactly equivariant. This paper considers domains with symmetry breaking factors where exactly equivariant networks can be suboptimal.
Approximate Equivariant Architectures There has been recent interest in exploring approximate equivariance and approximately equivariant neural networks (Finzi et al., 2021;Wang et al., 2022c;Romero and Lohit, 2022;van der Ouderaa et al., 2022;McNeela, 2023;Petrache and Trivedi, 2024;Samudre et al., 2024). Wang et al. (2022cWang et al. ( , 2024b) ) use a linear combination of exactly equivariant convolution kernels with learnable weights to achieve relaxed equivariance and discover symmetry breaking factors. van der Ouderaa et al. (2022) define a nonstationary kernel and a tunable frequency parameter to control the amount of approximate equivariance. McNeela (2023) propose using a neural network to approximate the exponential map from the Lie algebra to the group to learn almost equivariant functions. Petrache and Trivedi (2024) give theoretical bounds on when approximate equivariance can improve generalization. However, none of these works studied approximate equivariance in RL, the main focus of this work.
Closest to our setting is Residual Pathway Priors (Finzi et al., 2021), which considered soft equivariance constraints in model-free RL. They construct a relaxed equivariant neural network layer as the sum of an exactly equivariant and a non-equivariant layer with a prior on the equivariant layer. We take a different approach in this work and use relaxed group convolutions Wang et al. (2022c), which are flexible enough to learn different outputs for each transformation.
Learning with Latent Symmetry Other works also apply equivariant neural networks to domains with latent symmetry. These are cases where the full state has exact symmetry but only partial observations with an unknown group action are available to the model. Park et al. ( 2022) learn the out-of-plane rotations from 2D images using a symmetric embedding network while others have learned 3D rotational features from images using manifold latent variables (Falorsi et al., 2018) or disentanglement (Quessard et al., 2020). Wang et al. (2022a) find that equivariant models where the group acts directly on observation space perform well in RL even with camera skew or occlusions. They define extrinsic equivariance (transformed samples are outside the data distribution) and show that it can benefit in some scenarios but can also be harmful in certain cases (Wang et al., 2024a). Unlike these works where the observation is partial and does not contain full information about the state, we assume that the domains are fully observable and consider various symmetry breaking factors.
In this section, we provide some background on symmetry groups and equivariant functions. As building blocks of exactly and approximately equivariant networks, we also describe exact and relaxed group convolutions, respectively.
A symmetry group G is a set equipped with a binary operation that satisfies associativity, existence of an identity, and existence of inverses. A group can act on vector space X via a group representation ρ X which homomorphically assigns each element g ∈ G an invertible matrix ρ X (g) ∈ GL(X). For example, for a finite group G, the regular representation acts on R |G| by permuting basis elements {e g : g ∈ G} as
That is, transformations of the input x by g correspond to transformations of the output by the same group element. We can enforce this constraint in equivariant neural networks to learn only over the space of equivariant functions by replacing linear layers with group or steerable convolutional layers. One benefit of enforcing equivariance is lower sample complexity as the network searches over a reduced function class.
One method of constructing equivariant network layers is by group convolution (Cohen and Welling, 2016), which we briefly describe here. Group convolutions map between features which are signals over the group f : G → R. For inputs not natively of this form, a lift operation must first be performed. Let ψ θ : G → R be the convolutional kernel parameterized by θ. A Gequivariant group convolutional layer is defined as
Equivariance follows from the fact that the kernel depends only on the product g -1 h and not the specific elements (g, h). For example, if we consider equivariance across translations, we obtain the standard convolution where h, g ∈ Z 2 and g -1 h = h -g. Another possible approach to constructing equivariant network layers is with G-steerable convolutions (Cohen and Welling, 2017), which can generalize to continuous groups.
A key component of our method is the relaxed version of the group convolution (Wang et al., 2022c). The kernel ψ is replaced with several kernels {ψ l } L l=1 and the output is composed as a linear combination. The relaxed group convolution is defined as
where w l are the relaxed weights and each ψ l θ are constrained to be exactly equivariant. Note that as w l (h) depends on the specific element h, this breaks the strict equivariance of the group convolution. Wang et al. (2022c) also introduce relaxed versions of steerable convolutions, see Wang et al. (2022c) or Appendix C.2 for more details.
There have been several different definitions of approximate, relaxed, or partial equivariance. In this paper, we use the definition given by Petrache and Trivedi (2024). We give some background to build up to the definition. Let G be a group and f : X → Y, x → y be the task function.
Definition 1 (Equivariance Error). For g ∈ G and x ∈ X, the equivariance error ee(f, g, x) is defined as
Equivariance error measures exactly how far a function is from perfect equivariance with respect to G for a particular x. For an exactly G-equivariant function, ee(f, g, x) = 0 for all g ∈ G and x ∈ X.
Definition 2 (ε-stabilizer). The ε-stabilizer of f and G is defined as
The ε-stabilizer gives the set of group elements for which the equivariance error is under some threshold.
We adopt the definition of approximate equivariance where f has bounded equivariance error for all g ∈ G, in contrast to partial equivariance, where Stab ε (f, G) < G.
We first theoretically characterize the problem by defining approximately equivariant Markov decision processes (MDP). We then prove that environments with approximate symmetry admit approximately invariant Q functions. This motivates our method of using approximately equivariant neural networks to learn the policy and Q function.
Consider an infinite-horizon discounted-reward Markov decision process (MDP) represented by a tuple M = (S, A, P, R, γ) with state space S, action space A, instantaneous reward function R : S × A → R, a transition function P : S × A → ∆ S and discount factor γ ∈ (0, 1).
Let π : S → ∆ A be a policy giving the probability π(a|s) of taking action a in state s. The expected cumulative reward of using the policy starting from state s (or state s and action a) are the value functions defined as follows
The goal is to find a policy π * that maximizes the expected return with an initial state distribution ξ
We denote
Let G be a group acting on S and A. Denote the action of an element g ∈ G on s and a by gs and ga, respectively. We now extend the definition of Equivariant MDPs (Van der Pol et al., 2020) to cases where the symmetry is approximate.
where the Minkowski functional w.r.t F is
For the total variation distance ρ F (f ) := 1 2 (max fmin f ) and for Kantorovich metric ρ F (f ) := ∥f ∥ Lip .
The following theorem provides a characterization of the gap between the value functions in the original and symmetry transformed domain, for the (G, ϵ R , ϵ P )-invariant MDP described in Definition 4. Theorem 1 highlights that the Q-function is approximately group-invariant, where the approximation is now a function of the reward and transition mismatch, the discount factor, and the Minkowski functional evaluated on the optimal value function. Theorem 1. Let the rewards R be bounded R min ≤ R ≤ R max , 0 ≤ γ < 1 and let g ∈ G be an onto mapping. For any state s and action a, we have
.
Theorem 1 implies that when the invariance mismatch is small -i.e., when the domain has only minor symmetry violations -the Q-function is approximately group-invariant. A proof is provided in Appendix A. Note that, in Theorem 1, when the Kantorovich metric is used for uncertainty characterization, ρ F (V * ) = ∥V * ∥ Lip , where ∥ • ∥ Lip is the Lipschitz norm of the value function (Gelada et al., 2019). For total variation distance, ρ
Also, from Theorem 1, it is clear that when γ ∈ [0, 1), we obtain a non-trivial characterization, while γ = 1 results in a trivial and uninformative bound. This is the limitation of the infinite horizon setting, and can be remedied by considering an arbitrary finite-horizon setup. We do this for the sake of completeness in Appendix B. We not only show that the finite horizon setup allows for a time-dependent transition function, but also obtain an approximate group-invariance of the time dependent Q-function in terms of similar elements that appear in Theorem 1.
There are different ways to use the above results. One can discover how approximate the value functions are and learn α, or one can incorporate approximate equivariance into the model and leverage the benefits of equivariance. We take the latter approach and consider approximately equivariant networks for the policy and critic in domains with inexact symmetry.
We propose approximately equivariant versions of two commonly used actor-critic algorithms, DrQv2 (Yarats et al., 2021) and SAC (Haarnoja et al., 2018). In doing so, we generalize exactly equivariant versions of SAC (Wang et al., 2022b) and DrQv2 (Wang et al., 2022a) from previous works by replacing strictly equivariant layers with relaxed equivariant layers. Illustrative Example We first illustrate how to apply our proposed approximately equivariant actorcritic architecture on the Reacher domain; see Figure 2. The objective is to actuate a two-joint arm so that the end effector reaches the red point. The state is a stack of consecutive images s ∈ R C×H×W and the action a ∈ R 2 corresponds to torques for the first and second arms. There is clear rotational and reflectional symmetry in this domain. If the state (image)
is rotated, the action should be invariant to rotations as they are angular torques. If the state is reflected, then the action would also correspondingly be flipped (in sign). However, as in the example in Figure 1, the first joint is more responsive to positive torques, which breaks rotational and reflectional symmetry.
For this domain, we implement approximate equivariance to the group D 2 of vertical reflections and π rotations. The group D 2 transforms the input states by image transformations, where the input images are reflected or rotated. Latent representations are images z : R 2 → R C where g ∈ D 2 acts on the pixel axes by image transformation and on the channel axis by permutations corresponding to the regular representation of D 2 , i.e. (gz)(x, y) = ρ reg (g)z(g -1 •(x, y)). Note that the latent representations can be high-dimensional, consisting of a direct sum of several different or repeated low-dimensional representations of D 2 . For the output, the torques a 1 and a 2 are scalars that change sign under reflection but are invariant under rotations.
Encoder, Policy, and Critic We extend exactly equivariant versions of SAC (Wang et al., 2022b) and DrQv2 Wang et al. (2022a) by replacing each group convolution with relaxed group convolutions for the encoder, policy, and critics. Practically, each relaxed group convolution layer contains L exactly equivariant kernels ψ l and the output is a linear combination of the outputs of these convolutions and relaxed weights w l (g). The w l (g) also transform as the regular representation of G, see Section 3.1 for the definition for finite groups.
The encoder E and the policy π are approximately equivariant. The latent state z output by E is defined to transform as the direct sum of regular representations of G. The action representation is domainspecific. The critics are approximately invariant and output scalars q (s,a) that are fixed by G, i.e. transform via the trivial representation. For more details, please see Section 5 and Appendix C.
In the case of continuous groups, we can also construct relaxed steerable versions of the encoder, policy, and critics. Analogous to the group convolution case, we can replace the exactly equivariant steerable convolutions with relaxed steerable convolutions. See Appendix C for more details.
We experiment on how approximately equivariant RL compares to methods with exact equivariance and no equivariance in domains with both exact symmetry and various symmetry breaking factors, and to elucidate when approximate equivariance should be preferred. We consider standard continuous control domains and stock trading with real-world data.
We first experiment on four continuous control domains in DeepMind Control Suite (Tassa et al., 2018). Similar to Wang et al. (2022a), we consider a subset of the domains which have apparent symmetry. Acrobot, Cartpole, and BallInCup have reflectional symmetry described by the group D 1 and Reacher has D 2 symmetry. For all domains, the observations are a stack of 3 consecutive RGB images.
We modify the domains to carefully control the type and degree of symmetry breaking that is present. We first remove fixed background features such as random stars in the sky and checkered floors (see Figure 4). These features break symmetry to some extent since they do not transform with the underlying state, but give a form of mild symmetry breaking termed extrinsic equivariance, which has an inconsistent impact on equivariant models (Wang et al., 2022a). We then introduce several different symmetry breaking factors for each domain: 1) repeat action -the action is repeated twice in a certain region of the domain, 2) gravity -gravity is modified from the force vector (0, 0, -9.81) to (a, -a, -9.81) where a ̸ = 0, and 3) reflect action -the action direction is flipped in certain regions of the domain. repeat action and reflect action test local symmetry breaking factors, while gravity tests a global symmetry breaking factor. See Appendix D.1 for more details.
Models For the continuous control tasks, we implement an approximately equivariant(ApproxEquiv) version of a SOTA image-based RL algorithm DrQv2 (Yarats et al., 2021). We compare with exactly equivariant (ExactEquiv) and non equivariant (NonEquiv) versions of the same architecture. We largely use the hyperparameters from Yarats et al. ( 2021) but reduce the latent dimension for more tractable computation for all methods. We also compare against an approximately equivariant model, Residual Pathway Priors (RPP) (Finzi et al., 2021), and a self-supervised symmmetry-aware model, SiT (Weissenbacher et al., 2024). We extend RPP to the DrQv2 architecture by using RPP layers in the encoder, policy, and critics. We find that RPP is somewhat sensitive to the speed τ of the critic moving average (as mentioned in the original paper), and had to reduce its value for Acrobot and BallInCup for stability. We also extend SiT to the DrQv2 architecture by using an SiT as the encoder and standard MLPs for the policy and critics.
Although we adapted the code from the official SiT implementation, we were unable to modify the input image sizes and had to use the image size used in the original paper (64px).
Results Figure 3 show the total episode reward over training. As expected, we confirm that NonEquiv has much lower sample efficiency than the models with a symmetry bias. In the repeat action and reflect action variants of Acrobot, ApproxEquiv significantly outperforms ExactEquiv and RPP. It does slightly worse than ExactEquiv on the Reacher domain but beats RPP, suggesting that the symmetry breaking we introduced was not strong enough to achieve incorrect equivariance. It is also possible that ExactEquiv can infer the symmetry breaking factors from the 3 frames of input, making the task a case of extrinsic equivariance where an equivariant model can succeed Wang et al. (2022a). In CartPole and BallInCup, all methods perform similarly and learn an optimal policy quickly. In domains with exact symmetry (original), our method ApproxEquiv performs similarly to ExactEquiv, showing there is no cost in performance by giving the model the ability to adapt to symmetry breaking in cases where it is not needed. This result supports Proposition 3.1 from Wang et al. (2024b), which proves that relaxed group convolutions initialized to be exactly equivariant stay exactly equivariant when trained with exact data symmetry.
We visualize the relaxed weights of the first layers of the encoder and policy over all runs in Figure 5. If these weights are equal, the model is equivariant; the more they differ the more the model has relaxed the symmetry constraint. For Acrobot and CartPole, the weights differ more for the modified domains than the original symmetric domain, especially for the encoder, while the policy weights vary more for the modified domains of BallInCup. This indicates the relaxed equivariant models have adapted to the symmetry breaking in the domains.
To quantitatively evaluate the models, we select the best-performing policy from all runs and measure the total reward over 50 episodes. The results echo the training curves in Figure 3, where ApproxEquiv performs well, particularly in the domains with symmetry breaking factors (see Table 1).
To test whether approximately equivariant models are robust to noisy observations, we also consider variants of the domains where Gaussian noise are added to the input images only at test time (σ = 0.02 for Acrobot and Reacher, σ = 0.06 for CartPole and BallInCup).
Interestingly, we find that our approach is more robust to noisy inputs than ExactEquiv or NonEquiv, especially on the BallInCup and Reacher domains. We further experiment with training on noisy data and test on noisy domains to see which policies are more robust, see Table 4 in Appendix E. We find that in the BallInCup domains, the approximately equivariant agent is still more robust to noise than the fully equivariant or non equivariant baselines.
We also consider a stock trading task using real world price data, formulated as an MDP (Liu et al., 2018). Given a fixed amount of initial cash, the objective is to learn the optimal number of stocks to buy and sell (once daily) to maximize the portfolio value. The state consists of the current cash balance, the stock prices, the number of shares in the current portfolio, and other technical indicators of each stock. The actions are the number of stocks to buy and sell for each stock. The reward is the scaled difference in portfolio values between consecutive timesteps. We assume that the market dynamics are not affected by our trading. There is a small 0.1% transaction cost for every trade. We use real financial data scraped from Yahoo Finance (yfi, 1997) and consider the stocks in the Models For this domain, we use SAC (Haarnoja et al., 2018) as our RL algorithm and consider equivariance to both the translation group and scaletranslation group across the time dimension. Temporal translations can be useful as the most recent history of stock prices inform your actions, and this information may be approximately preserved across time.
Temporal scaling could also be beneficial as there could be market seasonality, which is only approximately shared across different time scales. As our actions do not affect stock prices, which in turn is directly correlated with the reward, we learn an approximately invariant policy and invariant critic for both symmetry groups. As before we compare approximately equivariant, strictly equivariant, and unconstrained models. We evaluate each method on the final portfolio value (equivalent to the total episode reward), annualized return, and the Sharpe ratio (Sharpe, 1994), which is a standard financial metric that measures an asset's risk-adjusted performance. We also include as baselines a uniform holding strategy Uniform, where we initially buy equal values of each stock and hold, and the Dow Jones index ^DJI.
Results Table 2 lists the average test results of the learned policies on the stock trading domain. The ApproxEquiv model for both translation (T) and scaletranslation (ST) outperform all baselines, with annualized returns of 10.6% and 12.0% respectively. The Exact ST-Equiv model outperforms NonEquiv, while the Exact T-Equiv model does worse. These observations suggest that temporal scale and translation symmetries can be good biases in analyzing financial data and that translation symmetry may be more approximate than scale. We also visualize 10 episode rollouts of the best-performing policies in Figure 6, with the portfolio values on the left and transaction costs on the right. The Approx ST-Equiv method achieves the highest portfolio value for most timesteps and incurs lower transaction costs than the exactly equivariant policies. We note that overall the annualized returns are fairly low, as the test dataset from 2021-01-01 to 2024-07-01 includes both the COVID-19 pandemic and 2022 stock market decline.
We visualize the relaxed weights of the first layer of the encoder across translation (left) and scale (right) in Figure 7. For translation, our model places higher weights on the very last timestep. This matches our intuition as the most recent stock prices and portfolio holdings would be most informative in determining the optimal action. For scale, we find that the relaxed weights do not differ greatly, but there is increased variance with increasing scale.
We proposed a novel approximately equivariant architecture using relaxed group convolutions for modelfree reinforcement learning. Our experimental results on continuous control domains and a stock trading problem with real-world data demonstrate that the approximately equivariant model performs similarly to an exactly equivariant model in domains with perfect symmetry but outperforms it in most domains with symmetry breaking factors. This suggests that our method can act as a much more flexible alternative to exactly equivariant agents that can boost sample efficiency in a wider variety of settings and is also more robust to perturbations.
Limitations and Future Work While we did consider real-world data in the stock trading domain, our continuous control domains used simplified observations and synthetic symmetry breaking. Furthermore, exactly equivariant networks perform better in some modified domains than others (Reacher vs. Acrobot).
Another limitation is that, as with all equivariant networks, the symmetry group and how it acts on the state and action spaces need to be known in advance.
An interesting future direction could be to quantify exactly what types of symmetry breaking factors could lead to higher performance for approximately equivariant RL, possibly by measuring equivariance error.
Other future work includes proving bounds on the optimal policy π(s) and π(gs) or applying approximately equivariant RL in robotic manipulation, where kinematic constraints or obstacles can break symmetry.
1. For all models and algorithms presented, check if you include:
(a) A clear description of the mathematical setting, assumptions, algorithm, and/or model.
[Yes] (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm.
[No] (c) (Optional) Anonymized source code, with specification of all dependencies, including external libraries.
[Yes]
2. For any theoretical claim, check if you include:
(a) Statements of the full set of assumptions of all theoretical results. [Yes] (b) Complete proofs of all theoretical results.
[Yes] (c) Clear explanations of any assumptions. [Yes] 3. For all figures and tables that present empirical results, check if you include:
(a) The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL).
[Yes] (b) All the training details (e.g., data splits, hyperparameters, how they were chosen).
[Yes] (c) A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times).
[Yes] (d) A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider).
[Yes] 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator If your work uses existing assets. [Yes] (b) The license information of the assets, if applicable. [Yes] (c) New assets either in the supplemental material or as a URL, if applicable. [Yes] (d) Information about consent from data providers/curators. [No] (e) Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. [Not Applicable] 5. If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to participants and screenshots. [Not Applicable] (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. [Not Applicable] (c) The estimated hourly wage paid to participants and the total amount spent on participant compensation. [Not Applicable]
Proof. We will prove the results using induction. First, note that the result is true for T by definition. Suppose the result is true for t + 1, and consider the differential at time t,
gs t , ga t )| ≤ |R(s t , a t ) -R(gs t , ga t )| + γ S V t+1 (s t+1 )P (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P (gs t+1 |gs t , ga t ) ≤ ϵ R + γ S V t+1 (s t+1 )P (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P (s t+1 |s t , a t ) + γ S V t+1 (gs t+1 )P (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P (gs t+1 |gs t , ga t )
The last inequality follows by using the decomposition using Minkowski's functional. Further, note that
by induction assumption, and the fact that when g is onto sup
The result follows.
Proof. We have by definition,
Similarly, we have
We now prove Theorem 1. Let B(S) denote the Banach space of bounded real-valued functions on S. We define the Bellman optimality operator B : B(S) → B(S) such that for any uniformly bounded function V ∈ B(S),
It is known that V * is the (unique) fixed point of B, i.e., BV * = V * . We note that V * also satisfies the following equation for any s V * (gs) = sup a∈A R(gs, ga) + γ S V * (gs ′ )P (gs ′ |gs, ga) .
To see this, consider the following arguments.
Q * (gs, ga)P (s ′ |gs, ga).
Since g ∈ G permutes the elements of G, re-indexing the integral using s′ = gs ′ , we have
Q(gs ′ , ga ′ )P (gs ′ |gs, ga).
Consider a sequence of value functions V (n) on the symmetry transformed domain as follows: V (0) (gs) = 0 and V (n+1) = BV (n) . For an arbitrary T , we have using Proposition 1 for any t ∈ {1, • • • , T },
where
From Proposition 2, we have, noting that V(s) = sup a Q(s, a), that
By Banach fixed point theorem, we know that lim
. Therefore, taking the limit, we have
A similar argument establishes the result for Q using the onto function g. The claims in Theorem 1 follows by recognizing that V t and Q t exactly equal V * and Q * .
As is clear from Theorem 1, when γ → 1, the bound becomes trivial and not useful. In this section we will briefly discuss the case when the discount factor γ = 1. In this setting, we allow the transition functions to be a function of t.
Proposition 3. Let |R(gs t , ga t ) -R(s t , a t )| ≤ ϵ R and d F P t (gs ′ t | gs t , ga t ), P t (s ′ t | s t , a t ) ≤ ϵ P (t). For a finite-horizon MDP of duration T , we have |Q t (s t , a t ) -Q t (gs t , ga t )| ≤ α t , |V t (s t ) -V t (gs t )| ≤ α t where α T +1 = 0 and for t ∈ {1, 2, • • • , T },
Proof. The proof proceeds as in Proposition 1. We have
gs t , ga t )| ≤ |R(s t , a t ) -R(gs t , ga t )| + S V t+1 (s t+1 )P t (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P t (gs t+1 |gs t , ga t ) ≤ ϵ R + S V t+1 (s t+1 )P t (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P t (s t+1 |s t , a t ) + S V t+1 (gs t+1 )P t (s t+1 |s t , a t ) -S V t+1 (gs t+1 )P t (gs t+1 |gs t , ga t )
The result follows by recursion.
Group convolutions (Cohen and Welling, 2016) generalize standard convolutions, which are translationequivariant, to be equivariant to a group G. Group convolutions act on signals over the group f : G → R. As many data samples are not natively of this form (e.g. an image), the input must first be lifted onto a function in G. For example, let f 0 : Z 2 → R be the input signal, a grayscale image, and H = D 2 be the group. The lifting convolution lifts
where h ∈ H. Practically, the lift operation creates |H|, the order of group H, images by acting on x by h -1 . Typically the lift operation is the first layer of the network, followed by subsequent group convolutions, nonlinearities, or other equivariant layers. We use relaxed versions of the lift and group convolutions as described in Wang et al. (2022c) and the main paper.
As an alternative to group convolutions, one can use steerable convolutions (Weiler et al., 2018) that use weight tying to generalize to continuous groups and are more parameter-efficient. Let H < O(2) be the subgroup which acts on R 2 by matrix multiplication on the input and output channel spaces R c and R d by ρ in and ρ out , respectively. Then G = H ⋉ R 2 . Given input signal f : R 2 → R c , then standard convolution over R 2 with kernel ψ
for all h ∈ H. Intuitively, this kernel constraint ensures that the output features transform by ρ out when the input features are transformed by ρ in . Kernels that satisfy this constraint have been solved for many common subgroups of E(2), see Weiler and Cesa (2019) for more details.
Using the example of grayscale images as in Section C.1, let the input feature be f : Z 2 → R and {ψ k } K k=1 be a precomputed, nontrainable equivariant kernel basis of K kernels that satisfy Eq. ( 9). Assume that both the number of input and output channels is 1 and let w ∈ R K be the trainable coefficients of the kernels. Then a G-steerable convolution is defined as
where x ∈ Z 2 is the spatial position and w k is the weight associated with kernel ψ k .
Relaxed Steerable Convolution As described in Wang et al. (2022c), one can also use relaxed versions of steerable convolutions by letting the trainable weights w also depend on y. A relaxed G-steerable convolution is defined as
Allowing the trainable weights w k to also depend on the absolute spatial position y breaks the equivariance constraint in Eq. ( 9).
By replacing relaxed group convolutions with relaxed steerable convolutions, we can also design a variant of our proposed approximately equivariant RL architecture (Figure 8).
Acrobot We use the swingup task. The domain consists of two joints where the goal is to apply torque to the inner joint so that both joints are near vertical. We use D 1 as the symmetry group, i.e. vertical reflection, and the action a ∈ R transforms via the sign representation ρ sign , where ρ sign (flip)(a) = -a. For variants, we consider 1) repeat action -the action is repeated when the inner joint is in the fourth quadrant and 2) gravity -gravity ⃗ g = [0, 0, -9.81] is modified to [-2, 2, -9.81].
CartPole We consider the swingup task. The domain consists of a pole swinging on a cart and the goal is to move the cart left or right (a ∈ R) to make the pole upright. The symmetry group and action representation are the same as in Acrobot, D 1 and ρ sign . For variants, we consider 1) repeat action -the action is repeated when the pole is in the first quadrant, 2) gravity -gravity is modified to [0.2, -0.2, -9.81], and 3) reflect action -the pole angle is in [0, π 4 ]. Gravity is modified less than in Acrobot as too high values forced the cart out of frame.
The domain consists of a ball attached to the bottom of the cup and the goal is to move the cup to catch the ball inside the cup. The action (x, z) ∈ R 2 is the cup's spatial position. The symmetry group is D 1 and the action representation is ρ sign ⊕ ρ 0 , where the x position transforms via the sign representation and the z transforms via the trivial representation ρ 0 . For variants, we consider 1) repeat action -the ball x position greater than 0.0 and z position is greater than 0.3, 2) gravity -gravity is modified to [-2, 2, -9.81], and 3) reflect action -same as repeat action.
Reacher We consider the hard task. The domain consists of two joints and the goal is to apply torques to make the end effector reach the target. The action a ∈ R 2 . The symmetry group is D 2 , i.e. vertical reflections and π rotations, and the action transforms via the quotient representation 2ρ quot , where the torques for both joints are invariant to rotations and flip signs for vertical reflections. For variants, we consider 1) repeat action -the inner joint angle is in [0, π 2 ] and 2) reflect action -the inner joint angle is in [ π 2 , π].
For all DeepMind Control Suite (DMC) domains, we fix the episode length to 1000 and use RGB image of size 85 × 85. We considered four domains of varying difficulty, of which Acrobot is the hardest. In the original DrQv2 implementation (Yarats et al., 2021), the encoder reduces the spatial dimensions to 35 × 35, which is then flattened to be input to the policy and critic. We follow Wang et al. (2022a) and further reduce the spatial dimensions to 7 × 7 for faster training for all models. We reduce the replay buffer size from 1,000,000 to 500,000 to slightly reduce the memory footprint. All other hyperparameters are kept the same as in Yarats et al. (2021).
For the exactly equivariant and approximately equivariant models, we reduce the number of channels by |G| where |G| is the order of the group to preserve roughly the same number of parameters as the non-equivariant model. We use L = 1 filters for the approximately equivariant model in all experiments.
RPP contains both the non-equivariant layers and exactly equivariant layers and thus has roughly twice as many parameters as ExactEquiv. For the critic moving average speed τ , we use the default τ = 0.01 for CartPole and Reacher and τ = 0.009 for Acrobot and Ball in Cup.
The plots in Figure 3 show the mean reward of 10 episodes, evaluated every 20,000 environment steps. For the results in Table 1, we use σ = 0.02 for Acrobot and Reacher and σ = 0.06 for CartPole and Ball in Cup.
The continuous control experiments were run on single GPUs of different types. Acrobot was run on an Nvidia RTX 4090 and all other experiments were run on an Nvidia RTX 2080 Ti. We note that the wall clock time for training both exactly and approximately equivariant models is longer than that for a non equivariant model, even though they are generally more sample efficient. This is because equivariant neural networks often incur more overhead in implementation -for group convolutions, the kernel must be transformed and the outputs must be stacked and for steerable convolutions, the basis must be projected onto matrices at every forward pass.
We formulate the stock trading problem as an MDP as described in Liu et al. (2018). The state consists of the cash balance c t , the stock prices p n t , the number of shares in the current portfolio h n t , and other technical indicators i n t for time t stock n ∈ {1, . . . , N }. The actions x n t are the number of stocks to buy and sell for each stock n and are bounded to [-M, M ] where M was set to 100. The reward r t is the scaled difference in portfolio values between consecutive timesteps and we assume that the market dynamics are not affected by our trading. There is a small transaction cost ϵ n = 0.001 for every trade. Initially, the portfolio contains 0 shares and the cash balance is 1,000,000. This can be formulated as a constrained program as follows
Table 4 shows the results from training policies with noisy inputs and evaluating their robustness to noise at test time. This experiment tests whether our approximately equivariant method is truly more robust to noise than other baselines trained with noise augmentation. We find that our approximately equivariant method is more robust than baselines for the modified domain, even when trained with noise augmentation.