Partial observability is a key characteristic of many real-world reinforcement learning (RL) control problems where the agent lacks access to the system state, and is restricted to operate based on the observable past, a.k.a. the history. Such control problems are commonly encoded as partially observable Markov decision processes (POMDPs) [15], which are the focus of a significant amount of research effort. Offline learning/online execution is a common RL framework where an agent is trained in a simulated offline environment before operating online, which offers the possibility of using latent information not generally available in online learning, e.g., the simulated system state, or the state belief from the agent's perspective [6,14,16,25,26,34].
Offline learning methods are in principle able to exploit this privileged information during training to achieve better online Figure 1: Memory-Four-Rooms-9x9, a procedurally generated navigation task which requires information-gathering and memorization. The agent must avoid the bad exit and reach the good exit, which is identifiable by the color of the beacon. performance, so long as the resulting agent does not use the latent information during online execution. Specifically, actor-critic methods [17,31] are able to adopt this approach via critic asymmetry, where the policy and critic models receive different information [9,18,20,26,32,36,37], e.g., the history and latent state, respectively. This is possible because the critic is merely a training construct, and is not required or used by the agent to operate online. By the very nature of actor-critic methods, critic models which are unable or slow to learn accurate values act as a performance bottleneck on the policy. Consequently, critic asymmetry is a powerful tool which, if carried out with rigor, may provide significant benefits and bootstrap the agent's learning performance.
Unfortunately, existing asymmetric methods use asymmetric information heuristically, and demonstrate their validity only via empirical experimentation on selected environments [9, 18, 20, 21, 25-28, 32, 36, 37]; the lack of a sound theoretical foundation leaves uncertainties on whether these methods are truly able to generalize to other environments, particularly those wich feature higher degrees of partial observability (see Figure 1). In this work, (a) we analyze a standard variant of asymmetric actor-critic and expose analytical issues associated with the use of a state critic, namely that the state value function is generally ill-defined and/or causes learning bias; (b) we prove an asymmetric policy gradient theorem for partially observable control, an extension of the policy gradient theorem which explicitly uses latent state information; (c) we propose a novel unbiased asymmetric actor-critic method, which lacks the analytical issues of its biased counterparts and is, to the best of our knowledge, the first of its kind to be theoretically sound; (d) we validate our theoretical findings through empirical evaluations on environments which feature significant amounts of partial observability, and demonstrate the advantages of our unbiased variant over the symmetric and biased asymmetric baselines.
This work sets the stage for other asymmetric critic-based policy gradient methods to exploit asymmetry in a principled manner, while learning under partial observability. Although we focus on advantage actor-critic (A2C), our method is easily extended to other critic-based learning methods such as off-policy actor-critic [8,33], (deep) deterministic policy gradient [19,29], and asynchronous actorcritic [22]. Offline training is also the dominant paradigm in multiagent RL, where many asymmetric actor-critic methods could be similarly improved [9,18,20,21,27,28,32,36,37].
The use of latent information during offline training has been successfully adopted in a variety of policy-based methods [7,9,18,20,26,32,34,36,37] and value-based methods [7,21,27,28]. Among the single-agent methods, asymmetric actor-critic for robot learning [26] uses a reactive variant of DDPG with a state-based critic to help address partial observability; belief-grounded networks [25] use a belief-reconstruction auxiliary task to train history representations; and Warrington et al. [34] and Chen et al. [6] use a fully observable agent trained offline on latent state information to train a partially observable agent via imitation.
Asymmetric learning has also become popular in the multi-agent setting: COMA [9] uses reactive control and a shared asymmetric critic which can receive either the joint observations of all agents or the system state to solve cooperative tasks; MADDPG [20] and M3DDPG [18] use the same form of asymmetry with individual asymmetric critics to solve cooperative-competitive tasks; R-MADDPG [32] uses recurrent models to represent non-reactive control, and the centralized critic uses the entire histories of all agents; CM3 [37] uses a state critic for reactive control; while ROLA [36] trains centralized and local history/state critics to estimate individual advantage values. Asymmetry is also used in multi-agent value-based methods: QMIX [28], MAVEN [21], and WQMIX [27] all train individual Q-models using a centralized but factored Q-model, itself trained using state, joint histories, and joint actions.
In this section, we review background topics relevant to understand our work, i.e., POMDPs, the RL graphical model, standard (symmetric) actor-critic, and asymmetric actor-critic.
Notation. We denote sets with calligraphy X, set elements with lowercase 𝑥 ∈ X, random variables (RVs) with uppercase 𝑋 , and the set of distributions over set X as ΔX. Occasionally, we will need absolute and/or relative time indices; We use subscript 𝑥 𝑡 to indicate absolute time, and superscript 𝑥 (𝑘) to indicate the relative time of variables, e.g., 𝑥 (0) marks the beginning of a sequence happening at an undetermined absolute time, and 𝑥 (𝑘) is the variable 𝑘 steps later. We also use the bar notation to represent a sequence of superscripted variables x = (𝑥 (0) , 𝑥 (1) , 𝑥 (2) , . . .).
A POMDP [15] In the partially observable setting, the agent lacks access to the underlying state, and actions are selected based on the observable history ℎ, i.e., the sequences of past actions and observations. We denote the space of realizable 1 histories as H ⊆ (A × O) * , and the space of realizable histories of length 𝑙 as H 𝑙 ⊆ (A × O) 𝑙 . Generally, an agent operating under partial observability might have to consider the entire history to achieve optimal behavior [30], i.e., its policy should represent a mapping 𝜋 : H → ΔA. The beliefstate 𝑏 : H → ΔS is the conditional distribution over states given the observable history, i.e., 𝑏 (ℎ) = Pr(𝑆 | ℎ), and a sufficient statistic of the history for optimal control [15]. We define the history reward function as R(ℎ, 𝑎) = E 𝑠 |ℎ [R(𝑠, 𝑎)]; from the agent's perspective, this is the reward function of the decision process. We denote the last observation in a history ℎ as 𝑜 ℎ , and say that an agent is reactive if its policy 𝜋 : O → ΔA only uses 𝑜 ℎ rather than the entire history. A policy's history value function 𝑉 𝜋 : H → R is the expected return following a realizable history ℎ,
which supports an indirect recursive Bellman form,
Some of the theory and results developed in this document concerns whether certain RVs of interest are well-defined; therefore, we review the RVs defined by POMDPs. The environment dynamics and the agent policy jointly induce a graphical model (see Figure 2) over timed RVs 𝑆 𝑡 , 𝐴 𝑡 , and 𝑂 𝑡 . Note that only timed RVs are defined A probability is a numeric value associated with the assignment of a value 𝑥 from a sample space X to an RV 𝑋 , e.g., Pr(𝑋 = 𝑥). Although it is common to use simplified notation to informally omit the RV assignment (e.g., Pr(𝑥)), it must always be implicitly clear which RV (𝑋 ) is involved in the assignment. In the reinforcement learning graphical model, a probability is well-defined if and only if (a) it is grounded (implicitly or explicitly) to timed RVs (or functions thereof); or (b) it is time-invariant (i.e., it can be impicitly grounded to any time index). For example, Pr(𝑠 ′ | 𝑠, 𝑎) is implicitly grounded to the RVs of a state transition Pr(𝑆 𝑡 +1 = 𝑠 ′ | 𝑆 𝑡 = 𝑠, 𝐴 𝑡 = 𝑎), and although the time-index 𝑡 is not clear from context, the probability is time-invariant and thus well defined. As another example, Pr(𝑠 | ℎ) is implicitly grounded to the RVs of a belief Pr(𝑆 𝑡 = 𝑠 | 𝐻 𝑡 = ℎ), where the time-index 𝑡 is implicitly grounded to the history length 𝑡 = |ℎ|, which makes the probability well defined.
Policy gradient methods [31] for fully observable control can be adapted to partial observable control by replacing occurrences of the system state 𝑠 with the history ℎ (which is the Markov-state of an equivalent history-MDP). In advantage actor-critic methods (A2C) [17], a policy model 𝜋 : H → ΔA parameterized by 𝜃 is trained using gradients estimated from sample data, while a critic model V : H → R parameterized by 𝜗 is trained to predict history values 𝑉 𝜋 (ℎ). Note that we annotate parametric critic models with a hat V , to distinguish them from their analytical counterparts 𝑉 𝜋 . In A2C, the critic is used to bootstrap return estimates and as a baseline, both of which are techniques for the reduction of estimation variance [10]. The actor and critic models are respectively trained on L policy (𝜃 ) + 𝜆L neg-entropy (𝜃 ) and L critic (𝜗).
Policy Loss. The policy loss L policy (𝜃 ) = -E ∞ 𝑡 =0 𝛾 𝑡 R(𝑠 𝑡 , 𝑎 𝑡 ) encodes the agent's performance as the expected return. The policy gradient theorem [17,31] provides an analytical expression for the policy loss gradient w.r.t. the policy parameters,
Value 𝑄 𝜋 (ℎ 𝑡 , 𝑎 𝑡 ) is replaced by the temporal difference (TD) error 𝛿 𝑡 to reduce variance (at the cost of introducing modeling bias),
Critic Loss. The critic loss
𝑡 is used to minimize the total TD error, the gradient of which should propagate through V (ℎ 𝑡 ), but not through the bootstrapping V (ℎ 𝑡 +1 ).
Negative-Entropy Loss. Finally, the negative-entropy loss is commonly used, L neg-entropy (𝜃 ) = -E [ 𝑡 H [𝜋 (𝐴 𝑡 ; ℎ 𝑡 )]], in combination with a decaying weight 𝜆, to avoid premature convergence of the policy model and to promote exploration [35].
While asymmetric actor-critic can be understood to be an entire family of methods which use critic asymmetry, for the remainder of this document we will be specifically referring to a non-reactive and non-deterministic variant of the work by Pinto et al. [26], which uses critic asymmetry to address image-based robot learning. Their work uses a reactive variant of deep deterministic policy gradient (DDPG) [19] trained in simulation, and replaces the reactive observation critic V (𝑜) with a state critic V (𝑠); the variant we will be analyzing applies the same critic substitution to A2C. In practice, this state-based asymmetry is obtained by replacing the TD error of Equation ( 6) (used in both the policy and critic losses) with
Although [26] claim that their work addresses partial observability, their evaluation is based on reactive environments which are effectively fully observable; while the agent only receives a single image, each image provides a virtually complete and occlusion-free view of the entire workspace. In practice, the images are merely high-dimensional representations of a compact state.
In this section, we analyze the theoretical implications of using a state critic under partial observability, as described in Section 3.4, and expose critical underlying issues. The primary result will be that the time-invariant state value function 𝑉 𝜋 (𝑠) of a non-reactive agent is generally ill-defined. Then, we show that the time-invariant state value function 𝑉 𝜋 (𝑠) of a reactive agent is well-defined under mild assumptions, but generally introduces a bias into the training process which may undermine learning. Finally, we show that the time-invariant state value function 𝑉 𝜋 (𝑠) of a reactive agent under stronger assumptions can be both well-defined and unbiased. Later, in Section 5, we provide a more general alternative which guarantees well-defined and unbiased time-invariant state-based value functions for arbitrary policies and control problems.
Informally, the issue with 𝑉 𝜋 (𝑠) is that the state alone does not contain sufficient information to determine the agent's future behavior-which generally depends on the history-and is thus unable to accurately represent expected future returns. Ironically, state values suffer from a form of history aliasing, i.e., being unable to infer the agent's history from the system's state. This is particularly evident in control problems which require the agent to perform forms of information gathering (a common occurrence in partially observable control) which are not reflected in the system state, e.g., reach a certain spot to observe a piece of information which is necessary to determine future optimal behavior and solve the control task. In such cases, the state alone does not generally indicate whether the agent has collected the necessary information in the past or not, and is therefore unable to represent adequately whether the current state is a positive or negative occurrence. Formally, we will show that 𝑉 𝜋 (𝑠) is generally not a well-defined quantity and, even in special cases where it is well-defined, generally introduces a bias in the learning process caused by the imperfect correlation between histories and states; in essence, the average value of histories inferred from the current state is not an accurate estimate of the current history's value.
Methodology. We note that replacing the history critic is intrinsically questionable: the policy gradient theorem for POMDPs (Equation (4)) specifically requires history values, and replacing them with other state-based values will generally result in biased gradients and a general loss of theoretical guarantees. Therefore, we analyze state values 𝑉 𝜋 (𝑠) as stochastic estimators of history values 𝑉 𝜋 (ℎ) and consider the corresponding estimation bias, i.e., the difference between the expected estimate E 𝑠 |ℎ [𝑉 𝜋 (𝑠)] and the ground truth estimation target 𝑉 𝜋 (ℎ) for any given history ℎ.
A policy's state value function 𝑉 𝜋 : S → R is tentatively defined as the expected return following a realizable state 𝑠,
which, if well-defined, supports an indirect recursive Bellman form,
In Equation ( 9), we note the term Pr(𝑎 | 𝑠), which encodes the likelihood of an action being taken from a given state. Because the agent policy depends on histories (not states), this term is not directly available, but must be derived indirectly by integrating over possible histories. Further, because 𝑠 is timeless, and no additional context is available to narrow down time, there is no choice but to integrate over histories of all possible lengths.
Equation ( 11) reveals the probability term Pr(ℎ | 𝑠), which encodes the likelihood of a history having taken place in the past given a current state. While Pr(ℎ | 𝑠) may look harmless, it is the underlying cause of serious analytical issues. As discussed in Section 3.2, a probability is only well-defined if associated with well-defined RVs, and unfortunately such RVs do not exist for Pr(ℎ | 𝑠). On one hand, timed RVs Pr(𝐻 𝑡 = ℎ | 𝑆 𝑡 = 𝑠) cannot be used, because Equation (11) integrates over the sample space of all histories, and not just those of a given length 𝑡. On the other hand, time-less RVs Pr(𝐻 = ℎ | 𝑆 = 𝑠) cannot be used, because such time-less RVs do not exist in the RL graphical model. Ultimately, Pr(ℎ | 𝑠) is mathematically ill-defined, which consequently causes both Pr(𝑎 | 𝑠) and 𝑉 𝜋 (𝑠) to be ill-defined as well.
Theorem 4.1. In partially observable control problems, a timeinvariant state value function 𝑉 𝜋 (𝑠) is generally ill-defined.
The practical implications of an ill-defined value function are not obvious; even though the analytical value function 𝑉 𝜋 (𝑠) is ill-defined, the state critic's V (𝑠) training process is based on valid calculations over sample data, which results in syntactically valid updates of the critic parameters. However, given that asymptotic convergence is theoretically impossible when 𝑉 𝜋 (𝑠) is ill-defined, the critic's target will continue shifting indefinitely based on the recent batches of training data, even when unbiased Monte Carlo return estimates are used to train the critic (without bootstrapping). In practice, the effects are not necessarily catastrophic for all control problems, and likely vary depending on the amount of partial observability, on the agent's need to gather and remember information, and on the specific state and observation representations.
In principle, timed value functions 𝑉 𝜋 𝑡 (𝑠) represent a straightforward solution to all these issues (see appendix [2]). However, learning a timed critic model is likely to pose additional learning challenges, due to the need to generalize well and accurately across time-steps. Rather, we will demonstrate that there are special cases of the general control problem which do guarantee well-defined time-invariant value functions 𝑉 𝜋 (𝑠) (see Sections 4.2 and 4.3). However, before that, we can already show that, even when 𝑉 𝜋 (𝑠) is guaranteed to be well-defined, it is not guaranteed to be unbiased. Theorem 4.2. Even when well-defined, a time-invariant state value function 𝑉 𝜋 (𝑠) is generally a biased estimate of 𝑉 𝜋 (ℎ), i.e., it is not guaranteed that
Proof. Consider two histories which are different, ℎ ′ ≠ ℎ ′′ , and result in different action distributions, 𝜋 (𝐴; ℎ ′ ) ≠ 𝜋 (𝐴; ℎ ′′ ), but are associated with the same belief, 𝑏 (ℎ ′ ) = 𝑏 (ℎ ′′ )-a fairly common occurrence in many POMDPs (see appendix [2]). On one hand, because the two histories result in different behaviors, future trajectories and rewards will differ, leading to different history values, 𝑉 𝜋 (ℎ ′ ) ≠ 𝑉 𝜋 (ℎ ′′ ). On the other hand, because the two beliefs are equal, the expected state values must also be equal,
held for all histories, then it would hold for ℎ ′ and ℎ ′′ too, which implies 𝑉 𝜋 (ℎ ′ ) = E 𝑠 |ℎ ′ [𝑉 𝜋 (𝑠)] = E 𝑠 |ℎ ′′ [𝑉 𝜋 (𝑠)] = 𝑉 𝜋 (ℎ ′′ ) -a simple contradiction. Therefore, either 𝑉 𝜋 (ℎ ′ ) ≠ E 𝑠 |ℎ ′ [𝑉 𝜋 (𝑠)] or 𝑉 𝜋 (ℎ ′′ ) ≠ E 𝑠 |ℎ ′′ [𝑉 𝜋 (𝑠)] (or both). □
We show that 𝑉 𝜋 (𝑠) is well-defined if we make two assumptions about the agent and environment: (a) that the policy is reactive (a common but inadequate assumption); and (b) that the POMDP observation function depends only on the current state, O : S → ΔO, rather than the entire state transition (a mild assumption). Under these assumptions, we can expand Pr(𝑎 | 𝑠) by integrating over the space of all observations (rather than all histories),
In this case, Pr(𝑜 | 𝑠) is time-invariant, and can therefore be implicitly grounded to RVs of any time index Pr(𝑂 𝑡 = 𝑜 | 𝑆 𝑡 = 𝑠). This leads to a well-defined value 𝑉 𝜋 (𝑠) which, however, generally remains biased compared to 𝑉 𝜋 (ℎ), per Theorem 4.2. In addition to Theorem 4.2, which is applicable in a more general setting, see appendix [2] for two additional proofs which also take into account the specific assumptions made here. Broadly speaking, the bias is caused by the fact that hidden in 𝑉 𝜋 (𝑠) is an expectation over observations 𝑜 which are not necessarily consistent with the true history ℎ; each proof covers this issue from different angles.
Although the value function is well-defined under reactive control, there are still two significant issues which preclude these assumptions from representing a general solution: (a) reactive policies are inadequate to solve many POMDPs; and (b) the value function bias may prevent the agent from learning a satisfactory behavior.
We show that the state value function is both well-defined and unbiased under two assumptions: (a) that the policy is reactive (a common but inadequate assumption); and (b) that there is a bijective abstraction 𝜙 : O → S between observations and states (an unrealistic assumption). The abstraction 𝜙 encodes the fact that the environment is not truly partially observable, but rather that states and observations fundamentally contain the same information, albeit at different levels of abstraction. For example, in the control problems used by Pinto et al. [26], and an image displaying a workspace without occlusions is a low-level abstraction (observation), while a concise vector representation of the object poses in the workspace are a high-level abstraction (state).
In this case, the action probability term Pr(𝑎 | 𝑠) does not need to be obtained indirectly by integrating other variables; rather, bijection 𝜙 can be used to relate it to the policy model Pr(𝑎 | 𝑠) = 𝜋 (𝑎; 𝜙 -1 (𝑠)). Contrary to the previous cases, the overall state value function 𝑉 𝜋 (𝑠) is not only well-defined, but also unbiased. Proof. The bijection between 𝑜 ℎ and 𝑠 not only implies a manyto-one relationship between histories and states, but also fully determines the agent's state-conditioned action. In the following derivation, we use these facts to determine the first action and reward, a process which can be repeated indefinitely for future actions and rewards.
(repeat process until end of episode)
The benefit of using a state critic under this scenario is that the critic model can avoid learning a representation of the observations before learning the values [26]. Naturally, the main disadvantage of this scenario is that most POMDPs do not satisfy the bijective abstraction assumption; if anything, this assumption is intrinsically incompatible with partial observability, and any POMDP which satisfies this assumption is really an MDP in disguise. Nonetheless, if a control problem only deviates mildly from full observability, it is likely that a state critic will benefit the learning agent despite the theoretical issues.
In this section, we introduce unbiased asymmetric actor-critic, an actor-critic variant able to exploit asymmetric state information during offline training while avoiding the issues of state value functions exposed in Section 4. Consider the history-state value function 𝑉 𝜋 (ℎ, 𝑠) [5], defined as the expected return following a realizable history-state pair ℎ and 𝑠,
which supports an indirect recursive Bellman form,
Note that the history ℎ and state 𝑠 cover different and orthogonal roles: the history ℎ determines the future behavior of the agent, while the state 𝑠 determines the future behavior of the environment. Compared to the history value 𝑉 𝜋 (ℎ), the state information in 𝑉 𝜋 (ℎ, 𝑠) provides additional context to determine the agent's true underlying situation, its rewards, and its expected return. Compared to the state value 𝑉 𝜋 (𝑠), the history information in 𝑉 𝜋 (ℎ, 𝑠) provides additional context to determine the agent's future behavior, which guarantees that 𝑉 𝜋 (ℎ, 𝑠) is well-defined and unbiased.
Theorem 5.1. For arbitrary control problems and policies, 𝑉 𝜋 (ℎ, 𝑠) is an unbiased estimate of 𝑉 𝜋 (ℎ), i.e., 𝑉 𝜋
As we have done for state values 𝑉 𝜋 (𝑠), we are interested in the properties of history-state values 𝑉 𝜋 (ℎ, 𝑠) in relation to history values 𝑉 𝜋 (ℎ). Theorem 5.1 shows that history and history-state values are related by 𝑉 𝜋 (ℎ) = E 𝑠 |ℎ [𝑉 𝜋 (ℎ, 𝑠)], i.e., history-state values are interpretable as Monte Carlo (MC) estimates of the respective history values. In expectation, history-state values provide the same information as the history values, therefore an asymmetric variant of the policy gradient theorem can be formulated. Theorem 5.2 (Asymmetric Policy Gradient).
Proof. Following Theorem 5.1, we have
Therefore,
As estimators, history-state values 𝑉 𝜋 (ℎ, 𝑠) can be described in terms of their bias and variance w.r.t. history values 𝑉 𝜋 (ℎ). Beyond providing the inspiration for the MC interpretation, Theorem 5.1 already proves that 𝑉 𝜋 (ℎ, 𝑠) is unbiased, while its variance is dynamic and depends on the history ℎ via the belief-state Pr(𝑆 | ℎ); in particular, low-uncertainty belief-states result in low variance, and deterministic belief-states result in no variance. Given that operating optimally in a partially observable environment generally involves information-gathering strategies associated with low-uncertainty belief-states, the practical variance of the historystate value is likely to be relatively low once the agent has learned to solve the task to some degree of success.
Inspired by Theorem 5.2, we propose unbiased asymmetric A2C, which uses a history-state critic V : H × S → R trained to model history-state values 𝑉 𝜋 (ℎ, 𝑠),
Because V (ℎ, 𝑠) receives the history ℎ as input, it can still predict reasonable estimates of the agent's expected future discounted returns; and because it receives the state 𝑠 as input, it is still able to exploit state information while introducing no bias into the learning process, e.g., for the purposes of bootstrapping the learning of critic values and/or aiding the learning of history representations.
Although the history-state value is analytically well-defined, it remains worthwhile to question why the inclusion of the state information should help the actor-critic agent at all. We attempt to address this open question, and consider two competing interpretations, which we call state-as-information and state-as-a-feature.
State as Information. Under this interpretation, state information is valuable because it is latent information unavailable in the history, which results in more informative values which help train the policy. However, we argue that this interpretation is flawed for two reasons: (a) The policy gradient theorem specifically requires 𝑉 𝜋 (ℎ), which contains precisely the correct information required to accurately estimate policy gradients. In this context, history values already contain the correct type and amount of information necessary to train the policy, and there is no such thing as "more informative values" than history values. (b) In theory, the history-state value in Theorem 5.2 could use any other state sampled according to s ∼ 𝑏 (ℎ), rather than the true system state, which would also result in the same analytical bias and variance properties. In practice, we only use the true system state due to it being directly available during offline training; however, we believe that its identity as the true system state is analytically irrelevant, which leads to the next interpretation of state.
State as a Feature. We conjecture an alternative interpretation according to which the state can be seen as a stochastic high-level feature of the history. Consider a history critic V (ℎ); to appropriately model the value function 𝑉 𝜋 (ℎ), V (ℎ) must first learn an adequate history representation, which is in and of itself a significant learning challenge. The critic model would likely benefit from receiving auxiliary high-level features of the history 𝜙 (ℎ). The resulting critic V (ℎ, 𝜙 (ℎ)) remains fundamentally a history critic, as the auxiliary features are exclusively a modeling/architecture construct. Next, we consider what kind of high-level features 𝜙 (ℎ) would be useful for control. While the specifics of what makes a good history representation depend strongly on the task, there is a natural choice which is arguably useful in many cases: the belief-state 𝑏 (ℎ). Because the belief-state is a sufficient statistic of the history for control, providing it to the critic model V (ℎ, 𝑏 (ℎ)) is likely to greatly improve its ability to generalize across histories. Finally, we conjecture that any state sampled according to the belief-state 𝑠 ∼ 𝑏 (ℎ)-including the true system state-can be considered a stochastic realization of the belief-state feature, resulting in the history-state critic V (ℎ, 𝑠). According to this interpretation, the importance of the state in the history-state critic is not in its identity as the true system state, but as a stochastic realization of hypothetical belief-state features, and presumably any other state sampled from the belief-state s ∼ 𝑏 (ℎ) could be equivalently used.
We compare the learning performances of five actor-critic variants. A2C, A2C-asym-s, and A2C-asym-hs are respectively (symmetric) A2C with history critic V (ℎ), asymmetric A2C with state critic V (𝑠), and asymmetric A2C with history-state critic V (ℎ, 𝑠). To demonstrate that the environments feature significant partial In contrast, our proposed unbiased asymmetric variant A2Casym-hs displays some of the best learning characteristics across all environments. In Cleaner, Memory-Four-Rooms-7x7, and Memory-Four-Rooms-9x9, its performance matches that of A2C (Figures 3f to 3h), while in Car-Flag it matches that of A2C-asyms (Figure 3e). In and of itself, this indicates that A2C-asym-hs is able to exploit whichever source of information (history or state) happens to be more suitable in practice to solve a given task. On top of that, A2C-asym-hs demonstrates strictly better final performance and/or convergence speed than both A2C and A2C-asym-s in Shopping-5 and Shopping-6 (Figures 3c and3d), demonstrating that it is not only able to use the best source of information, but also of combining both sources to achieve a higher best-of-bothworlds performance. This ability is pushed one step further and demonstrated in Heaven-Hell-3 and Heaven-Hell-4, where A2Casym-hs is the only method capable of learning to solve the task at all (Figures 3a and3b). These results strongly demonstrate the importance of exploiting asymmetric information in ways which are theoretically justified and sound, as done in our work.
To further inspect the behavior of each critic, Figure 4 shows the evolution of critic values over the course of training for important history-state pairs in Heaven-Hell-4. We use 4 deliberately chosen history-state pairs which are particularly important in this environment. In each case, the agent is located at the fork between heaven and hell, and the cases differ by the position of heaven (left or right) and whether the agent has previously performed the information-gathering sequence of actions necessary to know the position of heaven (by visiting the priest).
Unsurprisingly, we first note that critic values are correlated with the respective agent's performance (Figure 3b). Beyond that, the critics show certain individual characteristics: namely, the critics which focus on a single aspect of the join history-state output the exact same values for different history-states. Although hard to see, the A2C critic V (ℎ) outputs are identical in Figures 4a and4b, as those values are associated with the same histories (but not the same states). Similarly, the A2C-asym-s critic V (𝑠) outputs are identical in Figures 4a and4c and Figures 4b and4d respectively, as those values are associated with the same states (but not the same histories). This confirms a straightforward truth: that the state critic V (𝑠) is intrinsically unable to differentiave between values associated to different histories if they happen to be associated with the same state, which can be particularly detrimental in such informationgathering and memory dependent tasks. On the other hand, the A2C-asym-hs critic V (ℎ, 𝑠) has the ability to output different values, as needed, for each of the four cases. Note, in particular, that the A2C-asym-hs critic is able to associate a higher reward to the agent if it has already performed the information-gathering actions (Figures 4c and4d), compared to when it has not (Figures 4a and4b), which helps the agent determine that the information-gathering actions are important and should be performed.
In partially observable control problems, the offline training/online execution framework offers the peculiar opportunity to access the system's state during training, which otherwise remains latent during execution. Asymmetric methods trained offline can potentially exploit such privileged information to help train the agents to reach better performance and/or train more efficiently and using less data than before. While this idea has great potential, current state-of-the-art methods are motivated and driven by empirical results rather than theoretical analysis. In this work, we exposed fundamental theoretical issues with a standard variant of asymmetric actor-critic which made use of state critics 𝑉 𝜋 (𝑠), and proposed an unbiased asymmetric variant which makes use of history-state critics 𝑉 𝜋 (ℎ, 𝑠) and is the first of its kind to be analytically sound and theoretically justified. Although this represents a relatively simple change, its effects are profound, as demonstrated in both theoretical analysis and empirical results. Our evaluations confirm our analysis, and demonstrate both the issues with state-based critics and the benefits of history-state critics in environments which exhibit significant partial observability.
Although our evaluation only concerns A2C, the same concepts are easily extensible to other critic-based RL methods [8,19,22,29]. The potential for future work is varied. One possibility is to extend the theory of history-state value functions to optimal value functions 𝑄 * (ℎ, 𝑠, 𝑎), and develop theoretically sound asymmetric variants of value-based deep RL methods such as DQN [23]. Another possibility is to integrate asymmetric information with state-ofthe-art maximum entropy value/critic-based methods such as soft Q-learning [11], and soft actor-critic [12]. Finally, another venue for improvement is to extend our theory and approach to multiagent methods, potentially bringing theoretical rigor and improved performance [9,18,20,21,27,28,32,37].
Proc. of the 21st International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2022), P. Faliszewski, V. Mascardi, C. Pelachaud, M.E. Taylor (eds.), May 9-13, 2022, Online. © 2022 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.(a) State. (b) Observation.