Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.

Recently, there has been a series of off-policy algorithms derived from this perspective for learning policies with function approximations [2,22,48]. Notably, Soft Actor Critic (SAC) [22,23], based on a maximum entropy objective and soft Q-function, significantly outperforms on-policy [52,54] and off-policy [36,18,13] methods. Maximum a posteriori Policy Optimisation (MPO) [2], inspired by REPS [50], employs a pseudo-likelihood objective, and achieves high sample-efficiency and fast convergence compared to the variety of policy gradient methods [36,54,7]. Similar to MPO, Advantage Weighted Regression (AWR) [48], and its variant, Advantage Weighted Actor-Critic (AWAC) [42], also employ the pseudo-likelihood objective weighted by the exponential of advantages, and reports more stable performance than baselines both in online and offline [35] settings.
While these inference-based algorithms have similar derivations to each other, their empirical performances have large gaps among them when evaluated on the standard continuous control benchmarks, such as OpenAI Gym [6] or DeepMind Control Suite [58]. Critically, as each algorithm has unique low-level implementation or code design decisions -such as value estimation techniques, action distribution for the policy, and network architectures -aside from high-level algorithmic choices, it is difficult to exactly identify the causes of these performance gaps as algorithmic or implementational.
In this paper, we first derive MPO, AWR, and SAC from a single objective function, through either Expectation-Maximization (EM) or KL minimization, mathematically clarifying the algorithmic connections among the recent state-of-the-art off-policy actor-critic algorithms. This unified derivation allows us to precisely identify implementation techniques and code details for each algorithm, which are residual design choices in each method that are generally transferable to other algorithms. To reveal the sources of the performance gaps, we experiment with carefully-selected ablations of these identified implementation techniques and code details, such as tanh-squashed Gaussian policy, clipped double Q-learning [13], and network architectures. Specifically, we keep the high-level algorithmic designs while normalizing the implementation designs, enabling proper algorithmic comparisons. Our empirical results successfully distinguish between highly co-adapted design choices and no-co-adapted ones 2 . We identified that clipped double Q-learning and tanh-squashed policies, the sources of SoTA performance of SAC, are highly co-adapted, specific to KL-minimization-based method, SAC, and difficult to transfer and benefit in EM-based methods, MPO or AWR. In contrast, we discover that ELU [8] and layer normalization [4], the sources of SoTA performance of MPO, are transferable choices from MPO that also significantly benefit SAC. We hope our work can inspire more future works to precisely decouple algorithmic innovations from implementation or code details, which allows exact sources of performance gains to be identified and algorithmic researches to better build on one another.

Related Work
Inference-based RL algorithms RL as probabilistic inference has been studied in several prior contexts [60,61,34,11,57,45], but many recently-proposed algorithms [2,22,48] are derived separately and their exact relationships are difficult to get out directly, due to mixing of algorithmic and implementational details, inconsistent implementation choices, environment-specific tunings, and benchmark differences. Our work organizes them as a unified policy iteration method, to clarify their exact mathematical algorithmic connections and tease out subtle, but important, implementation differences. We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both 50,43,44,1,56,42] and 51,12,29,21,33,32] RL and achieve some of the most competitive performances on popular benchmarks [6,58]. REPS [50], an EM approach, inspired MPO, AWR, and our unified objective in Eq. 1, while Soft Q-learning [21], a practical extension of KL control to continuous action space through Liu and Wang [38], directly led to the development of SAC.
Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms' high-variance evaluation performances, reproducibility difficulty [9, 28, 65], and frequent code-level optimizations [25,63,10,3]. Henderson et al. [25] empirically showed how these RL algorithms have inconsistent results across different official implementations and high variances even across runs with the same hyper-parameters, and recommended a concrete action item for the community -use more random seeds. Tucker et al. [63] show that high performances of actiondependent baselines [19,17,37,20,67] SAC [22], and other algorithms (see Appendix B). We follow the notation of Sec. 3 & 4. We characterize them with the algorithm family (EM or KL), how policies are updated (πq and πp; SG stands for stochastic-gradient-based, and TR stands for trust-region-based), the choice of G(·), how G is estimated, and the parameterization of the policy. While the advantage function just can be interpreted as Q-function with baseline subtraction, we explicitly write A π when the state-value function is parameterized, not Q-function. ( * Note that DDPG and TD3 are not "inference-based", but can be classified as KL control variants.) how code-level optimizations, instead of the claimed algorithmic differences, actually led more to PPO's superior performances. Andrychowicz et al.
[3] describes low-level (e.g. hyper-parameter choice, and regularization) and high-level (e.g. policy loss) design choices in on-policy algorithms affects the performance of PPO by showing results of large-scale evaluations.
In contrast to those prior works that mainly focus on a single family of on-policy algorithms, PPO and TRPO, and evaluating their implementation details alone, our work focuses on two distinct families of off-policy algorithms, and more importantly, presents unifying mathematical connections among independently-proposed state-of-the-art algorithms. Our experiments demonstrate how some implementation choices in Table 1 and code details are co-evolved with algorithmic innovations and/or have non-trivial effects on the performance of off-policy inference-based methods.

Preliminaries
We consider a Markov Decision Process (MDP) defined by state space S, action space A, state transition probability function p : S × A × S → [0, ∞), initial state distribution p 1 : S → [0, ∞), reward function r : S × A → R, and discount factor γ ∈ [0, 1). Let R t denote a discounted return ∞ u=t γ u−t r(s u , a u ). We assume the standard RL setting, where the agent chooses actions based on a parametric policy π θ and seeks for parameters that maximize the expected return E π θ [R 1 ]. Value functions for a policy π are the expected return conditioned by a state-action pair or a state, that is, Q π (s t , a t ) := E π [R t |s t , a t ], and V π (s t ) := E π [R t |s t ]. They are called the state-action-value function (Q-function), and state-value function, respectively. The advantage function [5] is defined as A π (s, a) := Q π (s, a) − V π (s). The Q-function for a policy π is the unique fixed point of the Bellman operator T π defined by T π Q(s, a) = r(s, a) + π(a |s )p(s |s, a)Q(s , a ) ds da for any function Q : S × A → R. We denote a trajectory or successive state-action sequence as τ := (s 1 , a 1 , s 2 , a 2 , . . . ). We also define an unnormalized state distribution under the policy π by d π (s) = ∞ t=1 γ t p(s t = s|π).
Inference-based Methods For simplicity, we consider a finite-horizon MDP with a time horizon T for the time being. As a result, a trajectory τ becomes a finite length: τ := (s 1 , a 1 , . . . , s T −1 , a T −1 , s T ).
As in previous works motivated by probabilistic inference [34,2], we introduce to the standard graphical model of an MDP a binary event variable O t ∈ {0, 1}, which represents whether the action in time step t is optimal or not. To derive the RL objective, we consider the marginal log-likelihood log Pr(O = 1|π p ), where π p is a policy. Note that O = 1 means O t = 1 for every time step. As is well known, we can decompose this using a variational distribution q of a trajectory as follows: is the evidence lower bound (ELBO). Inference-based methods aim to find the parametric policy which maximizes the ELBO J (p, q).
There are several algorithmic design choices for q and Pr(O = 1|τ ). Since any q is valid, a popular choice is the one that factorizes in the same way as p, that is, where π p is a prior policy, and π q is a variational posterior policy. We may also choose which policy (π p or π q ) to parameterize. As for Pr(O = 1|τ ), the most popular choice is the following one [21,22,2,34,48]: where η > 0 is a temperature, and G is a function over S × A, such as an immediate reward function r, Q-function Q π , and advantage function A π . While we employ exp(·) in the present paper, there are alternatives [46,47,55,66] Incorporating these design choices, we can rewrite the ELBO J (p, q) in a more explicit form as; In the following section, we adopt this ELBO and consider its relaxation to the infinite-horizon setting. Then, starting from it, we derive MPO, AWR, and SAC.

A Unified View of Inference-based Off-Policy Actor-Critic Algorithms
In Sec. 3, we provided the explicit form of the ELBO J (π p , π q ). However, in practice, it is difficult to maximize it as the expectation E q depends on π q . Furthermore, since we are interested in a finite-horizon setting, we replace t∈[T ] E q with E dπ(s) . Note that the latter expectation E dπ(s) is taken with the unnormalized state distribution d π under an arbitrary policy π. This is commonly assumed in previous works [2,22]. This relaxation leads to the following optimization:  Table 1. We first explain how these algorithms can be grouped into two categories of approaches for solving Eq. 1, and then we elaborate additional implementation details each algorithm makes.

Unified Policy Iteration: Algorithmic Perspective
Eq. 1 allows the following algorithmic choices: how or if to parameterize π p and π q ; what optimizer to use for them; and if to optimize them jointly, or individually while holding the other fixed. We show that the algorithms in Table 1 can be classified into two categories: Expectation-Maximization control (EM control) and direct Kullback-Leibler divergence minimization control (KL control). , and we term it EM control. At high level, the algorithm non-parametrically solves for the variational posterior π q while holding the parametric prior π p = π θp fixed (E-Step), and then optimize π p holding the new π q fixed (M-Step). This can be viewed as either a generic EM algorithm and as performing coordinate ascent on Eq. 1. We denote θ p and π q after iteration k of EM steps by θ (k) p and π (k) q , respectively.

Expectation-Maximization (EM) Control
In E-step at iteration k, we force the variational posterior policy π (k) q to be close to the optimal posterior policy, i.e., the maximizer of the ELBO J (π θ (k−1) p , π q ) with respect to π q . EM control converts the hard-constraint optimization problem in Eq. 1 to solving the following Lagrangian, We analytically obtain the solution of Eq. 2, where Z(s) is the partition function.
In M-Step at iteration k, we maximize the ELBO J (π (k) q , π p ) with respect to π p . Considering the optimization with respect to π p in Eq. 1 results in forward KL minimization, which is often referred to as a pseudo-likelihood or policy projection objective, (3) where we may approximate Z(s) ≈ 1 M M j=1 exp(η −1 G(s, a j )) with a j ∼ π θ (k−1) p (·|s) in practice.

Direct Kullback-Leibler (KL) Divergence Minimization Control
In contrast to the EM control in Sec. 4.1.1, we only optimize the variational posterior π q while holding the prior π p fixed. In this scheme, π q is parameterized, so we denote it as π θq . This leads to KL control [59,62,51,12,29,21].
Equivalent to E-step in Sec. 4.1.1, we force the variational posterior policy π θq to be close to the optimal posterior policy, i.e, the maximizer of J (π p , π q ) with respect to π q . The difference is that instead of analytically solving J (π p , π q ) for π q , we optimize J (π p , π θq ) with respect to θ q , which results in the following objective, max θq E dπ(s)π θq (a|s) η −1 G(s, a) − log π θq (a|s) π p (a|s) .

"Optimal" Policies of EM and KL Control
While we formulate EM and KL control in a unified framework, we note that they have a fundamental difference in their definition of "optimal" policies. KL control fixes π p and converges to a regularizedoptimal policy in an exact case [15,64]. In contrast, EM control continues updating both π p and π q , resulting in convergence to the standard optimal policy in an exact case [51]. We also note that EM control solves KL control as a sub-problem; for example, the first E-step exactly corresponds to a KL control problem, except for the policy parameterization.

Unified Policy Iteration: Implementation Details
In this section, we explain the missing pieces of MPO, AWR, and SAC in Sec. 4.1. Additionally, we also describe the details of other algorithms [30,49,50,41,56,36,13,52,54] from the EM and KL control perspective. See Appendix B for the details.

MPO
Algorithm MPO closely follows the EM control scheme explained in Sec. 4.1.1, wherein π p = π θp is parametric, and π q is non-parametric.
Implementation [π q Update] This corresponds to the E-step. MPO uses a trust-region (TR) method. Concretely, it replaces the reverse KL penalty (second term) in the Lagrangian (Eq. 2) with a constraint and analytically solves it for π q . As a result, η is also optimized during the training by minimizing the dual of Eq. 2, which resembles REPS [50]: [π p Update] This corresponds to the M-step. MPO uses a combination of Stochastic Gradient (SG) ascent and trust-region method based on a forward KL divergence similarly to TRPO [52]. Concretely, it obtains θ (k) p by maximizing the objective (Eq. 3) with respect to θ p subject to E dπ(s) [D KL (π θ (k−1) p (·|s) || π θp (·|s))] ≤ ε. MPO further decompose this KL divergence to two terms, assuming Gaussian policies; one term includes only the mean vector of π θp (·|s), and the other includes only its covariance matrix. Abdolmaleki et al.
[2] justify this as a log-prior of MAP estimation, which is assumed as a second-order approximation of KL divergence.
[G and G Estimate] MPO uses the Q-function of π θ (k−1) p as G in the k-th E-step. It originally uses the Retrace update [40], while its practical implementation [27] uses a single-step Bellman update.
[π θ ] For a parameterized policy, MPO uses a Gaussian distribution with state-dependent mean vector and diagonal covariance matrix. The trust-region method in MPO's M-step heavily relies on this Gaussian assumption. Since a Gaussian distribution has an infinite support, MPO has a penalty term in its policy loss function that forces the mean of the policy to stay within the range of action space.

AWR and AWAC
Algorithm AWR slightly deviates from the EM control. In the M-step, AWR simply set Z(s) to 1.

Implementation [π q
Update] This corresponds the E-step. AWR and AWAC analytically solves the Lagrangian (Eq. 2). In contrast to MPO, they don't use trust-region method.
[π p Update] This corresponds to the M-step. At iteration k, AWR uses an average of all previous policies π p k := 1 k k−1 j=0 π θ (j) p instead of π θ (k−1) p (cf. Eq. 3). In practice, the average policy π p k is replaced by samples of actions from a replay buffer, which stores action samples of previous policies.
[G and G Estimate] AWR uses the advantage function of π p k as G in the k-th E-step, and learns the state-value function of π θ (k−1) p with TD(λ) [53]. Due to its choice ofπ p k , this avoids importance corrections [40]. In contrast, AWAC estimates the advantage via the Q-function with TD(0).
[π θ ] For a parameterized policy, both AWR and AWAC use a Gaussian distribution with statedependent mean vector and state-independent diagonal covariance matrix (a constant one for AWR). As in MPO, they uses a penalty term to keep the mean of the policy within the range of action space.

SAC
Algorithm Contrary to MPO and AWR, SAC follows the KL control scheme explained in Sec. 4.1.2, wherein the variational posterior policy π q = π θq is parameterized, and the prior policy π p is fixed to the uniform distribution over the action space A. SAC uses as G a soft Q-function: with H(π θq (·|s t )) being −E π θq [log π θq (a t |s t )|s t ]. Implementation [π q and π p Update] SAC performs stochastic gradient (SG) ascent updates of only π q , where the objective function is shown in Eq. 1. SAC has no operation equivalent to M-step in MPO and AWR, since it keeps the prior policy π p to the uniform distribution.
Similar to the temperature tuning in MPO,Haarnoja et al. [23] consider the dual function of entropyconstrained Lagrangian and treat the temperature η as a Lagrange multiplier, which results in, whereH is the target entropy to ensure that the entropy of the policy should be larger than it.
[G and G Estimate] SAC uses Q π θq soft as G, and TD(0)-like algorithm based on this soft Q-function. In policy evaluation, clipped double Q-learning [13], which retains multiple Q-functions (typically two) and takes the minimum, is employed to suppress the overestimation of Q value.
[π θ ] The policy of SAC is a squashed Gaussian distribution: SAC first samples a random variable u from a Gaussian distribution N (µ θ (s), Σ θ (s)) with state-dependent mean vector and diagonal covariance; then, it applies the tanh function to u and obtain the action a ∈ [−1, 1] |A| . (If a dimension of the action space is not [−1, 1], an appropriate scaling and shift by an affine function is applied after the squashing.) Tanh squashing prevents out-of-bounds action.

Empirical Comparison
To evaluate the empirical performance of these off-policy algorithms (MPO, AWR, AWAC, and SAC), we compare their performances on Open AI Gym MuJoCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant, Humanoid, and Swimmer, following Haarnoja et al. [23]. We reproduce all algorithms based on pytorch RL library [14], referring their original implementations [22,48,42,27]. Figure 1 shows that SAC outperforms others in four environments (Hopper, Walker2d, HalfCheetah, and Humanoid), while MPO in Ant and AWR in Swimmer achieves the best performance. Generally, SAC seems to perform consistently better. We extensively evaluate MPO, AWR, and SAC on the 28 tasks on DeepMind Control Suite and 3 MuJoCo manipulation tasks. See Appendix C and D for the details.

Evaluation on Implementational Choices
In Sec. 4.3, SAC shows notable results in most environments, while we revealed that the derivation and formulation of those methods resemble each other. To specify the effect of each implementational   or code detail on the performance, we experiment with extensive and careful one-by-one ablations on; (1) clipped double Q-learning, (2) action distribution for the policy, (3) activation and normalization, and (4) network size. The former two correspond to implementation details (the choice of G estimate and the parameterization of the policy), and the latter two correspond to code details (see Appendix G for further experiments on other implementation details, such as π p update or the choice of G). We also conclude several recommendations for the practitioners (Table 6).

Clipped Double Q-Learning
We hypothesize that clipped double Q-learning has a large effect on the notable performance of SAC, and could be transferable in EM control methods. To verify this, we test the effect of clipped double Q-learning. Instead of AWR, here we evaluate AWAC since it uses Q-function. Table 2 shows that single-Q SAC outperforms original one in HalfCheetah and Swimmer that do not have the termination of the episode, while struggles to learn in Hopper, Walker, Ant and Humanoid that have the episodic termination conditions. A hypothesis is that single-Q SAC obtains a more exploratory policy due to larger overestimation bias, which can help in environments where explorations are safe, but hurt in environments where reckless explorations lead to immediate terminations. In contrast, clipped double Q-learning does not help MPO or AWAC as significantly as SAC; most results do not change or lead to slight improvements over the originals. This suggests that clipped double Q-learning might be a co-dependent and indispensable choice to KL control methods.

Recommendation
Use clipped double Q-learning as a default choice, but you can omit it in EM control methods or non-terminal environments, such as HalfCheetah or Swimmer.

Action Distribution for the Policy
Another hypothesis is that the tanh-squashed policy (last column in Table 1) is an important and transferable design choice. We compare SAC without tanh transform (with MPO action penalty instead) to the original one, which results in drastic degradation (Table 3). This can be caused by the maximum entropy objective that encourages maximizing the covariance. These observations suggest that the high performance of SAC seems to highly depend on the implementational choice of the policy distribution. In contrast, MPO and AWR with tanh squashing don't enjoy such large performance gain, or achieve worse cumulative rewards. This also suggests that tanh-squashed policy might be highly co-adapted in SAC and less transferable to EM control methods. Note that for EM control, we must clip the actions to keep them within the supports of distributions; a ∈ [−1 + , 1 − ] |A| . We found that no clipping cause significant numerical instability. See Appendix H for the details.
Recommendation Use the original distributions for each algorithm. If you use the tanh-squashed Gaussian in EM control methods, clip the action to be careful for the numerical instability.

Activation and Normalization
The implementation of MPO [27] has some code-level detailed choices; ELU activation [8] and layer normalization [4]. We hypothesize that these, sometimes unfamiliar, code details in practical implementation stabilize the learning process, and contribute to the performance of MPO most.    Table 4 shows that layer normalization can contribute to a significantly higher performance of SAC in Ant, and replacing ReLU with ELU also improves performance a lot in Swimmer, where AWR is the best in Figure 1. In contrast, the performance of MPO drastically collapsed when we just removed ELU and layer normalization. This observation suggests that these code-level choices are not only indispensable for MPO, but transferable and beneficial to both KL and EM control methods. Additionally, we tested incorporating ELU and layer normalization to SAC in 12 DM Control tasks where MPO outperformed SAC, and observed that they again often benefit SAC performances substantially. See Appendix C for the details.
Recommendation It is worth considering to replace the activation function from ReLU to ELU and incorporate the layer normalization, to achieve the best performance in several tasks. Both of them are transferable between EM and KL control methods.

Network Size
While on-policy algorithms such as PPO and TRPO can use the common network architecture as in prior works [10,25], the architecture that works well in all the off-policy inference-based methods is still not obvious, and the RL community does not have agreed upon default choice. To validate the dependency on the networks among the inference-based methods, we exchange the size and number of hidden layers in the policy and value networks. We denote the network size of MPO ((256, 256, 256) for policy and (512, 512, 256) for value) as large (L), of SAC ((256, 256) for policy and value) as middle (M), and of AWR ((128, 64) for policy and value) as small (S) (see Appendix A for the details). Table 5 illustrates that SAC with large network seems to have better performance. However, as shown in Appendix F (Figure 8), the learning curve sometimes becomes unstable. While the certain degree of robustness to the network size is observed in SAC, EM control methods, MPO  and AWR, seem fragile and more dependent on the specific network size. This trend is remarkable in MPO. AWR (M) or (L) also struggles to learn in high-dimensional state tasks, such as Ant (111 dim), or Humanoid (376 dim). The results also imply that, in contrast to on-policy methods [3], the network size in the off-policy inference-based methods seems a less transferable choice.
Recommendation For SAC, use medium size network. Large size will also work, but the learning curve might be unstable. For MPO, we strongly recommend to stick to large size, because it is very sensitive to the network size. For AWR, using small size is a better choice, especially in high-dimensional state tasks, such as Ant, or Humanoid.  In this work, we present a taxonomy of inference-based algorithms, and successfully identify algorithm-specific as well as algorithm-independent implementation details that cause substantial performance improvements. We first reformulated recent inference-based off-policy algorithms -such as MPO, AWR and SAC -into a unified mathematical objective and exhaustively clarified the algorithmic and implementational differences. Through precise ablation studies, we empirically show that implementation choices like tanh-squashed distribution and clipped double Q-learning are highly co-adapted to KL control methods (e.g. SAC), and difficult to benefit in EM control methods (e.g. MPO or AWR). As an example, the network architectures of inference-based off-policy algorithms, especially EM controls, seem more co-dependent than on-policy methods like PPO or TRPO, which therefore have significant impacts on the overall algorithm performances and need to be carefully tuned per algorithm. Such dependence of each algorithmic innovation on specific hand-tuned implementation details makes accurate performance gain attributions and cumulative build-up of research insights difficult. In contrast, we also find that some code-level implementation details, such as ELU and layer normalization, are not only indispensable choice to MPO, but also transferable and beneficial to SAC substantially. We hope our work can encourage more works that study precisely the impacts of algorithmic properties and empirical design choices, not only for one type of algorithms, but also across a broader spectrum of deep RL algorithms. [29] Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, 2017.

References
[30] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Advances in neural information processing systems, 2008.
[31] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
[33] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In Advances in Neural Information Processing Systems, 2020.

Appendix A Network Architectures
In this section, we describe the details of the network architectures used in Sec. 4 and 5.
We mainly used 4 GPUs (NVIDIA V100; 16GB) for the experiments in Sec. 4 and 5 and it took about 4 hours per seed (in the case of 3M steps). Actually, we conducted exhaustive evaluations through the enormous experiments, and we hope our empirical observations and recommendations help the practitioners to explore the explosive configuration space.

B Relations to Other Algorithms
We here explain the relation of the unified policy iteration scheme covers other algorithms. While we mainly focused on AWR, MPO, and SAC in the this paper, our unified scheme covers other algorithms too, as summarized in Table 1: EM control algorithms: • PoWER [30]: πp (= π θ ) update is analytic. G = η log Q πp and Q πp is estimated by TD(1). π θ = N (µ θ (s), Σ θ (s)) . • RWR [49]: πp = π θ is updated by SG. G = η log r , and π θ = N (µ θ (s), Σ) . When the reward is unbounded, RWR requires adaptive reward transformation (e.g. u β (r(s, a)) = β exp(−βr(s, a)); β is a learnable parameter). • REPS [50]: πp is πq of the previous EM step (on-policy) or a mixture of all previous πq (off-policy), which is approximated by samples. G = A πp and estimated by a single-step TD error with a state-value function computed by solving a dual function. πq is assumed as a softmax policy for discrete control in the original paper. • UREX [41]: πp = π θ is updated by SG. G = Q πp and estimated by TD(1). π θ is assumed as a softmax policy for discrete control in the original paper. • V-MPO [56]: Almost the same as MPO, but a state-value function is trained by n-step bootstrap instead of Q-function. Top-K advantages are used in E-step.

KL control algorithms:
• TRPO [52]: πq = π θ = N (µ θ (s), Σ θ ). The KL penalty is converted to a constraint, and the direction of the KL is reversed. G = A πp and estimated by TD(1). πp is continuously updated to πq. • PPO with a KL penalty [54]: πq = π θ = N (µ θ (s), Σ θ ). The direction of the KL penalty is reversed. G = A πp and estimated by GAE [53]. An adaptive η is used so that DKL(πp || πq) approximately matches to a target value. • DDPG 3 [36]: πq = π θ is the delta distribution and updated by SG. G = Q πq and estimated by TD(0). η = 0 (i.e., the KL divergence and πp update are ignored). • TD3 2 [13]: It is a variant of DDPG and leverages three implementational techniques, clipped double Q-learning, delayed policy updates, and target policy smoothing. • BRAC [68] and BEAR [31] (Offline RL): When we assume πp = π b (any behavior policy), and omitting its update, some of the offline RL methods, such as BRAC or BEAR, can be interpreted as one of the KL control methods. Both algorithms utilize the variants of clipped double Q-learning (λ-interpolation between max and min). Figure 2: Benchmarking results on DeepMind Control Suite 28 environments. The performances are averaged among 10 random seeds. We use an action repeat of 1 throughout all experiments for simplicity.

C Benchmarks on DeepMind Control Suite
In this section, we show the benchmarking results on 28 tasks in DeepMind Control Suite (Figure 2). Each algorithm is run with 2.5M steps (except for humanoid domain; 14M steps), following Abdolmaleki et al. [2]. While the previous work mentioned that tuning the number of action repeats was effective [33], we used an action repeat of 1 throughout all experiments for simplicity. We also use the hyper-parameters of each algorithm presented in Appendix A. As discussed in Sec. 5.3, we incorporate ELU and layer normalization into SAC in several domains where SAC is behind MPO or AWR. ELU and layer normalization significantly improve performances, especially in pendulum_swingup and cartpole_swingup_sparse. Some of MPO results don't seem to match the original paper, but we appropriately confirmed that these results are equivalent to those of its public implementation [27].

D Benchmarks on MuJoCo Manipulation Tasks
We extensively evaluate their performance in the manipulation tasks ( Figure 3). The trend seems the same as the locomotion tasks, while AWR beats SAC and MPO in Striker, which means they fall into sub-optimal.

E Reproduction Results of AWR on MuJoCo Locomotion Environments
We re-implemented AWR based on PFRL, a pytorch-based RL library [14], referring its original implementation [48]. Figure 4 shows the performance of our implementation in the original experimental settings, also following hyper-parameters. We recovered the original results in Peng et al. [48] properly.

F Learning Curves
In this section, we present learning curves of the experiments in Sec. 5.    Table 3; ablation of Tanh transformation. A line that stopped in the middle means that its training has stopped at that step due to numerical error. We test SAC without tanh squashing, AWR with tanh, and MPO with tanh.  Table 5; experiment for finer network sizes. (S) stands for the small network size from AWR, (M) for the middle network size from SAC, and (L) for the large network size from MPO.

G Additional Experiments for Deeper Analysis of Implementation Details
We share the additional experimental results for deeper analysis of implementation details and co-adaptation nature. We report the final cumulative return after 3M steps for Ant/HalfCheetah/Walker2d/Swimmer, 1M steps for Hopper, and 10M steps for Humanoid. All results below are averaged among 10 random seeds. These extensive experimental observations below suggest not only the co-adaptive nature and transferability of each implementation and code detail (discussed in Sec. 6), but also the properties of each kind of algorithm (KLbased and EM-based); KL-based methods, such as SAC, shows the co-dependent nature in implementation details (clipped double Q-learning, Tanh-Gaussian Policy) but robustness to the code details related to neural networks. In contrast, EM-based methods, such as MPO and AWR, show the co-dependent nature in code details but robustness to the implementation details. We hope these empirical observations from our experiments are valuable contributions to the RL community.
π p Update We test different types of πp Update as summarized in Table 1 to investigate the effectiveness of implementation choices. We prepare 4 variants: (1) MPO or AWAC with a uniform prior, (2) SAC with target policy instead of a fixed uniform prior, (3) MPO without trust-region (only SG update). The details of (1) -(3) are described below: (1) We use the actions sampled from uniform distribution as well as the samples from the policy at the past iteration π θ (k−1) p for the M-step in EM-controls; aj ∼ αUnif. + (1 − α)π θ (k−1) p , α ∈ (0, 1]. These variants are much closer to SAC (using uniform distribution as πp). We test α = 0.25, 0.5, 0.75 for both MPO and AWAC.
(2) We copy the parameter of πq at a certain interval and use it as πp in the objective of KL Control, similar to MPO/PPO/TRPO. It seems "KL-regularized" actor-critic, rather than "soft" (entropy-regularized  Table 11 shows that SAC without soft Q degrades its performance over 5 tasks except for Ant, while it is not so drastic compared to clipped double Q or Tanh-Gaussian policy. In contrast, MPO with soft Q slightly improves the performance (over 4 tasks), and AWAC with soft Q slightly also does (over 3 tasks). These trends are similar to the clipped double Q or Tanh-Gaussian policy. We think these experiments support our empirical observation: KL-based methods, such as SAC, show the robustness to the code details, while EM-based methods, such as MPO and AWR, show the co-dependent nature in code details but robustness to the implementation details. Network Size for AWAC To investigate the co-dependent nature between implementation and code details more precisely, we add the network size ablation of AWAC, whose implementations stand between MPO and AWR. AWAC differs πp Update and network size (the default choice of AWAC is (M)) from MPO (in fact, MPO uses TD (0) in open-source implementation [27] and we assume the difference of π θ might be minor). Also, AWAC differs G and G estimate from AWR.
The results of AWAC (Table 12) show a similar trend to AWR in high-dimensional tasks (Ant, Humanoid); a larger network did not help. We may hypothesize that πp Update of AWR/AWAC, mixture+SG, is not good at optimizing larger networks, compared to SG + TR of MPO. In contrast, especially, Hopper and Walker2d show a

H Failed Ablations
This section provides the failure case of ablations on tanh-squashed distributions and exchanging network architectures, which shows the catastrophic failure during training, and unclear insights.

H.1 Action Distribution for the Policy: Without Action Clipping
We observe that naive application of tanh-squashing to MPO and AWR significantly suffers from numerical instability, which ends up with NaN outputs (Table 15 and Figure 9). As we point out in Sec. 5.2, the practical solution is to clip the action within the supports of distribution surely; a ∈ [−1 + ,    Table 15. We test SAC without tanh-squashed distribution, AWR with tanh, and MPO with tanh. SAC without tanh transform (using MPO action penalty instead) results in drastic degradation of the performance, which can be caused by the maximum entropy objective that encourages the maximization of the covariance. AWR and MPO with tanh squashing become numerically unstable. A line that stopped in the middle means that its training has stopped at that step due to numerical error.

H.2 Network Architecture: Whole Swapping
In contrast to prior works on TRPO and PPO, the network architecture that works well in all the off-policy inference-based methods is not obvious, and the RL community doesn't have an agreeable default choice. Since the solution space is too broad without any prior knowledge, one possible ablation is that we test 3 different architectures that work well on at least one algorithm.
To validate the dependency of the performance on the network architecture, we exchange the configuration of the policy and value networks, namely, the size and number of hidden layers, the type of activation function, network optimizer and learning rate, weight-initialization, and the normalization of input state (See Appendix A). All other components remain the original implementations.
However, this ablation study might end up the insufficient coverage and the unclear insights. We broke down the network architecture comparison into the one-by-one ablations of activation and normalization, and experimented with finer network sizes. These results suggest that these off-policy inference-based algorithms might be fragile with other network architectures and more co-dependent with architectures than on-policy algorithms. A line that stops in the middle means that training has stopped at that step with numerical error due to NaN outputs.      Table 21: Results in Swimmer-v2 environment ( † numerical error happens during training). All results are averaged over 10 seeds and we also show their standard deviations.