Soft policy optimization. As the underlying actor-critic algorithm in MoPAC, we adopt SAC [1]} to benefit from the exploration induced by the soft policy updates based on the maximum entropy principle, counterbalancing the effect of the exploitation induced by MPRs. Nevertheless, MoPAC can be applied to any known off-policy actor-critic algorithm. In SAC, the training of the policy alternates between a soft policy evaluation step based on the soft Bellman backup operator [2]}, and a soft policy improvement step that minimizes the expected KL divergence: \(J_{\pi }(\theta , D)=\mathbb {E}_{s_t \sim D} [ D_{KL}(\pi || \exp {\lbrace Q^{\pi }-V^{\pi }\rbrace } )]\) , where \(Q^{{\pi }}\mbox{, } V^{\pi }\)  are the soft Q-function and soft VF of policy \(\pi \)  respectively.
