An essential class of gradient-free methods are evolutionary algorithms [1]}, [2]}, [3]}.
Recently, [4]} showed that a set of evolutionary methods called Evolutionary Strategies (ES) is competitive in RL by reformulating agent training as an optimization task.
In RL, the goal is to find the agent's action distribution \(\pi \)  (the policy), maximizing some cumulative reward function \(\textsf {J}\) .
The policy is usually parameterized with a set of parameters \(\mathbf {\theta }\) .
It follows that the reward is a function of the parameters of the policy: \(\textsf {J}(\pi (\mathbf {\theta })) = \textsf {J}(\mathbf {\theta })\) .
The idea of [4]} and similar works of [6]}, [7]}, [8]}, [9]}, [10]} is to directly optimize the cumulative reward function \(\textsf {J}(\mathbf {\theta }) = \textsf {J}(\theta _1, \theta _2, \ldots , \theta _d)\)  with respect to the parameters of the policy.
We pursued a similar approach to transform a traditional Markov Decision Process into an optimization problem (we provide the details in Appendix REF ).
Although the agents trained with ES often demonstrate more rich behavior and better generalization compared to traditional gradient-based policy optimization, the convergence of ES is often slow [10]}.
