The issues associated with bounded actions in Deep RL policy gradients has been explored previously. One approach is to use the clipped action, referred to here as \(\hat{a}_h\) , directly in the policy gradient estimate in place of the unclipped action \(a_h\) . Unfortunately this has been shown to introduce significant bias into the policy gradient estimation process [1]} by effectively corrupting the empirical average (REF ). The clipped action policy gradient method [2]} attempts to alleviate this bias issue by replacing the gradient with calculations based on the cumulative distribution function (CDF). However, computing the CDF can be non-trivial (such as with Gaussian distributions where no closed form exists). The work of [3]} employs the use of a \(\tanh \)  squashing function to transform an unbounded Gaussian distribution to a bounded distribution as \(\hat{a}_h = \tanh (a_h)\) , where \(a_h\)  is the action produced by the (Gaussian) policy (REF ). However, the \(\tanh \)  transformation does not solve the “windup issue,” as the input to the \(\tanh \)  function can grow without bound with no further effect on the output. The work of [1]} investigates the applicability of the Beta distribution, which has bounded support, to represent the stochastic policy. One drawback of the Beta distribution is that the policy must be nearly deterministic for near-limit action values to be chosen with high probability [2]}.
