[leftmargin=*]
Following [1]}, \(\pi _{\mathbf {\phi }}\)  is trained to maximize
\(\mathbb {E}_{(s,a) \in {\mathcal {B}}_{\mathrm {fake}}} \left[\log \left(D_{\mathbf {\psi }}(s,a)\right)\right].\)

We are motivated by [2]} to use LeakyReLU activation function in both the generator and the discriminator, with negative_slope being the default value \(0.01\) .

For stable training, we follow [2]} to use a reduced momentum term \(\beta _1 = 0.4\)  in the Adam optimizers of the policy and the discriminator networks; and to use learning rates \(\eta _{\mathbf {\phi }}= \eta _{\mathbf {\psi }}= 2 \times 10^{-4}\) .

To avoid discriminator overfitting, we are motivated by [4]} and [5]} to use one-sided label smoothing with soft and noisy labels.
Specifically, random numbers in \([0.8, 1.0)\)  are used as the labels for the “true” sample \({\mathcal {B}}_{\mathrm {true}}\) .
No label smoothing is needed for the “fake” sample \({\mathcal {B}}_{\mathrm {fake}}\) , i.e., their labels are all 0.

Binary cross entropy loss between the labels and the discriminator outputs is used as the loss for the discriminator training in the underlying GAN structure.

We are motivated by TD3 [6]} and GAN to update the policy \(\pi _{\mathbf {\phi }}(\cdot \,|\,s)\)  once per \(k\)  updates of the critics and the discriminator.

