In this section, we give detailed information about the experiments. All experiments are conducted on a custom built machine with 36-core Intel Broadwell Xeon CPUs with 64 GB of RAM and are terminated in tens of minutes. In all tasks, both our method and policy gradient are trained to convergence and are capped at 1000 epochs. For each epoch, the gradient is estimated from a batch of 6000 step data sampled from the controller interacting with the plant and is applied to update the parameters. The trajectory length is capped at 200. The average reward from the sample of trajectories from the 6000 steps are reported. We show 500 epoch plots in Fig. REF  for clearer capture of the convergence process. The experiments and models are coded in Python [1]} with Tensorflow [2]}, CVXPY [3]} and MOSEKwww.mosek.com. We choose the learning rate of 1e-3, picked from grid search from 1e-1, 1e-2, 1e-3, 1e-4 to give best reward at convergence for policy gradient on the inverted pendulum task. We use ADAM optimizer [4]} and gradient clipping to a maximum magnitude of 10 for each parameter. We use \(\tanh \)  activation for all our neural network controllers. The implementation of policy gradient is consistent with [5]}. And our method adds the projection updates on the same code base. The initial guess for the Lyapunov matrix \(P^0\)  of Algorithm REF  is constructed using the method introduced in [6]}, which computes the Lyapunov matrix for output feedback control problem through LMIs. The initial guess for \(\Lambda ^0\)  is an identity matrix.
