The Deep RL results are compared against a benchmark solution obtained through nonlinear programming (NLP) techniques. To generate the NLP benchmark we utilize Pyomo [1]}, a general, open-source, Python-based algebraic modeling language developed at Sandia National Labs. The extension pyomo.dae [2]} integrates with Pyomo to accommodate optimization problems with differential and algebraic equations (DAEs). In this work a Backward Euler finite difference scheme with a fixed timestep of 2 seconds is used to approximate the shuttle DAEs. The optimal time horizon length is determined by the solution of the NLP. The final output from solving the NLP is a full trajectory along the horizon. The underlying optimization is solved through the interior-point solver IPOPT [3]}. Details of using pyomo.dae in an aerospace optimal control context can be found in [4]}.
