The agent is composed of a feedforward ANN and is trained in local “RL cycles" within each iteration, where the agent rolls out and collects experience in simulations generated by the model, optimizes over the experience using PPO, and tests the learned policy in the live environment to log performance with cumulative rewards achieved. Depending on what is desired of the architecture, there can be a fixed \(E_{RL}\)  cycles performed at each iteration where \(E_{RL}\ge 1\) , or until a desired average cumulative reward is achieved. At each timestep \(t\) , the agent receives as input the concatenation of the latent state vector \(z_t\) , and hidden and cell state vectors \(h_t\) , \(c_t\)  from the LSTM controller in the DNC. This is similar to how the Controller receives input from the CVAE and LSTM Model in the World Models framework [1]}, and works well in practice.
