We give the off-policy learners 25 demonstration trajectories, each of length 1000. As described above, our non-sequential models are MLPs with two hidden layers of size 256 and ReLu activations. Our sequential models are LSTMs with hidden size 256 followed by an MLP with one hidden layer of size 256. We use a history of length 5 for all experiments and train all learners with a MSE loss and an Adam optimizer [1]} with learning rate \(3e-4\) . Our sequence models are given access to the last 5 states and the last 4 actions and are asked to predict the next action. We evaluate MSE and \(J(\pi )\)  by rolling out 100 trajectories and averaging.
