While the process of training is agnostic to any particular learning algorithm, backpropagation through time with stochastic gradient descent applied to the Adam optimizer [1]} with a learning rate of \(0.001\)  is used to minimize the loss in this instance. The neural network used to generate the program is represented by a 2-hidden layer LSTM with 64 cells in each layer. The network output vector is the dimensionality of the number of possible characters plus a set of auxiliary neurons. The auxiliary neurons allow for the network to learn signals can be used in the input layer at the next character generation timestep. The generated character during training is sampled categorically using a softmax on the character neuron output vector for the category probabilities. During reproductive evaluation, the generated character is taken from the character neuron with the maximum value. The input information at generation time \(t\)  is the output of the network at time \(t-1\)  along with a set of \(N=8\)  random-noise neurons sampled from a normal distribution. At the first generation timestep \(t = 0\) , the output vector portion of the input is represented by a one-vector.
