Models Examined We train both Transformers of 44.3M parameters and bi-directional attentional LSTM of 6.2M parameters on the tasks, using the Fairseqhttps://github.com/pytorch/fairseq framework to conduct the experiments. We minimize the sum of log probabilities of the correct character via the Adam optimizer [1]}. For both models, we use a batch size of 128 split across 4 NVIDIA RTX 2080 GPUs, trained for 500 epochs using floating-point 16 precision. This results in an average training time of 18 hours. We then generate the output classification using beam search evaluated using the model checkpoint with the best validation accuracy, saved every 50 epochs. We include more hyper-parameter details in the appendix.
<FIGURE><FIGURE><FIGURE><FIGURE>