Substitutions and swaps were experimented with extensively in previous work (Heigold et al., 2017; <cite>Belinkov and Bisk, 2018</cite>) , but deletion and insertion were not. Deletion and insertion pose a different challenge to character encoders, since they alter the distances between character sequences in the word, as well as its overall length.