Systems that are trained on clean data generally perform poorly when faced with such errors at test time (Heigold et al., 2017; <cite>Belinkov and Bisk, 2018</cite>) . One potential solution is to introduce noise at training time, an approach that is similar in spirit to the use of adversarial examples in other areas of machine learning (Goodfellow et al., 2014) and natural language processing (Ebrahimi et al., 2018) . So far, using synthetic noise at training time has been found to only improve performance on test data with exactly the same kind of synthetic noise, while at the same time impairing performance on clean test data (Heigold et al., 2017; <cite>Belinkov and Bisk, 2018</cite>) . We desire methods that yield good performance on both clean text as well as naturally-occurring noise, but this is beyond the reach of current techniques.