The robustness of a downstream task varies a lot depending on the specific task and the type of noise. For instance, topic modeling has previously been shown to be robust to deletion of random words, whereas the insertion of new words and phonetic substitution errors has a larger negative impact on topic stability [1]}. Another relevant downstream task, neural machine translation with character-based models, has shown to struggle even with small perturbations to the input data [2]}.
