has a long history in deep learning, where it was initially used as a greedy layer-wise method to improve optimization [1]}, [2]}, [3]}, [4]}, [5]}.
In the context of unsupervised feature learning for image classification, different tasks related to denoising autoencoders have been considered, such as in-painting [6]}, colorization [7]} or de-shuffling of image patches [8]}.
In natural language processing, denoising autoencoders have been applied by masking or randomly replacing some tokens of the input, and reconstructing the original sequence, leading to the BERT model [9]}.
Similar methods have been proposed to pre-train sequence-to-sequence models, by considering additional kind of noises such as word shuffling or deleting [10]}, [11]}.
