Unexpected output in a simple fill-mask task
This model tends to change some characters in the input which are not in '[MASK]' label.
For example, input:"我在在预测[MASK]结果", output:"我在这预测这个结果".
I mean, this model will not only generate '[MASK]' to a new one, but also generate something in the other part.（我在在 to 我在这）
Is there a solution to make this model just finish the simple fill-mask task? Thanks.
This model is trained to perform DAE (denoising auto-encoding) task, which not only predicts masked tokens but also corrects the noise tokens. So it will sometimes correct the 我在在 to 我在这.
A general solution to only generate on masked tokens is to further train the model to perform mask generation task, instead of DAE tasks.
But a quick solution is to do some post-processing of the generated text that keep the unmasked parts unchanged using string replace.