Pre-training with code mutants. During pre-training, we train our model similar to the way current localization and repair models are trained [1]}. Here, the training objective is not to identify real bugs but rather to identify and transform mutated code snippets back into the real code snippets. For this task, we naturally start with a general corpus of Github code snippets (e.g. function implementations). This corpus can often easily be obtained by mining the recent version of popular open source projects. Since bugs are scarce in open source projects, we can safely assume that most of the code snippets are likely correct.
For training, we mutate each code snippet in our corpus at max \(k\)  times which produces a dataset of at max \(k\)  unique mutants per code snippetThe number of code rewrites applicable to a code snippet and hence the number of unique mutants per code snippet is limited by design and might be lower than \(k\) . We never introduce mutation duplicates..
During our experiments, we decided to employ an unusually large number of mutants per code snippet (\(k\)  = 100) since we observed that this improves the performance after fine-tuning. We employ the original code corpus as training examples of unmutated correct code.
Based on the two datasets, RealiT  is then trained to (1) distinguish mutants from real code, (2) identify the mutated location (if any) and (3) find the original replaced token. Since the dataset of mutants is up to \(k\)  times larger than the set of correct code snippet (by construction), we additionally supersampleDuring training, mutants and correct programs are sampled at the same frequency. As the set of mutants is up to \(k\)  times larger, we uniformly (super-)sample correct programs multiple times to match the number of mutants seen during training. each correct code snippet such that RealiT  is trained on correct and mutated code snippets at the same frequency. This avoids that the model is biased towards detecting mutants.
