Pre-training and fine-tuning. Pre-training on large corpora of fuzzy data and then fine-tuning on a specific task with a smaller dataset has been shown to be highly successful in domains such as natural language processing [1]}, [2]}, image processing [3]} and most recently programming language processing [4]}, [5]}. In contrast to RealiT, these techniques are often pre-trained on a generic unrelated task where data is available before fine-tuning them on a specific task. RealiT, however, is trained and fine-tuned with same architecture with largely the same objective of identifying and repairing buggy (or mutated) code.
