In this phase, we continue to pre-train the RoBERTa model on the code corpus with the MLM (mask language model) method, and then use the pre-trained model to fine-tune the code classification task. RoBERTa [1]} is similar to Bert (Bidirectional Encoder Representations from Transformers)[2]}, and DeepSCC uses Transformer as the method's main framework because Transformer can more thoroughly capture the bidirectional relationship in the text.
In particular, we treat the code as text and use the method MLM to constantly fine-tuning the roberta-base modelhttps://huggingface.co/roberta-base on the corpus to obtain our pre-trained language model. During the fine-tuning process, we do not tune the parameters of the model's bias and LayerNorm.weight weights, and use the AdamW method to fine-tune the other parameters.
