In this phase, the data cleaning and filtering are consistent with previous work [1]}.
However, we find that the previous code classification methods treat the code word as the basic unit. Its disadvantage is that it cannot effectively solve the out-of-vocabulary (OOV) problem. That means there exist some words, which are not in the training set but in the testing set.
To solve the OOV problem, we use the Byte-Pair Encoding (BPE) proposed by Sennrich et al. [2]}. It is a mixture between character-level and word-level representations. Using BPE can avoid a large number of "[UNK]" symbols in the test set, as "[UNK]" symbols may decrease the performance of the pre-trained model. For example, the original code snippet is “def split_lines(s): return s.split(`\(\backslash \) n')", and the result after using BPE segmentation is “def", “Ġsplit", “_", “lines", “(", “s", “)", “:", “Ġreturn", “Ġs", “.", “split", “(", “'", “Ċ", “'", “)". blackHere Ġ means that it is the first subword of a subword division.
